The present approach of Apply Compact LZX mode as FILEDISK with UsedSize = 3 - 5 GB and Apply WimBoot mode as RAMDISK with UsedSize 200 - 600 MB works quite well.
So there is no need to chop Or reduce size which has the advantage that the Win10x64 OS has full functionality.
What I can do is integrate Reduce WinSxS of cdob as extra Option in Apply Compact modes of VHD_WIMBOOT. That does not reduce the functionality.
But I suggest to apply it before Capture, doing it this way will help reduce the captured WIM file too (which is good for Wimboot too), or maybe better if added as a separate and independent feature, to let the user run it when he prefers, even on already made final VHD Compact mode installations, letting this way to test it, before make a rebuild.
Here it is WinSxS_Reduce_Trusted as a separate tool
Download and Unpack here and just double click to run WinSxS_Reduce_Trusted.cmd to Reduce WinSxS folder in Offline Windows e.g. Mounted VHD
The Used Size inside Win10x64 2004 Compact LZX VHD is reduced in 10 seconds of time from 5.36 GB down to 4.66 GB - Reduction of 0.70 GB in Compact LZX Install
In my case initially WinSxS has 51.969 files in 15.463 folders - After WinSxS Reduce in 10 seconds I have 57 files in 22 folders
More Info:
-Double click on WinSxS_Reduce_Trusted.cmd to Reduce WinSxS folder in Offline Windows e.g. Mounted VHD
- Program requires to be Trusted Installer in Windows 7/8/10 x64 OS and can also be used in Win10XPE x64 environment
- In WinSxS_Reduce_Trusted.cmd the App AdvancedRun.exe is used in case of 7/8/10 x64 OS to launch WinSxS_Reduce_x64.exe as Trusted Installer
Credits and Thanks to:
- cdob for making the core batch program base_winsxs.cmd slightly modified by wimb as base_winsxs_2.cmd
it works fine, but for 2 pieces of software, for which I have to restore the original manifests subfolder. I think I have already posted this minor perplexity weeks ago. I still don't know which files from the original manifests the system needs in order for the software to work back again, otherwise I would add only those files. is there anyway to determine that before copying back the whole manifests subfolder?
I think only way to find those required for your needs is removing one by one untill your problematic programs faill, just by curiosity I checked the folder Windows\WinSxS\Manifests on my Win10x64 2004 and in my case it has 17,044 files, almost all of them are smaller than 1 KB (this means 4 KB of disk used size each). Too much work to find the culprit(s)
Or maybe send to recycle bin on batches of 50 or 100 files each time and once the programs faill, restore that batch and remove smaller size batches, to isolate in which batch is one of the culprits.
almost all of them are smaller than 1 KB (this means 4 KB of disk used size each).
Only for the record not really-really (on NTFS).
If they are below 700 bytes or so (it depends on their filename , that usually is very loong for those files) they will take no storage clusters on a 512 bytes/sector disk, as they will be resident in the $MFT, JFYI:
No, approximatebecause the amount of characters in the file name is part of the available space.
The most that can be stored in a "normal" 1024 bytes $MFT (on 512 bytes/sector) is EXACTLY 744 bytes with a filename consisting of a single character and no extension, I posted a formula to calculate this exactly in the given link.
Additionally, the exact way the file is created may (or may not) lower the limit by 8 bytes.
Same goes for 4K $MFT records (on 4 KB/sector) the EXACT maximum is 3776 bytes, but it will be normally a little less.
The $MFT won't shrink, of course, its size is pre-determined when you create (format) the NTFS filesystem and is usually (largely) enough.
But it will grow if needed, so, if you create on purpose a smallish $MFT, then fill it with a zillion little files (no matter if they become $MFT resident) and you later delete them the $MFT will grow and remain larger.
If you want to have the smallest possible NTFS, you can create a very small volume, enlarge it to the desired size, then copy to it all the files from a volume where they were already pruned/deleted.
Like it is done to have a contiguous-contiguous filesystem extent for large files:
This way the pre-created $MFT won't be large enough to index all the files (resident or not resident) and it will grow little by little to the "minimum needed" size.
Depending on the overall size of the volume, this may well result in a noticeable difference in the available space, but most probably also compression will be (slightly) better.
Wonko
Well after this very long preamble, let's go to the main subject of the topic.
Wonko mentioned several tools on his quoted links, and commented depending of the tool he gets different results.
I remembered I have already available SwiftSearch on my PC and decided to give it a try for this task. I have used it before on Wimboot installations to see if a file is a real size file or only a Pointer to the coupled WIM (cero size on disk in this case).
Then I created C:\Test.txt file of 700 bytes, and copied it to other partitions on my HD, and also to C: renamed as Test-2.txt, and in all cases the size on disk is 0, wich I take as located on %MFT.
Then copied it again as Test-XXXXXXXXXXXXXXXXXX.txt, and just because the name is longer now, it is not located anymore on $MFT, and now same 700 bytes size file is consuming 4 K or 4,096 bytes, a full cluster size, which is the minimum used size a file stored outside of $MFT can be on NTFS.
Yes, I checked them on a normal installed Win10 first, (never ran WinSxS_Reduce), then ran WOF_Compress_x64 on that folder, only 26 files were compressed and 1 exclusion, no change even those compressed are using 4,096 bytes on disk. No doubt they are always outside $MFT.
Location:The Outside of the Asylum (gate is closed)
Italy
Posted 02 October 2020 - 07:47 AM
As a normal behaviour, a file can only be "created" in the $MFT, i..e. if you create a new file in Notepad (0 bytes) and you start typing in it (less than the around 700 character threshold) and save it, the file will go in the $MFT, if you add characters/bytes to it and then save, the file will "migrate" to the filesystem cluster (BTW what was written before the first save will remain in the $MFT and be duplicated in the cluster).
Most probably the WOF compression does some kind of "in place" compression, so since the file before compression was on a cluster, it remained on the cluster after compression, even if it could fit inside the $MFT.
An experiment would be to see what happens if the smallest files are copied (to another fileystem) and back.
Likely something may change on 4K sector disks, as the threshold goes up to around 3700 bytes.
I selected the smallest file for testing 131 bytes and 4 k size on disk (NOT on $MFT), but when copied to D: its size on disk is 0, wich means it is on $MFT now.
Location:The Outside of the Asylum (gate is closed)
Italy
Posted 02 October 2020 - 08:33 AM
I selected the smallest file for testing 131 bytes and 4 k size on disk (NOT on $MFT), but when copied to D: its size on disk is 0, wich means it is on $MFT now.
Booting from Win10XPE_x64_LZX.vhd, mounted 10x64-2004-LZX.vhd to H:
I wanted to go further and copied the folder H:\Windows\WinSxS\Manifests to D:, deleted the original and then copied back the folder to original location. Now a lot of files are 0 size on disk ( = on $MFT), also I was able to find the aproximate size limits for the transition:
If file size is 462 bytes or smaller it is 0 size on disk = located on $MFT
If file is 471 bytes or bigger it is 4,096 size on disk = located outside $MFT
Of course as we know it depends also of file name length.
EDIT: On Transition value picture just made, the same VHD is mounted on G:, I'm sure about the case on 462 bytes, but not shown on the picture
Location:The Outside of the Asylum (gate is closed)
Italy
Posted 02 October 2020 - 06:29 PM
Good , and - at the end of the day - how many files are now entirely in the $MFT?
If they are - say - half of the 17,408 you listed, let's say 8,500 rounded down, that would mean 8.500*4.096=34,816,000 bytes saved.
The appreciation is - like beauty - in the eye of the beholder, is that a pile of 24 floppy disks or an insignificant fraction of a stupidly large 1 TB disk?
Of course as we know it depends also of file name length.
EDIT: On Transition value picture just made, the same VHD is mounted on G:, I'm sure about the case on 462 bytes, but not shown on the picture
alacran
And opening the unmounted VHD with 7-zip, for all files on $MFT Compressed size is same as file size, but if the compressed size is 4,096 or bigger it means they are outside the $MFT.
Also we can see in attached picture some files with same 462 bytes or smaller size, stored outside of %MFT, just because their names are longer.
so what is the situation of the files at the end of the day? ?se llega a cierta conclusión o no se llega a conclusión ninguna? A line must be drawn between failure and success.
Location:The Outside of the Asylum (gate is closed)
Italy
Posted 03 October 2020 - 08:13 AM
so what is the situation of the files at the end of the day? ?se llega a cierta conclusion o no se llega a conclusión ninguna? A line must be drawn between failure and success.
Which line?
The experiment is of course (as expected BTW) a success, the point is only on the usefulness (or not) of implementing it "in production", and this - as said - is (and can only be) in the eye of the beholder.
As I see it, there is an advantage in having resident files (size and speed of access), but this advantage is unlikely to be noticeable.
When (if) the devices will be "native" 4K sectored, the advantage may become noticeable, because it will apply to a much wider number of files, though the saving will be proportionally smaller.
I.e.
with 512 bytes sector a file up to around 460 bytes (thanks to the stupidly long names, could have been more than 700) achieves a saving of 4096 bytes, i.e. a "compression ratio" of (1024+4096 vs. 1024) 1024/5120=20%
with 4K bytes sector a file up to around 3400 bytes (still thanks to the stupidly long names, could have been more than 3700) achieves a saving of 4096 bytes. i.e. a much lower "compression ratio" of (4096+4096 vs. 4096) 4096/8192=50% BUT it will apply to something like double the number of files.
math proves u right, as usual. the "discourse" raises a few layman's questions: can we reduce the number of letters in filenames and use a link fixer to make it all go back right again? will it be 4k? considering the power of modern hardware and the slightest hindrance it suffers from compression (together with the advantages that may accrue), can we go towards default-compression or no-compression-at-all (because the minimum space possible is already occupied) scenarios?
u rightly mentioned production - a few irrefutable points have been made de facto through the years:
1) for powerful hardware, the smaller the better; -->
2) --> albeit fractioned (wim-vhd), wimboot is no longer a curiosity; it can be used in production as well, as a dynamically stable system (even more so if duly compressed (see alacrán's area); -->
3) --> it could rightly be so also for 90% of unaware bloated-windows users, who would greatly benefit from the flying dutchman's applications;
4) for those who do not like combos, a duly compressed full vhd will do the same job in a slightly faster fashion (I say slightly because I have a few tests whereby some wim+vhd combos fare no worse (or sometimes even better), than its full-vhd counterparts); anyway see alacrán for further and wider feedback;
5) the brute-force-shrinking method is ultimately proving the most effective in terms of speed, not so in terms of saved space, which probably goes back to the point wonko and alacrán were initially making here (some limitation in free-space gaining as a result of limited used space shrinking).
Location:The Outside of the Asylum (gate is closed)
Italy
Posted 03 October 2020 - 11:48 AM
BTW (and a a side-side note) it is perfectly possible (at least it is possible on XP, probably on 7, cannot say on 10) to create a virtual disk that is "native" 4 K sectored using MS own virtual disk driver:
Once the NTFS volume is created on it (with 4096 bytes for each $MFT entry) the filesystem can be deployed to 512 bytes sector devices just fine with minimal "fixes", the experiment was made (and it is more complicated than what would be needed for a "simpler" 4096 bytes per $MFT entry) and worked just fine:
so what is the situation of the files at the end of the day? ?se llega a cierta conclusión o no se llega a conclusión ninguna? A line must be drawn between failure and success.
In accordance with topic title all the experiments were successful.
On any NTFS drive we can use SwiftSearch to check which files are stored on $MFT
On unmounted VHDs we can also use 7-zip to check which files are stored on $MFT
If the original folder Windows\WinSxS\Manifests is copied to another drive, then delete the original, and latter copied back, almost all files having about 460 bytes or smaller size will be now stored on $MFT, saving a good amount of used size on a NTFS drive. (Unless they are latter modified).
Saved espace will be a cluster (4 K or 4,096 bytes on NTFS) for each file relocated to $MFT
If all or any part of this info is useful for you or not, it's something you have to decide by yourself.