Jump to content











Photo

Using SwiftSearch to know if a little file is stored directly on $MFT


  • Please log in to reply
19 replies to this topic

#1 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 01 October 2020 - 08:11 PM

Using SwiftSearch to know if a little file is stored directly on $MFT

 

I decided to better start this new topic because I really hate when a topic is hijacked to talk about subjects not directly related to the topic.

 

To keep things togeter, I'm going to quote the posts where this talk started:

 

From: http://reboot.pro/to...-13#entry216384

Spoiler

 

From: http://reboot.pro/to...-13#entry216385

Spoiler

 

From. http://reboot.pro/to...-13#entry216386

Spoiler

 

From: http://reboot.pro/to...-13#entry216388

Spoiler

 

From: http://reboot.pro/to...-14#entry216393

Spoiler

 

From: http://reboot.pro/to...-14#entry216394

Spoiler

 

From: http://reboot.pro/to...-14#entry216401

Spoiler

 

Well after this very long preamble, let's go to the main subject of the topic.

 

Wonko mentioned several tools on his quoted links, and commented depending of the tool he gets different results.

 

I remembered I have already available SwiftSearch on my PC and decided to give it a try for this task. I have used it before on Wimboot installations to see if a file is a real size file or only a Pointer to the coupled WIM (cero size on disk in this case).

 

Then I created C:\Test.txt file of 700 bytes, and copied it to other partitions on my HD, and also to C: renamed as Test-2.txt, and in all cases the size on disk is 0, wich I take as located on %MFT.

 

Then copied it again as Test-XXXXXXXXXXXXXXXXXX.txt, and just because the name is longer now, it is not located anymore on $MFT, and now same 700 bytes size file is consuming 4 K or 4,096 bytes, a full cluster size, which is the minimum used size a file stored outside of $MFT can be on NTFS.

 

alacran

Attached Files

  • Attached File  Test.png   61.12KB   0 downloads

  • wimb likes this

#2 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 01 October 2020 - 08:23 PM

Nice demonstration of using SwiftSearch  :)  (and Not SwiftSeark as in the Title  ;) )

 

and Thanks to open new topic about Using SwiftSearch to know if a little file is stored directly on $MFT


  • alacran likes this

#3 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 02 October 2020 - 06:04 AM

I have used SwiftSearch to check the Size on Disk for the files in the Manifests folder After using WinSxS_Reduce_Trusted on my Win10x64 VHD

 

As shown All 25 files have Size on Disk greater 0 which means they are Not stored directly on $MFT 

 

Even the very small files in Manifests folder of my C:\ drive Win10x64 OS also have Size on Disk greater 0. 

 

Conclusion will be that the very long filenames (greater than 25 chars ?) prevents them from being stored directly on $MFT

 

Attached File  Manifests_2020-10-02_075936.jpg   150.56KB   0 downloads == Attached File  Manifests_2020-10-02_081038.jpg   282.03KB   0 downloads



#4 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 02 October 2020 - 06:45 AM

Yes, I checked them on a normal installed Win10 first, (never ran WinSxS_Reduce), then ran WOF_Compress_x64 on that folder, only 26 files were compressed and 1 exclusion, no change even those compressed are using 4,096 bytes on disk. No doubt they are always outside $MFT.

 

alacran

Attached Files



#5 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 02 October 2020 - 07:47 AM

As a normal behaviour, a file can only be "created" in the $MFT, i..e. if you create a new file in Notepad (0 bytes) and you start typing in it (less than the around 700 character threshold) and save it, the file will go in the $MFT, if you add characters/bytes to it and then save, the file will "migrate" to the filesystem cluster (BTW what was written before the first save will remain in the $MFT and be duplicated in the cluster).

 

Most probably the WOF compression does some kind of "in place" compression, so since the file before compression was on a cluster, it remained on the cluster after compression, even if it could fit inside the $MFT.

 

An experiment would be to see what happens if the smallest files are copied (to another fileystem) and back.

 

Likely something may change on 4K sector disks, as the threshold goes up to around 3700 bytes.

 

:duff:

Wonko



#6 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 02 October 2020 - 08:15 AM

I selected the smallest file for testing 131 bytes and 4 k size on disk (NOT on $MFT), but when copied to D: its size on disk is 0, wich means it is on $MFT now.

 

amd64_microsoft-windows-a..core-base.resources_31bf3856ad364e35_10.0.18362.1_en-us_3c88e48e710b3888.manifest

 

alacran

Attached Files


  • wimb likes this

#7 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 02 October 2020 - 08:33 AM

I selected the smallest file for testing 131 bytes and 4 k size on disk (NOT on $MFT), but when copied to D: its size on disk is 0, wich means it is on $MFT now.

 

amd64_microsoft-windows-a..core-base.resources_31bf3856ad364e35_10.0.18362.1_en-us_3c88e48e710b3888.manifest

 

alacran

Good, and what happens when it is copied back? (of course after having deleted the "original")

 

:duff:

Wonko



#8 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 02 October 2020 - 04:53 PM

Booting from Win10XPE_x64_LZX.vhd, mounted 10x64-2004-LZX.vhd  to H:

 

I wanted to go further and copied the folder H:\Windows\WinSxS\Manifests to D:, deleted the original and then copied back the folder to original location.  Now a lot of files are 0 size on disk ( = on $MFT), also I was able to find the aproximate size limits for the transition:

  • If file size is 462 bytes or smaller it is 0 size on disk = located on $MFT
  • If file is 471 bytes or bigger it is 4,096 size on disk = located outside $MFT

Of course as we know it depends also of file name length.

 

EDIT: On Transition value picture just made, the same VHD is mounted on G:, I'm sure about the case on 462 bytes, but not shown on the picture

 

alacran

Attached Files


  • wimb likes this

#9 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 02 October 2020 - 06:29 PM

Good :), and - at the end of the day - how many files are now entirely in the $MFT?

 

If they are - say - half of the  17,408 you listed, let's say 8,500 rounded down, that would mean 8.500*4.096=34,816,000 bytes saved.

 

The appreciation is - like beauty - in the eye of the beholder, is that a pile of 24 floppy disks or an insignificant fraction of a stupidly large 1 TB disk? :dubbio:

 

:duff:

Wonko


  • wimb and antonino61 like this

#10 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 02 October 2020 - 08:44 PM


Of course as we know it depends also of file name length.

 

EDIT: On Transition value picture just made, the same VHD is mounted on G:, I'm sure about the case on 462 bytes, but not shown on the picture

 

alacran

 

And opening the unmounted VHD with 7-zip, for all files on $MFT Compressed size is same as file size, but if the compressed size is 4,096 or bigger it means they are outside the $MFT.

 

Also we can see in attached picture some files with same 462 bytes or smaller size, stored outside of %MFT, just because their names are longer.

 

alacran

Attached Files



#11 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 03 October 2020 - 01:00 AM

so what is the situation of the files at the end of the day? ?se llega a cierta conclusión o no se llega a conclusión ninguna? A line must be drawn between failure and success.



#12 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 03 October 2020 - 08:13 AM

so what is the situation of the files at the end of the day? ?se llega a cierta conclusion o no se llega a conclusión ninguna? A line must be drawn between failure and success.

Which line?

The experiment is of course (as expected BTW) a success, the point is only on the usefulness (or not) of implementing it "in production", and this - as said - is (and can only be) in the eye of the beholder.

 

As I see it, there is an advantage in having resident files (size and speed of access), but this advantage is unlikely to be noticeable.

 

When (if) the devices will be "native" 4K sectored, the advantage may become noticeable, because it will apply to a much wider number of files, though the saving will be proportionally smaller.

 

I.e.

with 512 bytes sector a file up to around 460 bytes (thanks to the stupidly long names, could have been more than 700) achieves a saving of 4096 bytes, i.e. a "compression ratio" of  (1024+4096 vs. 1024) 1024/5120=20%

with 4K bytes sector a file up to around 3400 bytes (still thanks to the stupidly long names, could have been more than 3700) achieves a saving of 4096 bytes. i.e. a much lower "compression ratio" of (4096+4096 vs. 4096) 4096/8192=50% BUT it will apply to something like double the number of files.

 

 

:duff:

Wonko


  • antonino61 likes this

#13 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 03 October 2020 - 09:43 AM

math proves u right, as usual. the "discourse" raises a few layman's questions: can we reduce the number of letters in filenames and use a link fixer to make it all go back right again? will it be 4k? considering the power of modern hardware and the slightest hindrance it suffers from compression (together with the advantages that may accrue), can we go towards default-compression or no-compression-at-all (because the minimum space possible is already occupied) scenarios?

 

u rightly mentioned production - a few irrefutable points have been made de facto through the years:

1) for powerful hardware, the smaller the better; -->

2) --> albeit fractioned (wim-vhd), wimboot is no longer a curiosity; it can be used in production as well, as a dynamically stable system (even more so if duly compressed (see alacrán's area); -->

3) --> it could rightly be so also for 90% of unaware bloated-windows users, who would greatly benefit from the flying dutchman's applications;

4) for those who do not like combos, a duly compressed full vhd will do the same job in a slightly faster fashion (I say slightly because I have a few tests whereby some wim+vhd combos fare no worse (or sometimes even better), than its full-vhd counterparts); anyway see alacrán for further and wider feedback;

5) the brute-force-shrinking method is ultimately proving the most effective in terms of speed, not so in terms of saved space, which probably goes back to the point wonko and alacrán were initially making here (some limitation in free-space gaining as a result of limited used space shrinking).



#14 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 03 October 2020 - 11:48 AM

BTW (and a a side-side note) it is perfectly possible (at least it is possible on XP, probably on 7, cannot say on 10) to create a virtual disk that is "native" 4 K sectored using MS own virtual disk driver:

http://reboot.pro/to...storage-driver/

 

 

Once the NTFS volume is created on it (with 4096 bytes for each $MFT entry) the filesystem can be deployed to 512 bytes sector devices just fine with minimal "fixes", the experiment was made (and it is more complicated than what would be needed for a "simpler" 4096 bytes per $MFT entry) and worked just fine:

https://msfn.org/boa...ent-interfaces/

https://msfn.org/boa...ent-interfaces/

 

Point is that growing the $MFT entry to 4096 from 1024 (unless it is needed because of the 4096 byte/sector) is unlikely to provide any advantage.

 

:duff:

Wonko



#15 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 03 October 2020 - 12:31 PM

on some site or other I found set-dosfilename.ps1 and a ps instruction which I adapted in the following fashion:

.\Set-DosFileName.ps1 -folder "c:\windows\winsxs" -recurse

it does not seem to have done anything, though.



#16 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 03 October 2020 - 05:33 PM

so what is the situation of the files at the end of the day? ?se llega a cierta conclusión o no se llega a conclusión ninguna? A line must be drawn between failure and success.

 

In accordance with topic title all the experiments were successful.

  • On any NTFS drive we can use SwiftSearch to check which files are stored on $MFT
  • On unmounted VHDs we can also use 7-zip to check which files are stored on $MFT
  • If the original folder Windows\WinSxS\Manifests is copied to another drive, then delete the original, and latter copied back, almost all files having about 460 bytes or smaller size will be now stored on $MFT, saving a good amount of used size on a NTFS drive. (Unless they are latter modified).
  • Saved espace will be a cluster (4 K or 4,096 bytes on NTFS) for each file relocated to $MFT

If all or any part of this info is useful for you or not, it's something you have to decide by yourself.

alacran



#17 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 03 October 2020 - 10:21 PM

i find it very useful


  • alacran likes this

#18 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 04 October 2020 - 10:05 AM

on some site or other I found set-dosfilename.ps1 and a ps instruction which I adapted in the following fashion:

.\Set-DosFileName.ps1 -folder "c:\windows\winsxs" -recurse

it does not seem to have done anything, though.

 

My crystal ball is telling me that your source is this:

https://domandesunix...iati-windows-10

which is a machine translated copycat of:

https://superuser.co...ames-windows-10

 

where a largely clueless user and the original responder fell for the xyz fallacy:

https://jdebp.eu/FGA...red-banana.html

to be rescued (luckily) by MagicAndre1981  :thumbsup: .

 

Now, would have it worked, you would have all file in \winsxs renamed to their 8.3 "DOS" names.

 

Would the *whatever* part of the OS is looking for them use the full names or the 8.3 ones? :dubbio:

 

Maybe, just maybe, you could create (an inane amount of) hard-links with the current names to newly created shorter 8.3 named files. :unsure:

 

This way you could increase the number of files stored in the $MFT to include all those up to  720 or maybe 728 bytes, BUT doubling the $MFT entries.

 

So that should be done only for the files that do not already fit in the $MFT AND that are below 720 or maybe 728 bytes.

 

:duff:

Wonko



#19 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 04 October 2020 - 02:59 PM

Is that a time-consuming procedure?

#20 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 04 October 2020 - 03:21 PM

Is that a time-consuming procedure?

It depends.

 

I would say it is faster than both a sheep on a trampoline[1] and a sloth chase[2] but slower than a cupboard cat attack [3]

:duff:

Wonko

 

[1] 

[2] 

[3] https://www.youtube.com/watch?v=QefxGauB3_c






1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users