Jump to content











Photo

The mysterious $LogFile in NTFS


  • Please log in to reply
7 replies to this topic

#1 joakim

joakim

    Silver Member

  • Team Reboot
  • 885 posts
  • Location:Bergen
  •  
    Norway

Posted 09 April 2011 - 09:13 PM

For those who missed it, I've recreated boot.sdi for wimboots; http://www.msfn.org/...ize-of-bootsdi/ with a significant size reduction. Now at 300 Kb.

The barrier towards further reduction in size is the $LogFile that can't be smaller than 256 Kb in size, and the 32 Kb $MFT that obviously can't overlap with the logfile. It is likely a limitation in the ntfs driver that will let Windows see a partition as RAW if the logfile is smaller than 256 Kb.

Maybe the most interesting thing about this boot.sdi is that many of the metafiles have shared sectors on the partition (see details below). Don't worry, it's good enough for WinPE to boot on (the actual filesystem is not used, just the wim mounted on top of the partition). The only part of the logfile that is evaluated is the relevant content in $MFT as well as the first 512 bytes of the first restart page (header).

Now, to be able to create logfile with size lower than 2 Mb, we need to patch untfs.dll. I believe Mark Russinovich patched it the same way (although in memory using SoftIce) when he made the ntfs floppy in 97'. There is a good reason why he did not manage create the logfile any smaller. Here's the details of the patch;

At va 5b04cfc5;

[b8 00 00 20 00] -> [b8 00 00 04 00]


At va 5b04d324;

[c7 00 00 00 20 00] -> [c7 00 00 00 04 00]


At va 5b04d336;

[bf 00 00 20 00] -> [bf 00 00 04 00]


And if you also want to try it on a floppy then at va 5b0346fb;

[74 08] -> [eb 08]


The execution can be observed with olly (in my case), by using memory break on access on the dll..

Beware this was done on the XP sp2 version of untfs.dll (5.1.2600.2180). Just make the mod and place the dll together with chkdsk.exe/format.com in a separate directory. In the patch posted above 200000h = 2097152 bytes = 2 Mb, while 40000h = 262144 bytes = 256 Kb. However, the floppy can't be mounted by Windows, and must be investigated by other tools.

If using format.com (or chkdsk /f) it will make the logfile size defaulting as specified in the patch.

It obviously only makes sense to do on very small ntfs partition. After patching the dll you can create logfiles of size 256, 512 , 1024 too. To adjust the logfile size use something like;

chkdsk /l:256 driveletter:


The $Boot file only need to have the first 512 bytes available inside boot.sdi. The rest of it is only necessary if you boot off an actual HDD. Funny enough, MS changed the code inside the file from nt5.x to nt6.x, but it works equally well either way. Meaning the code for ntldr can boot bootmgr, as long as you rename the file (it can also also boot grldr). Basically a bootsector of any of the 3 can boot anyone of the 3 with just a rename of the file.

The partition image inside my boot.sdi is not valid and do not have any free space, so it can only be modified with a hex editor (the $MFT (or whole partition) was more or less completely rewritten by hand). The details of which sectors the individual system(meta)files occupy;

$Boot 0

$MFT 1-65

Root directory 66-73

$LogFile 74-586

$MFTMirr 136-143

$UpCase 136

$Secure:$SDS 137-392

$Bitmap 138

$AttrDef 139-143

The rest of the systemfiles are fully contained within $MFT


So what is it good for? Maybe faster pxe booting (boot.sdi). And a very good way of learning how ntfs works!! Oh, and btw you can free a few Mb on a rambooted disk image (logfile shrinking)..

Btw, any valid NTFS partition of version 3.1 can't be smaller than just below 700 Kb (at least not if used in Windows).

If anyone can create a valid filesystem with a logfile at 128 Kb or smaller I would be extremely interested in the details.

Joakim Schicht
  • Nuno Brito likes this

#2 AceInfinity

AceInfinity

    Frequent Member

  • Team Reboot
  • 222 posts
  • Location:Canada
  • Interests:Windows Security, Programming, Customizing & Crash Dump Analysis.
  •  
    Canada

Posted 01 April 2012 - 02:11 AM

Would you happen to know how it would be possible to enumerate through the Master File Table ($MFT) for directory/file searching purposes? I believe based on what i've read that it's about ~10 times faster than that of most other methods, but my knowledge about $MFT is slim. Lots of people are having issues with scanning based on the MFT records though, and based on certain issues in the code, forced to use less efficient code which would slow down the process to justify using $MFT for searching.

http://www.donationc...php?topic=22695

"StCroixSkipper" seems to be a guy interested in this as well.

#3 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 10,851 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 01 April 2012 - 11:22 AM

Would you happen to know how it would be possible to enumerate through the Master File Table ($MFT) for directory/file searching purposes?

I am not sure to get the question, meet NDFF (command line):
http://z3.invisionfr...hp?showtopic=65
http://ndff.hotbox.ru/en/
and a few apps that do the same (GUI):
http://reboot.pro/6848/
http://www.msfn.org/...alled-exe-file/

:cheers:
Wonko

#4 joakim

joakim

    Silver Member

  • Team Reboot
  • 885 posts
  • Location:Bergen
  •  
    Norway

Posted 01 April 2012 - 04:53 PM

I think Wonko already answered good enough about possibility of using $MFT as a means of searching for files/folders (yes it is possible). But it's not that easy to just create such a utility, as you need to decode the $MFT. Are you interested in that?

The usn is also something interesting, and I tried an attempt with the MS provided apis a few day ago, without much success. Maybe it's easier to decode it yourself, kind of like what this StCroixSkipper did. Thanks for the link.

#5 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 10,851 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 01 April 2012 - 05:10 PM

The nice thing about NDFF is that provides also the ndfftest.exe app.
On the other hand ntfs-search:
http://sourceforge.n...ts/ntfs-search/
makes available the source code, that could be a good resource for people wanting to delve deeper in the matter.

:cheers:
Wonko

#6 AceInfinity

AceInfinity

    Frequent Member

  • Team Reboot
  • 222 posts
  • Location:Canada
  • Interests:Windows Security, Programming, Customizing & Crash Dump Analysis.
  •  
    Canada

Posted 03 April 2012 - 02:16 AM

I'll look into this here, and joakim I also ready about USN through StCroixSkipper's blogs :)
If was also fairly interesting for me

#7 joakim

joakim

    Silver Member

  • Team Reboot
  • 885 posts
  • Location:Bergen
  •  
    Norway

Posted 03 April 2012 - 08:14 AM

Here's a few links to some threads I've been involved in with some interesting information, and that also have links to other resources.

http://www.forensicf...ewtopic/t=8010/
http://www.forensicf...ewtopic/t=8702/

My own project is at; http://code.google.com/p/mft2csv/ and it is fun when you have a working code sample that you can expand on. My code is not at all well written, but it works, and is fun to expand on it. Either way it's a very good way of learning the filesystem internals. For instance, I implemented parts of that code into my setMACE, to show that timestamp manipulation on NTFS can be done by physical disk writing. More interesting stuff can of course be done, but time is just not enough of.. Physical disk writing has its limitations, at least on nt6.x. But reading is not a problem, so information retrieval is only limited to your ability to decode the data. The usn journal and the logfile are certainly 2 interesting parts of the system.

#8 AceInfinity

AceInfinity

    Frequent Member

  • Team Reboot
  • 222 posts
  • Location:Canada
  • Interests:Windows Security, Programming, Customizing & Crash Dump Analysis.
  •  
    Canada

Posted 03 April 2012 - 11:15 PM

Thanks, I appreciate this, I guess it's time to get to my research.