Jump to content











Photo

Let's talk dirty (bit)


  • Please log in to reply
24 replies to this topic

#1 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 19 January 2013 - 08:07 PM

I know, I know, it does seem - besides queerly titled - "out of place".

It is not, rest assured.

 

Let's call it a "spin-off" from this thread/posts:

http://reboot.pro/to...ting-to-isosel/

http://reboot.pro/to...sosel/?p=165006

 

Where (the HECK) is the "dirty bit"? :unsure:

Is it a bit or a byte? :dubbio:

 

From the mouth of the wolf:

http://support.micro...kb/322275/en-us

The "dirty" bit is a bit in the boot sector (for FAT or FAT32 volumes), or in the MFT (for NTFS volumes), that is checked when Windows starts. This bit is checked to determine whether corruption has been detected in the file system. If the dirty bit is set on a volume, Autochk runs to correct any errors on the volume.

 

I strongly suspect that - as always - MS info is either misleading or wrong (or BOTH).

 

Here:

http://www.raymond.c...without-chkdsk/

the "bit" (that in the meantime has misteriously grown to a WHOLE byte :w00t: ) is clear enough for NTFS (the actual filesystem we are NOT interested in for matters related to ISOSEL/Isostick)

 

It is far less clear (and I suspect outright wrong :ph34r:) for FAT32 (whilst FAT16 is not analyzed at all).

Anyway the identified byte (not bit), confusingly said as being "41" without specifiying if it is hex or dec offset in the bootsector, may correspond BOTH if hex or if dec (i.e. hex 29) to reserved bytes/part of reserved word according to the ONLY source I do actually trust, the Starman's pages:

http://thestarman.na...mbr/MSWIN41.htm

 

Quite surprisingly some apparently more reasonable info can be gathered by reading some Apple :w00t: source code:

http://www.opensourc...dos.tproj/fat.c

 

 

 





/*
* Determine whether a volume is dirty, without reading the entire FAT.
*/
int isdirty(int fs, struct bootblock *boot, int fat)
{
int result;
u_char *buffer;
off_t offset;

result = 1; /* In case of error, assume volume was dirty */

/* FAT12 volumes don't have a "clean" bit, so always assume they are dirty */
if (boot->ClustMask == CLUST12_MASK)
return 1;

buffer = malloc(boot->BytesPerSec);
if (buffer == NULL) {
perr("No space for FAT sector");
return 1; /* Assume it was dirty */
}

offset = boot->ResSectors + fat * boot->FATsecs;
offset *= boot->BytesPerSec;

if (lseek(fs, offset, SEEK_SET) != offset) {
perr("Unable to read FAT");
goto ERROR;
}

if (deblock_read(fs, buffer, boot->BytesPerSec) != boot->BytesPerSec) {
perr("Unable to read FAT");
goto ERROR;
}

switch (boot->ClustMask) {
case CLUST32_MASK:
/* FAT32 uses bit 27 of FAT[1] */
if ((buffer[7] & 0x08) != 0)
result = 0; /* It's clean */
break;
case CLUST16_MASK:
/* FAT16 uses bit 15 of FAT[1] */
if ((buffer[3] & 0x80) != 0)
result = 0; /* It's clean */
break;
}

and 

 

 








/*
* Check the second (index 1) entry of the FAT. It should be set to an
* end of file mark. For FAT16 and FAT32, the upper two bits may be cleared
* to indicate "volume dirty" and "hard errors"; for FAT12, it must always
* be an EOF value.
*
* Also check the "clean" bit. If it is not set (i.e., the volume is dirty),
* set the FSDIRTY status.
*/
value = fat_get(1);
if (value == CLUST_ERROR)
return FSFATAL;
switch (boot->ClustMask)
{
case CLUST16_MASK:
if ((value & 0x8000) == 0)
mod |= FSDIRTY;
break;
case CLUST32_MASK:
if ((value & 0x08000000) == 0)
mod |= FSDIRTY;
break;
default:
break;
}
/*
* Figure out how many bits of the FAT[1] entry to compare. FAT16 and
* FAT32 use the upper two bits as flags, so we don't want to compare
* them. The >>2 part below leaves temp set to the cluster mask, but
* with the upper two bits zeroed. (Note that the cluster mask has
* the lower N bits set.) FAT12 has no flags, and uses all 12 bits.
*

Please note how the byte has suddenly reverted to a smaller bit BUT moved from the bootsector to the FAT.

Additionally it seems like TWO bits are involved :frusty:

 

It would be much appreciated if some of the peeps with more C reading capabilities than myself could check the above source code and possibly also check the corresponding Freedos and Linux ones.

 

The "good" MS originated reference for FAT is if course NOT the half-@§§ed above mentioned KB but this document:

http://staff.washing...c/fatgen103.pdf

which actually (page 18) does re-establish:

 

 

 

For FAT16 and
FAT32, the file system driver may use the high two bits of the FAT[1] entry for dirty volume flags (all
other bits, are always left set to 1). Note that the bit location is different for FAT16 and FAT32,
because they are the high 2 bits of the entry.

For FAT16:
ClnShutBitMask = 0x8000;
HrdErrBitMask = 0x4000;

For FAT32:
ClnShutBitMask = 0x08000000;
HrdErrBitMask = 0x04000000;

Bit ClnShutBitMask

– If bit is 1, volume is “clean”.
If bit is 0, volume is “dirty”. This indicates that the file system driver did not
Dismount the volume properly the last time it had the volume mounted. It
would be a good idea to run a Chkdsk/Scandisk disk repair utility on it,
because it may be damaged.
Bit HrdErrBitMask

– If this bit is 1, no disk read/write errors were encountered.
If this bit is 0, the file system driver encountered a disk I/O error on the
Volume the last time it was mounted, which is an indicator that some sectors
may have gone bad on the volume. It would be a good idea to run a
Chkdsk/Scandisk disk repair utility that does surface analysis on it to look
for new bad sectors.

that the bits are TWO and that they are NOT in the bootsector at all, but rather at the beginning of the FAT.

What is not (yet and to me) clear at all is how the second copy of the FAT is affected, and what happens when just one of the two copies have that bit changed. :unsure:

 

Another question is now (reserved to ElegantInvention) WHICH of the two bits (or both) does ISOSEL check? Only the "first one", i.e. the "real" volume dirty one? Or both?

 

And another one (opened to everyone, but particularly aimed at bblaauw that originally had the idea to run a DOS to change ISO_FILENAME.TXT AND owns an actual Isostick AND usually knows where his towel is :thumbsup:):

How about testing the same ISO_FILENAME.TXT writing in grub4dos? (and if needed settign the stupid bit(s) programmatically from grub4dos)

 

:cheers:

Wonko



#2 steve6375

steve6375

    Platinum Member

  • Developer
  • 7566 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films
  •  
    United Kingdom

Posted 19 January 2013 - 09:58 PM

There are two dirty bits (maybe more).

 

This is my understanding:

 

One of them is set when a volume is mounted by the OS. If you pull the power, then when you reboot, the bit will be set and chkdsk will want to run to check for a damaged FS. This is also why if you try to clone a mounted volume, when you boot the cloned drive, the OS will want to check the FS.

 

The other dirty bit is set (bit=0) when a read or write cluster error is encountered on that volume. This is not cleared on a clean shutdown and will remain until cleared (bit=1) by chkdsk or format or some other means.

 

re, FAT[1] and FAT[2]  - only one of these is 'active' and so only one will be marked as dirty. The other non-active copy is used for repair/recovery.

 

Many moons ago I wrote some code to check these bits (from DOS, as obviously the 'vol mounted' dirty bit will always be set whilst running Windows).

fsutil dirty  query c:    returns  'not dirty' under Windows - I suspect this is reporting the 'bad read/write' dirty bit state, not the 'volume not dismounted cleanly' dirty bit. This is why fsutil won't let you clear it.

 

From the Apple code you cited - this checks the bad-shutdown bit...

 

If bit=1 then volume is clean...

 FAT32 uses bit 27 of FAT[1] 2nd cluster entry*/ if ((buffer[7] & 0x08) != 0) result = 0; /* It's clean */ break; 

So if bit 3 of byte 7  (8th byte) in the FAT[1] cluster map is reset, then it is dirty and was not shut down cleanly

 FAT16 uses bit 15 of FAT[1] 2nd cluster entry*/ if ((buffer[3] & 0x80) != 0) result = 0; /* It's clean */ break;

So if bit 7 of the 4th byte in the FAT[1] cluster map is reset, it is dirty,

 

FAT spec says:

 

For FAT16:

ClnShutBitMask = 0x8000
HrdErrBitMask = 0x4000
 
For FAT32:
ClnShutBitMask = 0x08000000
HrdErrBitMask = 0x04000000

 


  • Motasem likes this

#3 DarkPhoeniX

DarkPhoeniX

    Frequent Member

  • Team Reboot
  • 452 posts
  • Location:In the middle of nowhere
  • Interests:Interesting Things
  •  
    South Africa

Posted 20 January 2013 - 01:44 AM

Over my head as Always:

https://i.chzbgr.com...4272/hFD044A27/



#4 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 20 January 2013 - 11:05 AM

re, FAT[1] and FAT[2]  - only one of these is 'active' and so only one will be marked as dirty. The other non-active copy is used for repair/recovery.

Good :) and HOW that (the one that is active) will be flagged and where exactly?
And how exactly the bits are set?
Byte would be 28h or 40 dec on FAT32:
http://thestarman.na...mbr/MSWIN41.htm
Bits 0-3 usage is clear.
Use of bit 7 not so much.
BUT the fatgen103 (page 12):

 

BPB_ExtFlags 40 2 This field is only defined for FAT32 media and does not exist on FAT12 and FAT16 media.
Bits 0-3 -- Zero-based number of active FAT. Only valid if mirroring is disabled.
Bits 4-6 -- Reserved.
Bit 7 -- 0 means the FAT is mirrored at runtime into all FATs.
-- 1 means only one FAT is active; it is the one referenced
in bits 0-3.
Bits 8-15 -- Reserved.

 

:unsure:

So it seems that the default in FAT12/16 is 2 FATs (of course) and if anything but 1 is set FAT mirroring is always active (i.e. seemingly there is no "active" FAT) and on FAT32 where this possibility is present it simply never happens as FAT mirroring is always "ON".

I.e. it seems to me that you can have either of 1 or 2 FAT's and if they are 2 then mirroring is normally always "ON" on both FAT16 and FAT32, but the letter has the possibility to have multiple FAT's AND disable FAT mirroring (i.e. making a single FAT "active").

The original doubt:

 

 

What is not (yet and to me) clear at all is how the second copy of the FAT is affected, and what happens when just one of the two copies have that bit changed.

 

 remains NOT cleared. :unsure:

 
 

 
Many moons ago I wrote some code to check these bits (from DOS, as obviously the 'vol mounted' dirty bit will always be set whilst running Windows).


fsutil dirty  query c:    returns  'not dirty' under Windows - I suspect this is reporting the 'bad read/write' dirty bit state, not the 'volume not dismounted cleanly' dirty bit. This is why fsutil won't let you clear it.

Well, no. :w00t:
The actually given kb says that fsutil simply doesn't apply to FAT or "non-NTFS" volumes:
http://support.micro...kb/322275/en-us
I would have thought that at least this part of it was correct.

From the Apple code you cited - this checks the bad-shutdown bit...

Yep. :thumbsup:

:cheers:
Wonko



#5 steve6375

steve6375

    Platinum Member

  • Developer
  • 7566 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films
  •  
    United Kingdom

Posted 20 January 2013 - 12:40 PM

OK - just tried some FAT32 USB pen tests under Win7.

 

In the first sector of the volume (BPB) - e.g. sector 63, if  byte 41h is set to 01h (instead of 00h), then fsutil dirty query says the volume is dirty. If I eject it and replug it, Win7 says it wants to run checkdisk. After running chkdsk, byte 41h is set to 0.  Note also that if I set bytes 4,5,6&7 in the FAT[1] to 00,00,00,00 then after chkdsk has scanned the volume, it changes these bytes to 00,00,00,08.

However, just setting the FAT[1] 2nd entry bytes 4-7 to 00,00,00,00 in itself does NOT prompt chkdsk to run and fsutil says it is clean. It seems only setting byte 41h to 01 causes the OS to see it as dirty and that chkdsk does mark a volume as clean by setting bit 27 in the 2nd FAT cluster entry (00,00,00,08) -  it is FF,FF,FF,FF on normal systems (first entry is F8,FF,FF,FF).

 

If I set byte 41h to FEh, fsutil sys it is NOT dirty and on replug I get no prompt to scan it.



#6 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 20 January 2013 - 02:33 PM

OK - just tried some FAT32 USB pen tests under Win7.
 
In the first sector of the volume (BPB) - e.g. sector 63, if  byte 41h is set to 01h (instead of 00h), then fsutil dirty query says the volume is dirty. If I eject it and replug it, Win7 says it wants to run checkdisk. After running chkdsk, byte 41h is set to 0.  
So the report by Raymond:

 http://www.raymond.c...without-chkdsk/
 
 is correct (and as expected the MS KB is BOTH misleading AND wrong).


 
Note also that if I set bytes 4,5,6&7 in the FAT[1] to 00,00,00,00 then after chkdsk has scanned the volume, it changes these bytes to 00,00,00,08.
However, just setting the FAT[1] 2nd entry bytes 4-7 to 00,00,00,00 in itself does NOT prompt chkdsk to run and fsutil says it is clean. It seems only setting byte 41h to 01 causes the OS to see it as dirty and that chkdsk does mark a volume as clean by setting bit 27 in the 2nd FAT cluster entry (00,00,00,08) -  it is FF,FF,FF,FF on normal systems (first entry is F8,FF,FF,FF).
Yes/no. :w00t:
WHICH values had the original (before you zero them) bytes "FAT[1] 2nd entry bytes 4-7"?
WHAT exactly do you mean by the above?
We are talking of FAT32, so each FAT entry is 4 bytes long, there is no such thing as "FAT[1] 2nd entry bytes 4-7 to 00,00,00,00" it can be either "FAT[1] bytes 4-7" OR "FAT[1] 2nd entry" (I know I am picky ;), but I feel like there is already enough confusion/misunderstandings/misnamings on this topic).
A FAT32 2nd entry set as 00,00,00,08 seems making to me little sense :dubbio: when freshly formatted it would be FF,FF,FF,FF (if a Label is given when formatting)  or FF,FF,FF,0F if no label is specified, and third entry would be FF,FF,FF,0F as well both with label or without.

 
If I set byte 41h to FEh, fsutil sys it is NOT dirty and on replug I get no prompt to scan it.
I am not sure to understand (actually I am sure I don't understand) the scope of this test, can you expand on this?

:cheers:
Wonko

#7 steve6375

steve6375

    Platinum Member

  • Developer
  • 7566 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films
  •  
    United Kingdom

Posted 20 January 2013 - 02:43 PM

The first two cluster entries in the FAT table are

 

F8 FF FF FF   (the first 4 byte 32-entry), followed by FF FF FF FF (the 2nd 32-bit entry).  The last byte (offset 7) thus has 8 bits, the bit that is the dirty bit is  0000 1000b or out of the 32-bits, bit 27.  I set all 4 bytes to zero and then ran chkdsk and it then set the last byte to 08 - thus proving that to set the dirty bit to 'clean' it is setting bit 27 of the 2nd cluster table entry to 1.

 

re.

 

If I set byte 41h to FEh, fsutil sys it is NOT dirty and on replug I get no prompt to scan it.

 

By setting all bits except bit 0 to 1's, and running fsutil, I am proving that fsutil is not looking at any of the other bits in the byte - i.e. there is not a 'read/write bad bit' anywhere in the same byte.

 

[Edited to correct typo]



#8 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 20 January 2013 - 04:01 PM

The first two cluster entries in the FAT table are
 
F8 FF FF FF FF   (the first 32-entry), followed by FF FF FF FF (the 2nd 32-bit entry).  

NO, at least not here (XP).
The second entry is FF FF FF FF IF a volume label has been applied when formatting.
IF no label has been specified, second entry is FF FF FF F0
 
 

 By setting all bits except bit 0 to 1's, and running fsutil, I am proving that fsutil is not looking at any of the other bits in the byte - i.e. there is not a 'read/write bad bit' anywhere in the same byte.
 

I see (or maybe I don't) :w00t:.
IF the fsutil code is actually looking for a bit, you are right :thumbsup:, but if we assume that the MS kb is fully or partially deceiving, and fsutil simply (and stupidly) checks for the WHOLE BYTE as being 01 if dirty, you have proved very little (seemingly you have just halved possibilities).
How many byte values will have bit 0 set to 1? :unsure:
Exactly (255+1)/2=128 i.e. any "odd" number in the range 0÷255.
If you try fsutil query on all these 128 values what happens? :dubbio:
In other words I suspect that while *something else* (and this something else being possibly very different on different OS) at booting (or at "mounting volume") time may check a FAT32 filesystem's bootsector for the value of bit 0 of byte 41h of the bootsector and consequently initiate Autochk/CHKDISK, the fsutil query function is simply "botched" (and this would justify the existence of the MS KB cited). 

:cheers:
Wonko



#9 steve6375

steve6375

    Platinum Member

  • Developer
  • 7566 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films
  •  
    United Kingdom

Posted 20 January 2013 - 04:53 PM

Sorry, i made  a typo

F8 FF FF FF FF   (the first 32-entry), followed by FF FF FF FF (the 2nd 32-bit entry).  The last byte (offset 7) thus has 8 bits, the bit that is the dirty bit is  0000 1000b or out of the 32-bits, bit 27.  I set all 4 bytes to zero and then ran chkdsk and it then set the last byte to 08 - thus proving that to set the dirty bit to 'clean' it is setting bit 27 of the 2nd cluster table entry to 1.

should read

F8 FF FF FF   (the first 32-entry), followed by FF FF FF FF (the 2nd 32-bit entry).  The last byte (offset 7) thus has 8 bits, the bit that is the dirty bit is  0000 1000b or out of the 32-bits, bit 27.  I set all 4 bytes to zero and then ran chkdsk and it then set the last byte to 08 - thus proving that to set the dirty bit to 'clean' it is setting bit 27 of the 2nd cluster table entry to 1.

 

 

32-bit entries are 4 bytes each!

1st entry = F8 FF FF FF

2nd entry = FF FF FF FF

 

good reference here



#10 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 20 January 2013 - 05:05 PM

The mechanism of the final 08 was clear to me (after your previous post explaining) notwithstanding the small typo :thumbsup:, but thanks for noticing and correcting it :).

 

BUT, again, second entry (at least here) is NOT FF FF FF FF in all cases. (it is IF volume label exists).

 

 

 

jaclaz



#11 steve6375

steve6375

    Platinum Member

  • Developer
  • 7566 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films
  •  
    United Kingdom

Posted 20 January 2013 - 05:06 PM

For byte 41h, I tried values of 01, 03, 09 and 9F all cause fsutil to report it as dirty and all cause chkdsk to be run when replugged. I think it is pretty clear that fsutil and the OS check just the single bit in the byte!



#12 steve6375

steve6375

    Platinum Member

  • Developer
  • 7566 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films
  •  
    United Kingdom

Posted 20 January 2013 - 05:18 PM

The mechanism of the final 08 was clear to me (after your previous post explaining) notwithstanding the small typo :thumbsup:, but thanks for noticing and correcting it :).

 

BUT, again, second entry (at least here) is NOT FF FF FF FF in all cases. (it is IF volume label exists).

 

 

 

jaclaz

The link I gave above says:

 

 

The second entry (cluster 1 in the FAT) nominally stores the end-of-cluster-chain marker as used by the formater, but typically always holds 0xFFF / 0xFFFF / 0x0FFFFFFF, that is, with the exception of bits 31-28 on FAT32 volumes these bits are normally always set. Some Microsoft operating systems, however, set these bits if the volume is not the volume holding the running operating system (that is, use 0xFFFFFFFF instead of 0x0FFFFFFF here).[63] (In conjunction with alternative end-of-chain markers the lowest bits 2-0 can become zero for the lowest allowed end-of-chain marker 0xFF8 / 0xFFF8 / 0x?FFFFFF8. Some operating systems may not be able to mount some volumes if any of these bits are not set, therefore the default end-of-chain marker should not be changed.)

Nothing about volume label???

 

Also see here - again no mention of volume label affecting it. Can you post a real example hex sector output of the 1st 16 bytes  of a non- FF FF FF FF  volume?

 

Sure you don't mean the bytes were FF FF FF 0F  for the OS boot volume which fits with StarMans observations?



#13 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 20 January 2013 - 07:39 PM

The mistery deepens :w00t:

Tests made (these should be easily reproducible):
1.created a sparse file mksparse.exe 64mFat32.img 64M
2.mounted it with IMDISK to a drive
3.formatted as FAT32 from within the IMDISK control panel (WITHOUT giving a label)
4.opened the device with tinyhexer, searched for F8FFFF landed (after a couple false positives) to sector 32, DATA:F8 FF FF 0F FF FF FF 0F FF FF FF 0F
5.re-formatted as FAT32 from within the IMDISK control panel (this time ASSIGNING a label)
6.opened the device with tinyhexer, as above landed to sector 32, DATA:F8 FF FF 0F FF FF FF FF FF FF FF 0F
7.unmonted the IMDISK drive
8.mounted the same image in a QEMU VM, booting from floppy
9.used DOS 7.1 FDISK to create a single, NON active primary partition covering 100%, shut down VM
10.mounted it with IMDISK to a drive
11.formatted as FAT32 from within the IMDISK control panel (WITHOUT giving a label)
12.opened the device with tinyhexer, as above landed to sector 32 (of the volume), DATA:F8 FF FF 0F FF FF FF 0F FF FF FF 0F
13.re-formatted as FAT32 from within the IMDISK control panel (this time ASSIGNING a label)
14.opened the device with tinyhexer, as above landed to sector 32, DATA:F8 FF FF 0F FF FF FF FF FF FF FF 0F
15.unmonted the IMDISK drive
16.wrote F8 FF FF FF FF FF FF FF FF FF FF FF to BOTH FAT tables
17.mounted it with IMDISK to a drive
18.re-formatted as FAT32 from within the IMDISK control panel (WITHOUT giving a label)
19.opened the device with tinyhexer, as above landed to sector 32 (of the volume), DATA:F8 FF FF 0F FF FF FF 0F FF FF FF 0F
20.unmonted the IMDISK drive
21.mounted in QEMU VM
22.formatted from DOS 7.1 (NO label) DATA:F8 FF FF 0F FF FF FF 0F FF FF FF 0F
23.formatted from DOS 7.1 /S (NO label) DATA:F8 FF FF 0F FF FF FF 0F FF FF FF 0F
24.formatted from DOS 7.1 (WITH label) DATA:F8 FF FF 0F FF FF FF 0F FF FF FF 0F
25.formatted from DOS 7.1 /S (WITH label) DATA:F8 FF FF 0F FF FF FF 0F FF FF FF 0F
26.same tests with a 520 Mb image, same results
26.mounted it with IMDISK to a drive
27.re-formatted as FAT32 from within the IMDISK control panel (ASSIGNING a label)
28.opened the device with tinyhexer, DATA:F8 FF FF 0F FF FF FF FF FF FF FF 0F
:frusty:

I have to think about some other (smarter and possibly automated) way to test. :dubbio:

:cheers:
Wonko

#14 elegantinvention

elegantinvention

    Frequent Member

  • Developer
  • 310 posts
  • Location:South Bend, Indiana, USA
  •  
    United States

Posted 23 January 2013 - 08:04 PM

Wow, long topic is long. I would have replied sooner but it took a while to find the time to read all this, and it kept growing during that time  :white_flag:

 

Another question is now (reserved to ElegantInvention) WHICH of the two bits (or both) does ISOSEL check? Only the "first one", i.e. the "real" volume dirty one? Or both?

Sorry if I missed anything in the above discussion, but here's how isostick does this:
As this topic has clearly established, the information on the dirty bit(s)/byte(s) is, well, inconsistent  :fool:

 

Failing to locate an official source outside of what little fatgen103 says, I fired up some debug firmware and watched which bits were written while modifying files on the drive, copying files to it, etc, under Windows 7. The lowest bit of byte 41h (65 decimal) in the FAT BPB would always be set to 1 prior to modifying the volume, indicating an operation is taking place (it's dirty). For copying files, as soon as the copy was complete Windows would immediately clear the bit back to 0. For modifying file contents, it would wait for several seconds of inactivity before clearing it. I did not test setting other bits in that byte to see how Windows handles it, but it definitely sets and clears only that bit when it is modifying the volume.

I can confirm that if that bit is 1, Windows will want to chkdsk, and it should be obvious why.

 

However, fatgen103 just says that is reserved for NT. How useful.

 

Admittedly (shamefully! :doh7: ) that is the extent to which isostick checks for a dirty volume. It uses that bit to determine if it's safe to try opening a file, though I am working on making the FAT32 code more error-resistant so it can read a bad volume without crashing.

 

The fun comes when you watch that bit on non-Windows OSes. MacOS X and Linux seem to set that bit when mounting and leave it set until the user unmounts the volume. This makes it difficult for isostick to know when it's safe to check the contents of iso_filename.txt or read isosel.bin, hence the limitations.

 

I keep forgetting to look into that FAT[1] cluster entry...  :dubbio:



#15 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 24 January 2013 - 10:32 AM

Failing to locate an official source outside of what little fatgen103 says, I fired up some debug firmware and watched which bits were written while modifying files on the drive, copying files to it, etc, under Windows 7. The lowest bit of byte 41h (65 decimal) in the FAT BPB would always be set to 1 prior to modifying the volume, indicating an operation is taking place (it's dirty). For copying files, as soon as the copy was complete Windows would immediately clear the bit back to 0. For modifying file contents, it would wait for several seconds of inactivity before clearing it. I did not test setting other bits in that byte to see how Windows handles it, but it definitely sets and clears only that bit when it is modifying the volume.

I can confirm that if that bit is 1, Windows will want to chkdsk, and it should be obvious why.

 

Very good. :)

While I doubt NOT that the lowest bit is affected, the point I was trying to highlight is that the source I found (unofficial) observed the SAME EXACT behaviour but described it differently:

  • if byte 41h (not bit) is always 0 and when it is "dirty" it's lowest bit is set to 1, it equates to say that the dirty "flag" is byte 41h with values 0=clean 1=dirty and Raymond's description is correct.

Just for the fun of it ;), I would insert another definition (which should be *as good*):

  • if value of byte 41h is even, volume is clean, if it is odd, it is dirty.

 

The description based on bit value is probably more technically coorect, of course.

 

Still the other "misteries" coming out from the cross observations of Steve6375 and myself need to be explained (or explained fully) :(.

 

The basic original question:

http://reboot.pro/to...sosel/?p=165006

 is now solved :thumbsup:, though and bblauw (if interested) can try setting the bit (or byte) through grub4dos.

 

(and still some checks on the FreeDOS source for FAT32 handling would be nice :whistling:)

 

:cheers:

Wonko



#16 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 24 January 2013 - 10:49 AM

In MultiBoot configurations Windows 8 quite often will set the NTFS dirty bit On for ALL local drives. :ph34r:

Next boot then chkdsk will run on ALL local drives.

 

Setting of the NTFS dirty bit can be prevented with this registry tweak to Disable Hiberboot

 

Windows Registry Editor Version 5.00

 

; Windows 8 Disable Hiberboot to prevent NTFS dirty bit in case of MultiBoot causing running of chkdsk

 

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Power]
"HiberbootEnabled"=dword:00000000

 

 

More Info here

 

http://www.trickday....t-in-windows-8/

http://www.msfn.org/...38#entry1016938

 

:cheers:



#17 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 24 January 2013 - 02:19 PM

Setting of the NTFS dirty bit can be prevented with this registry tweak to Disable Hiberboot

Yep, but not does not clear if it is a bit or a byte, nor does it clear effectively where it (be it a bit or a byte) it is located.

The cited article proposes something that is far from being easily replicable.

 

The byte (or bit) in question is set to 1 BOTH in the $MFT and in $MFTMirror.

It is NOT in a fixed position, though  it is definitely inside the fourth record:

1. $MFT

2. $MFTMirror

3. $LogFile

4. $VolumeP

but it's position in it is variable, it depends - as an example - on whether the Volume has a Label or not (or on how long is the Label)

Maybe a "better" way to find it is to look in the fourth record for FFFF FFFF and go back 32 bytes, then you will have either of 030100 (clean) or 030101 (dirty).

 

:cheers:

Wonko



#18 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 24 January 2013 - 02:42 PM

I did use WinHex to investigate $Volume as described in your ref

 

http://www.raymond.c...without-chkdsk/

 

and found that the description of Raymond corresponds exactly to my case,

but it might be that the Volume Label has effect.

 

In any case I could change the NTFS dirty bit and solve the chkdsk problem.

 

Quite annoying this behaviour of Windows 8 in MultiBoot environment.

 

The F8 Boot Options of Windows 8 normally cannot be accessed,

but that changes when Grub4dos or XP are added as boot option to Boot Manager menu,

but only when Windows 8 is NOT set as Default

(adjustable via bcdedit Or in Recovery Environment Or Advanced Boot Options).

In that case Win 8 Boot Options can be used as usual with F8.

Also Grub4dos and XP Boot Option only can be used without Reboot,

when Windows 8 Boot entry is NOT set as Default.



#19 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 24 January 2013 - 03:50 PM

and found that the description of Raymond corresponds exactly to my case,

Yes, IF the Volume has no label.

 

but it might be that the Volume Label has effect.

I just posted how it does, no "might", nor doubts about it.

 

Winhex, besides being Commercial has a "facilitated" approach when compared to direct editing (think grub4dos or batch).

Easiest would probably be to get the location of the $MFT from the bootsector (in practice multiply the $MFT LCN by cluster size), then offset of three records (3072 bytes)  and then parse for the FFFF FFFF, verify the presence of 0301 and then set/unset the following byte (or bit ;)).

 

 

:cheers:

Wonko



#20 DumSkidderik

DumSkidderik
  • Members
  • 3 posts
  •  
    Denmark

Posted 28 April 2013 - 10:52 AM

Pardon me for barging in here, but has anyone found a solution to editing iso_filename.txt, using let's say FreeDOS, then set the dirty bit properly allowing you to boot from the ISO afterwards?

I'm trying to workout a solution that allows me to boot from the flash-drive, and set the proper ISO, then boot from the CD-ROM drive of ISOStick.

I've tried various Linux live distros, they do however not see the flash-drive until you remove and re-insert the ISOStick, at which point the Linux distro can no longer find it's files etc. and wont have you edit Jack on the flash drive :(

FreeDOS allows me to edit, but doesn't clear the dirty-bit.

Tried forcing a WinPE install on the flash-drive, but WinPE and FAT32 do not appear to go hand in hand :(



#21 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 28 April 2013 - 12:16 PM

Well, first thing there is not a definite "all around" answer in this thread, so you will need to define which specific bit (more probably byte) you want to set. :unsure:

 

Are we talking of the byte at offest 41h in the bootsector/PBR?

 

Second, why are you wanting to use FreeDOS or a Linux?

Have you tried with the (much simpler) grub4dos? (using the fat add-on to access the file or writing to it with direct access, so that no dirty bit will be set)

Or use the Freedos for the editing of the file and later use grub4dos for just the bit (byte)? (If I recall correctly you can have direct disk access in FreeDOS using DEBUG, which I would define as the least friendly - let alone documented - app ever written for any DOS :ph34r:).

http://www.softpedia...DOS-Debug.shtml

http://thestarman.pc...debug/debug.htm

http://thestarman.pc...ebug/debug2.htm

 

:cheers:

Wonko



#22 DumSkidderik

DumSkidderik
  • Members
  • 3 posts
  •  
    Denmark

Posted 28 April 2013 - 02:31 PM

I dont want to use FreeDOS or Linux in particular, I was just looking for an easy way around the apparantly ISOSEL issues at hand. If I can't depend on it 100% I dont want to use it, then I'd rather have a way to boot to the flash drive, edit txt, reboot to CDROM.

You guys complete lost me on what specific bit we're talking about, I'm not an engineer :)



#23 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 28 April 2013 - 03:20 PM

I dont want to use FreeDOS or Linux in particular, I was just looking for an easy way around the apparantly ISOSEL issues at hand. If I can't depend on it 100% I dont want to use it, then I'd rather have a way to boot to the flash drive, edit txt, reboot to CDROM.

You guys complete lost me on what specific bit we're talking about, I'm not an engineer :)

Yes, maybe it's "our" fault, but if you describe WHAT (EXACTLY) you want to do (or try doing) maybe we can help you.

What it is not clear to me is what exactly you are wanting to do (goal), if you can specify/detail it, I can try to help you with a practical solution (means).

ElegantInvention posted here:

http://reboot.pro/to...y-bit/?p=166490

that his thingy checks lowest bit of byte 0x41 of the PBR, and later I suggested that since normally the whole byte is 0 and when "dirty" still the whole byte is 1, then changing the whole byte is equivalent.

Is that byte that you want to set/unset?

FreeDos may be more convenient to edit the file.

Grub4dos may be more convenient to edit the byte.

Are you willing to try both?

 

:cheers:

Wonko



#24 DumSkidderik

DumSkidderik
  • Members
  • 3 posts
  •  
    Denmark

Posted 29 April 2013 - 06:35 AM

Yes, maybe it's "our" fault, but if you describe WHAT (EXACTLY) you want to do (or try doing) maybe we can help you.

I believe I did :P

The problem I was having, was ISOSEL not working on most of the Pc's I intend to use ISOStick on. I could carry another laptop to plug it into, edit the iso_filename.txt, then plug it back into the PC that required the ISO in the first place, but I'd rather not.

I've found my solution now:

* Format USB Flash with RUFUS

* Install TinyCore Linux on the USB Flash

 

That way I can always boot into Linux (tinycore loads really fast, literally seconds and you're on), change the iso_filename.txt, and then reboot.

 

I was TRYING to do this with FreeDOS earlier, but it left the USB Flash in a dirty state, which in turn Means it wont be read next reboot.

 

I hope this clarifies things



#25 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 29 April 2013 - 09:37 AM

I hope this clarifies things

Yep, I guess everything is fine :).

 

:cheers:

Wonko






1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users