I have modified a version of RMPartUSB (used by RMPrepUSB) so that when formatting as FAT32, the first file cluster will be set to a 1MB boundary location. This may give faster file i/o speeds (may be noticeable on writes??) compared to previous versions.
Here is the output from RMPrepUSB DriveInfo with the new format.
I would guess that some small speed improvement may be noticeable when using small file writes and maybe someone would like to compare this new format to the previous one when doing file i/o operations?
RMPrepUSb DriveInfo for P1 displays:
FAT32
000B Bytes Per Sector = 512 (0200h)
000D Sectors Per Cluster = 8 (08h)
000E Reserved Sectors before first FAT = 1067 (042Bh)
0010 Number of FATs = 2 (02h)
0011 Root Entries = 0 (0000h)
0013 Total Log Sectors (small) = 0 (0000h)
0015 Media Descriptor = 248 (F8h) HDD
0016 Sectors per FAT table = 0 (0000h)
0018 Sectors per Track = 63 (003Fh)
001A Number of Heads per Cylinder = 255 (00FFh)
001C Hidden Sectors preceding Partition = 63 (0000003Fh)
0020 Total Log. Sectors (big) = 16193457 (00F717B1h)
0024 Log. Sectors per FAT = 15815 (00003DC7h)
0028 Mirroring Flags = 0 (0000h)
002A Version No. = 0 (0000h)
002C Cluster No. of Root Dir Start = 2 (00000002h)
0030 Log. Sector No. of FS Info Sector = 1 (0001h)
0032 First logical sector number of a copy of the three FAT32 boot sectors, typically 6 = 6 (0006h)
0040 Physical Drive Number = 128 (80h) First Fixed Disk
0042 Extended Boot Signature = 41 (29h)
0047 Volume Label = NO NAME
0052 FileSystem Type = FAT32
First FAT begins at LBA 1130
Second FAT begins at LBA 16945
Root Directory begins at LBA 32760
First file data (cluster 0) begins at LBA 32768
ListUSBDrives displays:
MountPoint = K:
Volume Label = RMPARTUSB
Volume Size = 8 GB (FAT32, 4 KB clusters, 8 GB free)
Volume Serial = C925-3A9D
Partition Type = FAT32x (0C), boot
Partition Align = 512 1st Cluster Align = 4 K
Volume Name = ?Volume{03f6d2f7-61ed-11e1-a1fe-005056c00008}
Partition Name = DeviceHarddisk3Partition1
Bus Type = USB
Drive Type = removable
Device Types = ---
NoMediaNoLetter = no (configure >= 3)
Volume DevID = STORAGEVOLUME_??_USBSTOR#DISK&VEN_PRETEC&PROD_08GB_REX100&REV_1.00#09021000000000000264762875&0#{53F56307-B6BF-11D0-94F2-00A0C91EFB8B}
Drive DevID = USBSTORDISK&VEN_PRETEC&PROD_08GB_REX100&REV_1.0009021000000000000264762875&0
USB DevID = USBVID_4146&PID_090209021000000000000264762875
Ctrl2 DevID = USBROOT_HUB204&4E48F5B&0
Host Ctrl DevID = PCIVEN_8086&DEV_293A&SUBSYS_020D1028&REV_023&2411E6FE&1&EF
Host Ctrl Name = Intel® ICH9 Family USB2 Enhanced Host Controller - 293A
Volume DosDevName = DeviceHarddiskVolume6
Disk DosDevNames = DeviceHarddisk3DR3, Device00000093
Removal Policy = orderly removal ('Optimize for performance')
WriteCache = no
Hotplug Device = no
Hotplug Media = no
Removable Media = yes
Partition Number = 1 of 1
Friendly Name = PRETEC 08GB REX100
Requested Power = 500 mA (bus powered)
USB Version = 2.1 (High-Speed)
USB Friendly Name = PRETEC 08GB REX100
USB Serial = 09021000000000000264762875 (exceeds the maximum length!)
USB Port Name = 8-4
Here are some results from Xcopying 52MBs of 1122 driver files of sys,dll and inf files to a freshly FAT32 formatted USB 3 Flash drive under Windows 7 64-bit OS and then overwriting the same files two more times:
OLD = formatted with unaligned older version
NEW ALIGNED VERSION = formatted with new cluster aligned version
times in seconds.
USB 2 PORT
OLD NEW ALIGNED VERSION
55 50
46 40
49 40
USB 3 PORT
OLD NEW ALIGNED VERSION
51 44.7
42.5 36
46 37
As you can see, the new version does appear to be faster whether using USB 2 or USB 3 ports!
I have also updated the NTFS code in v 2.1.647 now. If NTFS is selected and no other CHS option is selected in RMPrepUSB (e.g. ZIP or 64hd/32sec option is NOT selected), then the 1st NTFS partition will start at sector 2048 (Cyl 0 Hd 32 Sec 33).
Comparing small file copy times between old and new RMPrepUSB versions on USB 3 port formatted as NTFS gives:
(damn I just hit backspace and my focus was apparently off editing my post and Firefox went back to the previous page, then when I went forward all my typing was lost )
I tested using a couple of batch files calling robocopy. I need to learn looping, so I just copied and pasted a couple of times to have it repeat the copy 3 times for each copy job (see details below).
Running under regular Windows wasn't giving me somewhat exclusive disk access, so I booted to Safe Mode and let it run there untouched.
Test rig; Dell Latitude E5400, 2GiB RAM, Windows XP Professional SP3. Test UFD; SDCZ36-002G (sandisk cruzer 2GB).
First I tested RMPrepUSB v2.2.644, then I uninstalled and tested your new beta (v2.2.647).
Process I used for each version;
1) RMP; Partition Size: MAX, Bootloader: Syslinux, Filesystem: FAT32, Overrides: ALL unchecked, Copy OS Files: UNCHECKED. Upon Prepare, allowed default Syslinux choices. (see Drive Info output above for what the drive capacity after this).
3) Put iso in C:\, Used 7zip to extract iso contents to C:\linuxmint-12-gnome-dvd-64bit, made directories C:\copylogs and C:\testcopy
4) Created batch file that copies extracted files to UFD, then back to HDD, repeats twice, then repeats whole process with entire iso:
Spoiler
set hdsrc=c:\linuxmint-12-gnome-dvd-64bit\
set ufd=e:\
set wrk=c:\testcopy\
::Make sure UFD is clean
del /f /s /q %ufd%*.*
del /f /s /q /ah %ufd%*.*
del /f /s /q /as %ufd%*.*
rd /s/q %ufd%
::Copy all extracted contents of iso to UFD, don't show progress, copy all subdirectories, log.
robocopy %hdsrc% %ufd% *.* /NP /E /LOG+:c:\copylogs\RMP_content_1_tousb.log
::Clean temporary directory
rd /s/q %wrk%
mkdir %wrk%
::Copy all previously copied files from UFD to working directory, don't show progress, and log.
robocopy %ufd% %wrk% *.* /NP /E /LOG+:c:\copylogs\RMP_content_1_fromusb.log
::Repeat 2 more times, append each log.
::Make sure UFD is clean
del /f /s /q %ufd%*.*
del /f /s /q /ah %ufd%*.*
del /f /s /q /as %ufd%*.*
rd /s/q %ufd%
robocopy %hdsrc% %ufd% *.* /NP /E /LOG+:c:\copylogs\RMP_content_1_tousb.log
::Clean temporary directory
rd /s/q %wrk%
mkdir %wrk%
robocopy %ufd% %wrk% *.* /NP /E /LOG+:c:\copylogs\RMP_content_1_fromusb.log
::Make sure UFD is clean
del /f /s /q %ufd%*.*
del /f /s /q /ah %ufd%*.*
del /f /s /q /as %ufd%*.*
rd /s/q %ufd%
robocopy %hdsrc% %ufd% *.* /NP /E /LOG+:c:\copylogs\RMP_content_1_tousb.log
::Clean temporary directory
rd /s/q %wrk%
mkdir %wrk%
robocopy %ufd% %wrk% *.* /NP /E /LOG+:c:\copylogs\RMP_content_1_fromusb.log
::Repeat cycle with single iso file.
set hdsrc=c:\
set file=linuxmint-12-gnome-dvd-64bit.iso
::Make sure UFD is clean
del /f /s /q %ufd%*.*
del /f /s /q /ah %ufd%*.*
del /f /s /q /as %ufd%*.*
rd /s/q %ufd%
::Copy iso to UFD, don't show progress, copy all subdirectories, log.
robocopy %hdsrc% %ufd% %file% /NP /LOG+:c:\copylogs\RMP_iso_1_tousb.log
::Clean temporary directory
rd /s/q %wrk%
mkdir %wrk%
::Copy iso from UFD to working directory, don't show progress, and log.
robocopy %ufd% %wrk% %file% /NP /LOG+:c:\copylogs\RMP_iso_1_fromusb.log
::Repeat 2 more times with new logs.
::Make sure UFD is clean
del /f /s /q %ufd%*.*
del /f /s /q /ah %ufd%*.*
del /f /s /q /as %ufd%*.*
rd /s/q %ufd%
robocopy %hdsrc% %ufd% %file% /NP /LOG+:c:\copylogs\RMP_iso_1_tousb.log
::Clean temporary directory
rd /s/q %wrk%
mkdir %wrk%
robocopy %ufd% %wrk% %file% /NP /LOG+:c:\copylogs\RMP_iso_1_fromusb.log
::Make sure UFD is clean
del /f /s /q %ufd%*.*
del /f /s /q /ah %ufd%*.*
del /f /s /q /as %ufd%*.*
rd /s/q %ufd%
robocopy %hdsrc% %ufd% %file% /NP /LOG+:c:\copylogs\RMP_iso_1_tousb.log
::Clean temporary directory
rd /s/q %wrk%
mkdir %wrk%
robocopy %ufd% %wrk% %file% /NP /LOG+:c:\copylogs\RMP_iso_1_fromusb.log
(rd seemed necessary as the del didn't actually always remove directories even with the /s switch)
5) Booted to Safe Mode (normal, no networking), ran batch file, didn't touch computer, XP screen saver came on, but otherwise no issues.
Then I rebooted, uninstalled the old RMP and installed the new Beta version. I repeated the process, except with a modified batch file that named the log files 2 instead of 1, e.g. RMP_content_2_tousb.log instead of RMP_content_1_tousb.log
I wanted the copy TO logs separate from the copy FROM logs, and separate for the contents of the iso and the whole iso itself.
Here are the robocopy logs;
[EDIT, post was too long to add the logs inline with code and spoiler tags, so I will just attach ]
So, it was faster on the contents, but not for the single iso file.
Well, that was a bit of a project, lol. Let me know and I can archive and upload the actual log files if you want.
OK, I had said the iso seemed slower, and it did on my initial tests, however looking over the logs again, my final tests (the only logs I uploaded) seem to indicate the new version is faster in all cases. It might have been due to a previous batch file version where it didn't remove all data from directories (until I added rd).
Location:The Outside of the Asylum (gate is closed)
Italy
Posted 30 April 2012 - 09:11 AM
Hmmm. These results were expected but - for the record - the original idea: http://reboot.pro/16775/ was to have a handy way to carry tests with different alignments, as I suspect that each device might have a "better" value and expecially a "better value for a given average file size" i.e. it is possible that the 1 Mb boundary represents not in itself a "one size fits all" solution but that different alignments (still within the idea of having actual first file sector a multiple of cluster size) and (for example) having ALSO the first sector of FAT tables on such an alignment, may carry better performance. Let's see the posted example in #1: It is a first partition, and has 63 Hidden sectors (standard XP and earlier cylinder boundary alignment). The amount of reserved sectors (artiificially increased) is 1067. Thus first FAT starts at 63+1067=1130 Then you have two FAT's, each 15815, i.e. first file is at 1130+2*15815=32760 The cluster is 8 sectors, i.e. 4 Kb. 1130/8=141.25 32760/8=4095 Idea , what if also the first FAT table is aligned? Example 1: Hidden sectors (Sectors Before) 63 reserved sectors:1057 Fat size: 15816
Well, if I align it at 1MB then it will also cater for erase page sizes from 16k up to 1MB - so I just chose this one size fits all alignment of 1MB.
I calculate the first file cluster to be on a 1MB boundary (e.g. 32768 in the example I listed). So adjust the reserved sectors accordingly. I allow for the volume name entry which is why the 8 sector discrepancy.
I can get just the (first) FAT table aligned to a 1MB boundary but then the first file cluster would NOT be aligned and hence all other file clusters - also the 2nd FAT table would not be aligned and presumaby the FS will update both FAT table copies. So not sure that it would do any good - if I get time to experiment I will investigate further...
Location:The Outside of the Asylum (gate is closed)
Italy
Posted 30 April 2012 - 12:13 PM
Well, if I align it at 1MB then it will also cater for erase page sizes from 16k up to 1MB - so I just chose this one size fits all alignment of 1MB. I calculate the first file cluster to be on a 1MB boundary (e.g. 32768 in the example I listed). So adjust the reserved sectors accordingly. I allow for the volume name entry which is why the 8 sector discrepancy. I can get just the (first) FAT table aligned to a 1MB boundary but then the first file cluster would NOT be aligned and hence all other file clusters - also the 2nd FAT table would not be aligned and presumaby the FS will update both FAT table copies. So not sure that it would do any good - if I get time to experiment I will investigate further...
Cannot say if providing the 8 sector for the Volume name is a brilliant idea or not. It doesn't make any difference for Cluster alignment, but I wonder if it affects the page alignment that you are trying to pursue.
The suggestion was, once determined that the MINIMUM number of sectors for a single FAT table for that filesystem is 15815, artificially increment it to allow for BOTH the FAT tables (first one AND second one) AND the first file to fall on a multiple of a cluster size. If you prefer now you moved the first file position by inserting a SINGLE "big wedge" in the "reserved sectors", I was proposing to try doing the same by inserting TWO "smaller wedges" one in the reserved sectors and one in the actual FAT tables by adding some "excess sectors" to the minimum size needed. So, if you are OK with 32760 (or more generally with "right offset - 8"), you could try: Hidden sectors (Sectors Before) 63 Reserved sectors:1985 Fat size:16380 ("right alignement -8") or Fat size:16384 ("right alignment")
Cannot say if providing the 8 sector for the Volume name is a brilliant idea or not.
I was always under the impression that most OS's cache the FAT table and so the OS will not be doing lots of writes to the FAT table area, it is only the file clusters what will gets lots of I/O.
Location:The Outside of the Asylum (gate is closed)
Italy
Posted 30 April 2012 - 02:00 PM
I was always under the impression that most OS's cache the FAT table and so the OS will not be doing lots or writes to the FAT table area, it is only the file clusters what will gets lots of I/O.
Well, in sequential writes probably not, but on fragmented device? Run Contig or wincontig on a heeavily fragmented largish file ....
FORMAT 1 = RMPrepUSB 2.1.647 - First data cluster aligned to 1MB FORMAT 2 = FAT Table aligned to 1MB FORMAT 3 = Root Dir aligned to 1MB FORMAT 4 = RMPrepUSB 2.1.630 - no alignment
Times below are for xcopy of 1122 small files repeated 3 times on freshly formatted drive (so 2nd and 3rd time is overwriting the files) Times in seconds - repeated test returns result within 0.5 seconds, ie. they are reproducible (AV was turned off during testing)
F1 F2 F3 F4 45 47 44 50 copy to freshly formatted USB 3 Flash drive 34 41 36 42 overwrite same files 35 42 37 42 overwrte again
This shows F1 and F3 are the best. F3 is best (slightly) when writing to a fresh FS, but F1 is best when overwriting files on an existing FS.
The formats are listed below - output is from RMPrepUSb ListDrives and also ListUSBDrives.exe from Uwe Sieber
FORMAT 1 - RMPrepUSB v2.1.647 - First data cluster aligned
FAT32 000B Bytes Per Sector = 512 (0200h) 000D Sectors Per Cluster = 8 (08h) 000E Reserved Sectors before first FAT = 1067 (042Bh) 0010 Number of FATs = 2 (02h) 0011 Root Entries = 0 (0000h) 0013 Total Log Sectors (small) = 0 (0000h) 0015 Media Descriptor = 248 (F8h) HDD 0016 Sectors per FAT table = 0 (0000h) 0018 Sectors per Track = 63 (003Fh) 001A Number of Heads per Cylinder = 255 (00FFh) 001C Hidden Sectors preceding Partition = 63 (0000003Fh) 0020 Total Log. Sectors (big) = 16193457 (00F717B1h) 0024 Log. Sectors per FAT = 15815 (00003DC7h) 0028 Mirroring Flags = 0 (0000h) 002A Version No. = 0 (0000h) 002C Cluster No. of Root Dir Start = 2 (00000002h) 0030 Log. Sector No. of FS Info Sector = 1 (0001h) 0032 First logical sector number of a copy of the three FAT32 boot sectors, typically 6 = 6 (0006h) 0040 Physical Drive Number = 128 (80h) First Fixed Disk 0042 Extended Boot Signature = 41 (29h) 0043 Volume ID Serial No. = 1602651694 (5F86862Eh) 0047 Volume Label = NO NAME 0052 FileSystem Type = FAT32
First FAT begins at LBA 1130 Second FAT begins at LBA 16945 Root Directory begins at LBA 32760 First file data (cluster 0) begins at LBA 32768
Location:The Outside of the Asylum (gate is closed)
Italy
Posted 30 April 2012 - 04:45 PM
Well, type #2 makes NO sense, aligning ONLY the FAT AND NOT the Data area is obviously not that relevant, but there is seemingly a slight enhancement over #4 which would confirm my idea.
OK, I had said the iso seemed slower, and it did on my initial tests, however looking over the logs again, my final tests (the only logs I uploaded) seem to indicate the new version is faster in all cases. It might have been due to a previous batch file version where it didn't remove all data from directories (until I added rd).
You need to be very sure that the results are reproducible. I repeated each test 3 times. If I copied the same set of files 10 times (9 times overwriting) then the times were reproducible but strange - e.g. Instead of 44 36 36 36 36 36 etc. I got times like 44 35 36 36 37 35 35 36 ... Also, if I rebooted and tried again I got a similar pattern but slightly different times (usually +/- 1 sec or so). So all times were taken during the same boot session and I repeated previous format tests to check I still got the same times. Thanks for your work and confirmation. Robocopy does tend to do a lot of dir reads as well as writes, so I am not sure it is the best test as pure writes to a flash drive should show the most difference.
You need to be very sure that the results are reproducible. I repeated each test 3 times. If I copied the same set of files 10 times (9 times overwriting) then the times were reproducible but strange - e.g. Instead of 44 36 36 36 36 36 etc. I got times like 44 35 36 36 37 35 35 36 ... Also, if I rebooted and tried again I got a similar pattern but slightly different times (usually +/- 1 sec or so). So all times were taken during the same boot session and I repeated previous format tests to check I still got the same times. Thanks for your work and confirmation. Robocopy does tend to do a lot of dir reads as well as writes, so I am not sure it is the best test as pure writes to a flash drive should show the most difference.
I did reproduce each test 3 times, per my batch file although of course 10 times is better. Like I said, the batch copied iso contents to and from UFD and iso itself. I wanted to remove the files instead of overwrite because I thought that would be more consistent. Is my logic flawed in that regard?
I think, however, I should give it a rest between each 3 cycles, else the thing might start to give out. It seemed after a long test, rebooting to Windows or back to Safe Mode, and reinserting the drive, that sometimes Windows wouldn't mount it. I wonder what is "safe" for a copy test duty cycle on the electronics and flash memory.
I did not know using robocopy could be an issue, thanks, something to keep in mind. Now you have me wondering how robocopy compares to xcopy, copy, and the GUI copy.
As far as a difference of +1/-1, couldn't it just be margin of error? I mean even in Safe Mode there are factors that can affect disk and RAM usage. Maybe DOS would be better? Linux?
Hmmm. These results were expected but - for the record - the original idea: http://reboot.pro/16775/ was to have a handy way to carry tests with different alignments, as I suspect that each device might have a "better" value and expecially a "better value for a given average file size" i.e. it is possible that the 1 Mb boundary represents not in itself a "one size fits all" solution but that different alignments (still within the idea of having actual first file sector a multiple of cluster size) and (for example) having ALSO the first sector of FAT tables on such an alignment, may carry better performance.
So, I'm thinking about why each device might have a better value. I wonder about why this is with a NAND device like a UFD that uses wear-leveling. Because, the actual blocks are erased and written as needed by the controller. Maybe it is because the blocks are 16k, or multiples thereof that making a FAT size that... I don't know... isn't aligned with those block sizes? Or maybe the work the controller has to do to translate the locations from the CHS structure of FAT to the actual memory (if that is accurate, I don't know how to phrase what actually happens). Or maybe it is the controller memory bus size?
Let's see the posted example in #1: It is a first partition, and has 63 Hidden sectors (standard XP and earlier cylinder boundary alignment). The amount of reserved sectors (artiificially increased) is 1067. Thus first FAT starts at 63+1067=1130 Then you have two FAT's, each 15815, i.e. first file is at 1130+2*15815=32760 The cluster is 8 sectors, i.e. 4 Kb. 1130/8=141.25 32760/8=4095 Idea , what if also the first FAT table is aligned? Example 1: Hidden sectors (Sectors Before) 63 reserved sectors:1057 Fat size: 15816
Would these examples have a better or worse performance than the one in post #1?
Yes, that makes sense if a cluster size is say 4k or 8 sectors that you would want an exact multiple of this or else the table itself has unusable entries.
MS published the FAT32 spec, said
BPB_NumFATs 16 1 The count of FAT data structures on the volume. This field should always contain the value 2 for any FAT volume of any type. Although any value greater than or equal to 1 is perfectly valid, many software programs and a few operating systems’ FAT file system drivers may not function properly if the value is something other than 2. All Microsoft file system drivers will support a value other than 2, but it is still highly recommended that no value other than 2 be used in this field. The reason the standard value for this field is 2 is to provide redun- dancy for the FAT data structure so that if a sector goes bad in one of the FATs, that data is not lost because it is duplicated in the other FAT. On non-disk-based media, such as FLASH memory cards, where such redundancy is a useless feature, a value of 1 may be used to save the space that a second copy of the FAT uses, but some FAT file system drivers might not recognize such a volume properly.
Do you know if XP or higher will throw a fit if you just set the Number of FATs to 1?
I guess I could mod a BPB_NumFAT entry and wipe out the 2nd FAT after a format with a disk editor to experiment.
Or what if there was this crazy super customizable FAT32 format tool that you could enter choices, like, hmm let's see here, # of FATs 1, reserved sectors blah
Location:The Outside of the Asylum (gate is closed)
Italy
Posted 01 May 2012 - 07:40 AM
@breaker there is no issue whatsoever (if not that of losing some "second chance" in case of data corruption) in using a volume with FAT number = 1. We have no real way to know "what the heck" happens inside the stick, so any theory is as good as another one, the alignment of the filesystem to cluster size is an improvement on the "filesystem driver size", not really on the "controller" side, basically and AFAICU. The above is - see the links given in the elsewhere referenced post - valid on hard disks that have no such thing as "page" access or delete. With the use of big caches on disk and NCQ on most AHCI disks it is hard to tell how much this is actually "going through".
The "alignment on page" is a further (theoretical) improvement attempted by steve6375 and that would be additional to "cluster size" alignment.
As said, specifically on XP (and cannot say on later NT OS) there is an additional issue which is the SLOWness of the USB (or filesystem ) driver for FAT32, win2K simply blows it away.
BTW, there is NO such things as "unusable entries" in a table that is not aligned.
Let's see the posted example in #1: It is a first partition, and has 63 Hidden sectors (standard XP and earlier cylinder boundary alignment). The amount of reserved sectors (artiificially increased) is 1067. Thus first FAT starts at 63+1067=1130 Then you have two FAT's, each 15815, i.e. first file is at 1130+2*15815=32760 The cluster is 8 sectors, i.e. 4 Kb. 1130/8=141.25 32760/8=4095 Idea , what if also the first FAT table is aligned? Example 1: Hidden sectors (Sectors Before) 63 reserved sectors:1057 Fat size: 15816
OK, I think I am finally starting to understand. (after reading more about FAT and the link "Does FAT32 align...") Since FAT table and data volume has this structure: http://www.ntfs.com/fat-systems.htm and since the data region starts, like you said after; reserved sectors + (2 * count of sectors occupied by one FAT) Maybe with UFD, if you start the data region an integer value from the start of the disk you get better performance? But that must not be what you are saying either, because there are sectors for the root directory you didn't talk about...
I mean, you are saying the FAT should end at the end of a cluster, and the data should start at the start of an even cluster? For disk writes that are a power of 2, since flash memory uses these power of 2 blocks?
So, Microsoft says,
The start of the data region, the first sector of cluster 2, is computed as follows: FirstDataSector = BPB_ResvdSecCnt + (BPB_NumFATs * FATSz) + RootDirSectors;
In their doc..http://home.teleport.com/~brainy/fatgen102.pdf
Location:The Outside of the Asylum (gate is closed)
Italy
Posted 02 May 2012 - 11:52 AM
There is no such thing (anymore) as "root" directory as it was in FAT12/16 in FAT32. Data starts immediately after the second copy (if present) of the FAT table. or right after the first one if only one FAT exists. The first record, as steve6375 pointed out is normally the Label of the volume. Page #6 of the document you cited:
A FAT file system volume is composed of four basic regions, which are laid out in this order on the volume: 0 – Reserved Region 1 – FAT Region 2 – Root Directory Region (doesn’t exist on FAT32 volumes) 3 – File and Directory Data Region
Of course, had they written in English they would have written:
A FAT 12 or 16 file system volume is composed of four basic regions, which are laid out in this order on the volume: 1– Reserved Region 2– FAT Region 3– Root Directory Region (doesn’t exist on FAT32 volumes) 4 – File and Directory Data Region whilst a FAT 32 is composed of only three: 1– Reserved Region 2– FAT Region 3 – File and Directory Data Region
(and yes when you list 4 items you number them with ordinals from 1 to 4 and not with offsets from 0 to 3)
So, since the label doesn't move, he made it the size of a sector also?
Aha, I missed that part of the document, thanks. So, if I understand correctly, the idea is, since NAND flash erases entire pages, then writes, and FAT uses whatever cluster size (4k), that aligning the data at the start of the cluster will hopefully cause the OS FAT driver to make writes to the NAND in a more aligned fashion than if fragments of pages had to be written. So, as the OS FAT driver adds, say 4 4k clusters to a UFD, if said UFD's FAT data is aligned, and supposing (theoretically) the page is 16k or some multiple, there will (hopefully) be more writes to fewer pages because of said alignment.
Like you said, each device might have an optimal value.
Maybe I am extrapolating too far, but if I were to play a hunch, it would be interesting to perhaps not only have the part of the disk that never moves around be aligned with cluster size, but also, try to have it be a multiple of 16k, and see if that helps also?
Location:The Outside of the Asylum (gate is closed)
Italy
Posted 02 May 2012 - 06:51 PM
There are three theories that are somehow interconnected in this thread (actually 4) With regards to the same "clearer English" three Regions in FAT32:
1– Reserved Region 2– FAT Region 3 – File and Directory Data Region
First theory, confirmed by n experiments, is that aligning Region #3 to a multiple of cluster size of the filesystem gives a speed improvement in read/write operation, this is INDEPENDENT from the kind of device and also applies to "conventional", rotating hard disks (though them being much faster than a USB device the difference is less noticeable).
Second theory by steve6375, and at the moment confirmed, is that on "solid state" devices this Region #3 should be aligned to a multiple of page size.
Third theory (actually an evolution of the second and also by by steve6375) is that to further better speed this Region #3 is aligned to a multiple of page size with offset of -1 cluster (as this will normally hold the drive Label and thus it is not accessed often). This improvement seems also confirmed though seemingly causing only a slighter improvement.
Fourth theory (by yours truly) is that maybe if BOTH Region #3 and Region #2 are aligned to BOTH a cluster multiple and to a page size multiple, then there might be another small improvement. At the moment this last theory has not been tested (and BTW also the previous two need further testing).
The size steve6375 decided to choose for these preliminary tests makes a lot of sense as 1 Mb i.e. 1,048,576 can be expressed as: 524,288x2 262,144x4 131,072x8 65,536x16 32,768x32 16,384x64 8,192x128 4,096x256
so it does represent a "one size fits all" for filesystems with a 4K cluster, for several possible "page sizes" of the device.
There are three theories that are somehow interconnected in this thread (actually 4) With regards to the same "clearer English" three Regions in FAT32:
First theory, confirmed by n experiments, is that aligning Region #3 to a multiple of cluster size of the filesystem gives a speed improvement in read/write operation, this is INDEPENDENT from the kind of device and also applies to "conventional", rotating hard disks (though them being much faster than a USB device the difference is less noticeable).
That is interesting, and it sounds like preliminary testing indicates this is true. So, is there a theory as to why this is? Or what knowledge led us to this experiment?
Second theory by steve6375, and at the moment confirmed, is that on "solid state" devices this Region #3 should be aligned to a multiple of page size.
Yes, this makes sense due to the nature of NAND, so that if more clusters fit into a page size evenly, rather than spanning pages, you get a slight speed increase because the device doesn't have to erase more clusters than it would have if unaligned.
Third theory (actually an evolution of the second and also by by steve6375) is that to further better speed this Region #3 is aligned to a multiple of page size with offset of -1 cluster (as this will normally hold the drive Label and thus it is not accessed often). This improvement seems also confirmed though seemingly causing only a slighter improvement.
The terminology, sorry offset off -1 cluster. Offset from where, and it's not +1? Sorry for asking you to expand this for my noob knowledge.
Fourth theory (by yours truly) is that maybe if BOTH Region #3 and Region #2 are aligned to BOTH a cluster multiple and to a page size multiple, then there might be another small improvement. At the moment this last theory has not been tested (and BTW also the previous two need further testing).
I think this sounds true because every time a file is written to to region 3, the FAT tables are written, which would indicate the flash pages covering region 2 are completely erased and rewritten, at least the page size surrounding the entry, so potentially the entire FAT tables each time. Kind of a mind blower if your UFD is erasing and writing the entire region 2 each time a file is written.
The size steve6375 decided to choose for these preliminary tests makes a lot of sense as 1 Mb i.e. 1,048,576 can be expressed as: 524,288x2 262,144x4 131,072x8 65,536x16 32,768x32 16,384x64 8,192x128 4,096x256
so it does represent a "one size fits all" for filesystems with a 4K cluster, for several possible "page sizes" of the device.
Wonko
Yup, 4k has many common multiples with flash page sizes I would surmise. So, least common multiple for potential page sizes, I suppose?
Location:The Outside of the Asylum (gate is closed)
Italy
Posted 06 May 2012 - 09:43 AM
That is interesting, and it sounds like preliminary testing indicates this is true. So, is there a theory as to why this is? Or what knowledge led us to this experiment?
You must need to learn following links and reading Read here (already given link): http://reboot.pro/16775/ Re-read (attentively this time ) the whole thread an particularly post #7. In it there is a link to #5 on thread: http://reboot.pro/9897/ read that post AND links given in it.
The terminology, sorry offset off -1 cluster. Offset from where, and it's not +1? Sorry for asking you to expand this for my noob knowledge.
It's all in the eye of the beholder . For a number of reasons (using an additional 0 value) programmers tend to use "offset" as opposed to "normal" numbering. "Offset" is nothing but "ordinal" numbers shifted by one, i.e. you have a list of n elements and you start calling them with their offset i.e. fom the number of elements before. That is how LBA addressing works. First sector in LBA is "Sector 0" or "Sector that has 0 sectors before it" or "Sector at offset 0". The "first" theory says that you need to have a n=multiple of the cluster size BEFORE the "third region" (i.e. the "File and Directory Data Region" in the MS doc).
steve6375 thought that since the first cluster of that region is the "label" of the volume (which is very rarely changed and never accessed during file copying/reading/writing) it made sense to try using n=multiple of the cluster size before the first file (which since first cluster is occupied by the label will begin on second sector of the region). Consequently the offset to the third region has been changed from n (first theory) to n-1. I hope now it is clear .
Yup, guilty as charged. I do need to read everything again more carefully. Fooling around with disks and booting has been my favorite method of procrastination lately while I study for my computer networking course. It will be over relatively soon, then I can read instead of skim, etc.
Yeah, of course I have heard of offsets for a long time, but I wasn't sure offset from where, the LBA reference makes sense, thanks.
anyway, I did make a little time to boot my windows 8 install from an exfat UFD, so, going to post about that now...
)
]
but - for the record - the original idea:
, what if also the first FAT table is aligned?
....
) the whole thread an particularly post #7.
.
