60 replies to this topic

[robertcollier4]

Root Directory seems to be 63 out - perhaps they forgot to add on the partition offset?

They use a Data Start Sector = Root Directory, however the first two clusters are unused, so the first file data sector will be two clusters past the Root Dir sector

You are right, I tried it with another FAT32 partition with a different starting sector - and Partition Guru is not adding the starting sector partition offset. Once you add the "starting sector" to these values they match the values given in RMPrepUSB. I will report this bug to Eassos.

 

If Data Start Sector is 2 clusters past Root Dir Sector - then in the example of post #47 shouldn't Data Start Sector = Root Dir Sector + 16 instead of Root Dir Sector + 8? The cluster size is 4096 bytes, 1 cluster = 8 sectors. RMPrepUSB seems to (wrongly) be giving First File Data Sector  = Root Dir Sector + 1 cluster.

Edited by robertcollier4, 23 January 2013 - 01:58 PM.

[steve6375]

OK - sorry - had to double-check.

Actually I add one cluster because when RMPrepUSB creates a volume the first entry is the volume label, so the first file cluster is +8 sectors.

The 'root dir' already takes into account the two unused cluster table entries! Sorry about the confusion.

[robertcollier4]

Thanks, makes sense - Microsoft docs show that Root Dir Sectors is technically 0 in FAT32 because:

 

http://www.pjrc.com/.../ide/fat32.html

>> They lack "RootDirSectors", because FAT32 stores the root directory the same way as files and subdirectories, so RootDirSectors is always zero with FAT32. For FAT16 and FAT12, this extra step is needed to compute the special space allocated for the root directory.

 

http://www.microsoft...are/fatgen.mspx

>> Note that on a FAT32 volume the BPB_RootEntCnt value is always 0, so on a FAT32 volume RootDirSectors is always 0. 

>> FirstDataSector = BPB_ResvdSecCnt + (BPB_NumFATs * FATSz) + RootDirSectors;

 

RMPrepUSB values check:

Partition Offset = 63

BPB_ResvdSecCnt = 495

BPB_NumFATs = 2

FATSz = 3813

RootDirSectors = 0

 

So FirstDataSector = 63 (Partition Offset) + 495 + 2*3813 + 0 = 8184 (Correct)

 

Then http://en.wikipedia....llocation_Table

Volume Label. (Since DOS 2.0) Indicates an optional directory volume label, normally only residing in a volume's root directory. Ideally, the volume label should be the first entry in the directory (after reserved entries) in order to avoid problems with VFAT LFNs.

 

So FileDataStart = FirstDataSector + 1 volume label cluster = 8192 (Correct)

 

Thanks!

Edited by robertcollier4, 23 January 2013 - 02:49 PM.

[steve6375]

Ok.

Actually, it is fairly irrelevant. As long as the physical LBA sector position of a cluster is divisible by the cluster size, that is the best we can do. Files can be 1 cluster long or 1000's of clusters long, so their start position will be 'Modulo 1 cluster' - the important things is to ensure the start position of each cluster is divisible by 8 or 16 or 32, etc. - this ensures that no cluster will overlap an erase 'page' boundary.

[robertcollier4]

The MFT is usually not at the beginning of the volume, on a large volume it is at the 3GB position. NTFS will use any free cluster for a file, so it could be before the MFT or after.

The main thing would be to ensure that any file starts at a sector number (LBA number) that is divisible by 8.

If you install grub4dos to the NTFS formatted drive and then go to the grub4dos console and type

 

blocklist /$MFT

 

you can see where it starts and how many sectors it takes up. Or use it on any file on the drive (so you can just put one file on the drive and use blocklist on that?).

Note that blocklist seems to have a problem giving a result with some files (may be their sector position is many GB onto the drive and the number causes an overflow?).

 

for blocklist /$mft I get 6291456+39936, so 6291456 is divisible by 8.

Just some additional thinking out loud about NTFS alignment for SSDs. The newest SSDs from Intel have a page size of 8KB (16 sectors) and an erase block size of 2MB (256 pages or 4096 sectors). For these absolute newest SSD drives, divisble by 8 may not be enough for absolute optimal performance 

 

Found some info that the only reserved sectors in an NTFS partition are 16 sectors in the beginning for the Partition Boot Record (also called Volume Boot Record) broken down into "Boot Sector" (1 sector) and "Bootstrap Code" (15 sectors). So the data clusters start at Partition Offset + 16 sectors..

 

The newest tools create their partitions at 4096 sector (2MB) starting offset because Intel publishes that erase blocks are now 2MB. However - if 16 sectors are being used for the Partition Boot Record - then maybe we should be creating our NTFS partitions at starting sector 4080 (to get 4096 FirstDataSector = 4080 partition start + 16 sectors for PBR).

Edited by robertcollier4, 23 January 2013 - 03:25 PM.

[Wonko the Sane]

@robertcollier4

JFYI, some of the "history":

http://reboot.pro/to...-under-windows/

http://www.msfn.org/...n-its-clusters/

 

AND some results hinting how - besides the theoretical advantage of the alignment - the "real difference" in speed with FAT32 may be connected to the "dumbing down" that happened after 2K (probably there is *something* else that slows down he driver and which effect is *somehow* nullified or reduced by the proper alignement)   :

http://www.msfn.org/...d-on-usb-stick/

 

About NTFS, no.

A NTFS volume is "entirely" made of clusters (including the first 16 sectors).

If you check a NTFS bootrecord (first sector of a NTFS volume/flesystem) you will see how there are NO reserved sectors (unlike in FAT).

As a matter of fact the first 16 sectors are a file called $Boot in NTFS.

In other words a NTFS is "inherently" aligned, as long as it's start is aligned.

 

Wonko

[Leolo]

This is perhaps a bit offtopic, but if anyone is interested (I learned a lot in this thread and I'm going to boomark it. I hope you don't mind!), the UNIX guys are using a similar method to align their partitions:

https://forums.freeb...4k-disks.42725/

They create a partition ("slice" in unix parlance) which is correctly aligned to CHS boundaries (but is unaligned for SSDs and Advanced Format HDDs).

 

And then they add free space before the real partitions (which are located inside the "slice"), to align them for SSDs and AF hard disks.

The concept of "slices" (putting partitions inside a partition) reminds me a bit of MSDOS "Extended Partitions" and logical partitions. Does anyone know who invented the idea first?

Kind regards.

[Wonko the Sane]

This is perhaps a bit offtopic, but if anyone is interested (I learned a lot in this thread and I'm going to boomark it. I hope you don't mind!), the UNIX guys are using a similar method to align their partitions:

https://forums.freeb...4k-disks.42725/

They create a partition ("slice" in unix parlance) which is correctly aligned to CHS boundaries (but is unaligned for SSDs and Advanced Format HDDs).

 

And then they add free space before the real partitions (which are located inside the "slice"), to align them for SSDs and AF hard disks.

The concept of "slices" (putting partitions inside a partition) reminds me a bit of MSDOS "Extended Partitions" and logical partitions. Does anyone know who invented the idea first?

Kind regards.

....though  they are not "the UNIX guys", but rather the (good , possibly even "better" ) BSD guys .

The "slice" is BSD "jargon".

 

Wonko

[sixcentgeorge]

how do you named this format fat32 tool in latest Install_RMPrepUSB_Full_v*.zip ?

 

i have downloaded RMPTN but it seems made in 2002 . will you update it ? 

may be adding grub4dos mbr support beside io.sys and for the partition info : the grub4dos "vision" of the partition , i mean hd[0,0] or else settings that are due to the usb firmware and its partition system .

 

i used recently LinuxLive USB Creator : http://www.linuxliveusb.com/fr/homeand with its formatted usb drive i had very good transfers rate , may be it finds a nice solution for its formatting ? [ continuous 20 mb/s with a hm57 and a kingston dt g3 usb2 ]

may be that can help

[Logic]

Hey guys

 

I had a nice post about implementing trim on flash drives and another formatter that implements alignment and enabling write cache for FAT/32 etc, but the forum wouldnt let me post it?  A hickup I assume.

 

SSD and pen drives work with flashmemory which is organized in blocks of a unknown size, probably up to 512 KB, the manufactures don't tell us. For efficient writing the data clusters of the file system should be aligned as good as possible as these flash blocks to avoid unnecessary flash erases.
Therefore new pen drives usually come with a nicely aligned primary partition and a FAT32 file system with aligned clusters, provided the manufacturer made things right...

If a flash drive is partitioned or formatted by the user it is essential to use the right tool. Windows Vista and Windows 7/8 do a good job here. They align new partitions and data clusters very well. In contrast Windows XP creates partitions in the classic way at track or cylinder borders which nearly always results in unaligned partitions.

Data clusters: The NTFS file system has its tables somewhere in the middle of the partition, so the data clusters just start at the beginning, no alignment issues here. FAT/FAT32 have their allocation tables at the beginning, the data cluster follow after some 'reserved' sectors. Modifying the reserved sector count is the key to align the data clusters and this is what Vista and Win7 do very well while XP does a 4K alignment only.
But all this is relative to the partitions!. With non aligned partitions the data clusters are created non aligned too!

I've implemented the display of partition and cluster alignment in ListUsbDrives. As explained above it shows these information only if they are relevant. For SuperFloppy formatted drives there is no partition alignment and for NTFS drives the first cluster alignment is identical to the partition alignment.

Even with a perfectly aligned first cluster, with a cluster size smaller than the flash block size files are perfectly aligned only with a certain probility: If the cluster have half the size of the flash blocks the every second file is perfectly aligned, with a quarter every fourth and so on. Only with clusters of the size of the flash blocks and at least this alignment full write speed is guaranteed.
But larger clusters lead to more non-usable memory (slack space). And with clusters larger than 4KB there is no more NTFS compression.

By the way: TrueCrypt since V6.0 align the data as the container is aligned. From the history: "the start of the data area will always be aligned with the start of a host-filesystem/physical sector"

How to align a partition
Aligning a partition without data loss is reported being possible by means of GParted (based upon a bootable Linux) by moving the partition back and forth, but for a flash drive I think it is not worth the effort. Just delete and recreate it under Vista/Win7.
On "removable" drives the Windows disk management does no like to delete or create partitions. But the Vista/Win7 commandline tool DISKPART can.
Another alternative is the SD Memory Card Formatting Software which creates an partition with aligned clusters on USB "removable" drives. The partition is aligned only if the drive is 4GB or larger, so don't format smaller drives with NTFS after formatting them by means of this software!

A nice and free partition software for Windows which supports alignment is the "Active@ Partition Manager". By default it uses the classic alignment but is offers an option for the exact partition offset. Increase this number to the next value which can be evenly divided by 1024 to get a 512K alignment and decrease the size by the value you have increased the start offset with.
Then it shows a warning plus question "Partition geometry is not aligned with disk track borders. Align on track borders?", we answer "No" because we know better...
Letting format the drive with FAT resulted here in a 1K cluster alignment. Formatting again with XP (FORMAT or dialog) resulted in the expected 4K alignment. So, better let Windows do the formatting...

What will we get? Depends on the device... With an old 128MB SD card I had no improvement of the write speed at all while a Corsair Voyager 16GB writes large files now with 15 MB/s instead of 6.5 thanks to aligned 64K clusters.

 

http://www.uwe-siebe...usbstick_e.html

 

http://download1353....w/ForceTrim.zip

 

USB Write protect as Steve6375's files are forever being flagged by VirusTotal?

http://www.nareshmdr.com.np/downloads/

Edited by Logic, 23 May 2015 - 10:09 PM.

[sixcentgeorge]

uwe-sieber is very good , its dos xms - emm tool aka UMBPCI is nice ...with xp and later , that is bad he did not made something as good for handling mem and optimize it

thank for links