Jump to content











Photo
- - - - -

RMPrepUSB - faster FAT32 write access on Flash memory drives


  • Please log in to reply
60 replies to this topic

#26 Jonathan Berry

Jonathan Berry
  • Members
  • 6 posts
  •  
    Canada

Posted 21 January 2013 - 08:20 AM

What happened to this experiment?  I downloaded RMPrepUSB, and even the docs belonging to version 2.1.647 (the one referenced in this thread) do not mention the alignment question.  Actually formatting a USB stick with the current version seems to produce XP-friendly formatting, offset of 32,256 bytes.  I'm good with the Command Line, if that's the most direct way to do it.  Yes, I'm willing to sacrifice most of a megabyte of 8 GB, to gain 20% in file transfer speed.



#27 steve6375

steve6375

    Platinum Member

  • Developer
  • 6461 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars
  •  
    United Kingdom

Posted 21 January 2013 - 08:50 AM

See Version History on web site

 

2.1.647 28-04-2012 RELEASED on 2012-05-22 -  FAT32 format now starts first cluster at 1MB boundary. NTFS format will start 1st partition at sector 2048 instead of 63. Both these changes lead to >10% speed advantage when copying small files to a USB flash drive. Changed status to fully released, non-Beta on 2012-05-25. Uses grldr grub4dos version 0.4.6a
2.1.647 28-04-2012 RE-RELEASED on 2012-05-28 - now uses latest grub4dos 0.4.5c 2012-05-22 which was just released on same day I decided to release RMPrepUSB v2.1.647!
2.1.648 04-06-2012 RELEASED on 10-06-2012 as released version - F11 shortcut key now works!
2.1.648 qemu - contains more compatible version of QEMU
2.1.648 wee - new qemu + bugfix for wee install
2.1.650 Function to select any one of four primary partitions on a USB Flash drive to be the visible one under Windows.
2.1.651 Added QEMU boot direct from ISO and QEMU boot direct from HDD image options to File menu.
2.1.652 Can now install std mbr to non-USB drive (bugfix) + install grub4dos button tries floppy install if hdd install fails, new RMPartUSB.exe with Drive Info FAT16 BPB bugfix. new grub4dos version.
2.1.653 Prevent QEMU from trying to use more than 1160MB of virtual memory, fix minor WinContig bug (did not show drive letter in confirmation prompt)
2.1.654 30-12-2012 RELEASED WinContig (Shift+F2) function added so the user can just run WinContig on any drive manually. Updated grub4dos version included.
2.1.655 - Added Drive menu option to clear Read Only attributes on a drive and volume - useful if Win8ToGo has set the RO attribute on your USB drive!
2.1.656 - F4 (open menu.lst in Notepad), now will also open \grub\menu.lst and \boot\grub\menu.lst if they exist.
 


#28 Jonathan Berry

Jonathan Berry
  • Members
  • 6 posts
  •  
    Canada

Posted 21 January 2013 - 05:07 PM

Version History, of course.  Sorry, I didn't look there. I downloaded the latest version 2.1.654, which formatted a USB stick with an offset of 32,256 bytes.  I don't see 2.1.647, the one announced to offset by 1 MB, on your download page.  Would the intermediate version 2.1.648 work for my purpose of getting a 1 MB offset?  I did find the interface confusing.  When it failed to do what I wanted with XP (NTLDR) formatting, I tried Win7 formatting (1 MB offset, right?), but in fact RMPrepUSB did something quite different, creating two partitions, the main one having an offset again of 32,256 bytes.    I guess my trouble starts with trying to use a sophisticated tool for a simple (but valuable!) purpose.  The operation takes seconds, but learning how to do it takes hours.  I did not mention it in the original question, but the computer runs XP, SP3.



#29 steve6375

steve6375

    Platinum Member

  • Developer
  • 6461 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars
  •  
    United Kingdom

Posted 21 January 2013 - 05:40 PM

Any version after 647 has the FAT32 optimisation change.

You get 2 parititions if you tick the  'Boot as HDD (C: 2PTNS)' checkbox (not surprisingly)! This is recommended as some BIOSes won't boot from a USB flash drive that has a single partition - the 2nd partition is just a dummy one to force those BIOSes into thinking it is an HDD and booting it!

NTFS optimisation is already done by Windows (the Windows DLL is used to format the drive) - speed optimisation in RMPrepUSB is done for FAT32 only - it is not needed for NTFS as it should be already fairly optimum.  I just start the partition at 2048 which may improve things for a flash drive.



#30 Jonathan Berry

Jonathan Berry
  • Members
  • 6 posts
  •  
    Canada

Posted 21 January 2013 - 05:47 PM

Thanks.  So is there a way for me to format the USB stick to FAT32  with a 1 MB offset?  



#31 steve6375

steve6375

    Platinum Member

  • Developer
  • 6461 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars
  •  
    United Kingdom

Posted 21 January 2013 - 05:49 PM

It should be doing this. The 1st cluster should be at  a modulo 1Mb offset (not the partition) - how are you checking the format and what are you expecting?



#32 Jonathan Berry

Jonathan Berry
  • Members
  • 6 posts
  •  
    Canada

Posted 21 January 2013 - 05:56 PM

I deleted the partition, then ran RMPrepUSB 2.1.654 :

 

msinfo32 for Kingston.

Description Disk drive
Manufacturer (Standard disk drives)
Model Kingston DataTraveler 2.0 USB Device
Bytes/Sector 512
Media Loaded Yes
Media Type Removable media other than	floppy
Partitions 1
SCSI Bus Not Available
SCSI Logical Unit Not Available
SCSI Port Not Available
SCSI Target ID Not Available
Sectors/Track 63
Size 7.45 GB (8,003,197,440 bytes)
Total Cylinders 973
Total Sectors 15,631,245
Total Tracks 248,115
Tracks/Cylinder 255
Partition Disk #1, Partition #0
Partition Size 7.46 GB (8,010,809,856 bytes)
Partition Starting Offset 32,256 bytes
 



#33 Jonathan Berry

Jonathan Berry
  • Members
  • 6 posts
  •  
    Canada

Posted 21 January 2013 - 06:11 PM

"Partition Starting Offset 32,256 bytes"

 

I was expecting a number 1,xxx,xxx rather than 32,256.  Like on the computer's msinfo32 info for the main hard drive, a SSD:

 

Partition Disk #0, Partition #0
Partition Size 107.42 GB (115,341,262,848 bytes)
Partition Starting Offset 1,048,576 bytes

 

Am I barking up the wrong tree?


Edited by Jonathan Berry, 21 January 2013 - 06:12 PM.


#34 steve6375

steve6375

    Platinum Member

  • Developer
  • 6461 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars
  •  
    United Kingdom

Posted 21 January 2013 - 06:13 PM

That does not show it!

Use RMPrepUSB - Drive Info - type in P1.

See early posts in the discussion.

 

e.g.

First FAT begins at LBA 1130
Second FAT begins at LBA 16945
Root Directory begins at LBA 32760
First file data (cluster 0) begins at LBA 32768

 

The first file data (cluster 0) should be a multiple of 2048  (1MB)  - this is the change to speed up FAT32 writes on a USB flash drive.


  • Motasem likes this

#35 Jonathan Berry

Jonathan Berry
  • Members
  • 6 posts
  •  
    Canada

Posted 21 January 2013 - 06:25 PM

Thanks. I click "Drive Info", put "P1" in the textbox and press OK.  A file USBInfo.txt is created and opened.  At the bottom is this: 

 

First FAT begins at LBA 180
Second FAT begins at LBA 15446
Root Directory begins at LBA 30712
First file data (cluster 0) begins at LBA 30720

 

Is that a desired result?  32768 (your post above) = 30720 (my result) + 2048 (1 MB) so I would guess that it is what was wanted.  I was succeeding when all the time I thought I was failing !?


Edited by Jonathan Berry, 21 January 2013 - 06:28 PM.


#36 steve6375

steve6375

    Platinum Member

  • Developer
  • 6461 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars
  •  
    United Kingdom

Posted 21 January 2013 - 06:26 PM

Yep - 2048 * 15 = 30720  :good: 



#37 rocketero

rocketero

    Frequent Member

  • Advanced user
  • 155 posts
  •  
    United States

Posted 22 January 2013 - 12:01 PM

Does this utility has any benefits for an SDXC 64GB microsdcard?  or for the lower sized 32GB microsd ? thanks for any replies.



#38 steve6375

steve6375

    Platinum Member

  • Developer
  • 6461 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars
  •  
    United Kingdom

Posted 22 January 2013 - 12:04 PM

Does this utility has any benefits for an SDXC 64GB microsdcard?  or for the lower sized 32GB microsd ? thanks for any replies.

Probably, although I can't say how much. I depends on what you used to format it with previously and what filesystem it has (FAT32/NTFS???)



#39 steve6375

steve6375

    Platinum Member

  • Developer
  • 6461 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars
  •  
    United Kingdom

Posted 22 January 2013 - 12:39 PM

*
POPULAR

For anyone new to this discussion, here is the theory.

When writing to a USB Flash drive, the flash drive contoller circuitry must do a  read/erase/write cycle on a whole page as individual sectors cannot be erased in flash memory. Let us assume a 16k erase page size for this example.

Files consist of groups of clusters - e.g. 4k clusters. Now consider a filesystem that accesses a disk in clusters (CL). What if the clusters are arranged like this:

 

[Erase page block 1][Erase page block 2][Erase page block 3]

       [CL1][CL2][CL3][CL4][CL5][CL6][CL7][CL8]

 

Notice that CL3 overlaps an erase page boundary. So when the OS writes to cluster 3, the flash controller has to perform a read/erase/write cycle on Erase page block 1, and then another read/erase/write cycle on Erase page block 2 for the 2nd half of the cluster. The same is true of CL7. Thus we have caused two extra time-consuming read/erase/write cycles.

 

Now what if we ensure that a cluster ends on  a page boundary - we get:

 

[Erase page block 1][Erase page block 2][Erase page block 3]

     [CL1][CL2][CL3][CL4][CL5][CL6][CL7][CL8]

 

If each cluster is written separately we get 8 r/e/w cycles instead of 10.

 

To be honest I was expecting only a marginal (if any) improvement in file write times (especially as the above case is greatly simplified and exaggerated!), but when I measured the difference under Windows in practice, it was quite significant!


  • Nuno Brito, ilko, Porfyr and 4 others like this

#40 Dystopian

Dystopian
  • Members
  • 2 posts

Posted 22 January 2013 - 08:47 PM

Faster results will be with 
[Erase 1 page block][Erase 2 page block][Erase 3 page block]
[       CL1        ][       CL2        ][        CL3       ]

Here is russian dicussion about cluster aligning on block boundary. There is also dfb64 utility for block size searching.

#41 steve6375

steve6375

    Platinum Member

  • Developer
  • 6461 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars
  •  
    United Kingdom

Posted 22 January 2013 - 09:19 PM

Probably, but the erase page size is not easily known!



#42 rocketero

rocketero

    Frequent Member

  • Advanced user
  • 155 posts
  •  
    United States

Posted 23 January 2013 - 12:28 AM

Probably, although I can't say how much. I depends on what you used to format it with previously and what filesystem it has (FAT32/NTFS???)

 

It was formatted with Ubuntu Gparted app, as FAT32 for an Android ICS tablet. it has 2 primary partitions, the first and biggest is the FAT32 (49GB) and the second smaller as EXT4 (13GB). the total after formatted is around 59GB out of the 64GB that it's advertised (after converting 64gb to the real formatting)



#43 steve6375

steve6375

    Platinum Member

  • Developer
  • 6461 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars
  •  
    United Kingdom

Posted 23 January 2013 - 09:20 AM

OK, well try doing some write benchmarks and then reformat using RMPrepUSB Fat32



#44 robertcollier4

robertcollier4

    Member

  • Members
  • 32 posts
  •  
    United States

Posted 23 January 2013 - 12:07 PM

This is a great finding. Always creating my partitions 'aligned' with commercial tools, I have actually wondered about this for a long time knowing that partitions store metadata at the beginning but not previously knowing a way to check the "first file data sector" of a partition. I think this means that all the current partition utilities out there like "Paragon Alignment Tool" - and even all the partition alignment advice on the forums about using tools like Gparted is bunk and wrong??? Because everyone is telling you to use partition tools such as Gparted or Partition Guru, etc. to start your partition at a sector divisible by 2048 or 4096. However - it does not take into account the beginning file tables and log tables which are NOT sized in multiples of sector size or page size. So although the partition starts at an aligned sector - because of the beginning partition metadata - the data ends up all being un-aligned!

 

Following are the results from RMPrepUSB "Drive Info" - P1 for an "aligned" partition created with a commercial partition program vs. the partition created with RMPrepUSB new version.

 

Created with Partition Guru 4.20 - FAT32 at aligned sector 4096 (Gparted would give similar results):

COMMAND LINE: DRIVE=1 USBINFO USBSTART=P1 SURE 
Accessing Drive 1 - "SanDisk SDDR-113" (2,002,255,872 bytes)
USB Sector 4096
First FAT begins at LBA 4134
Second FAT begins at LBA 7939
Root Directory begins at LBA 11744
First file data (cluster 0) begins at LBA 11752
 
Created with RMPrepUSB 2.1.656 - Select FAT32 in Box 4 - Uncheck all other boxes in Box 4 - "Prepare Drive":
COMMAND LINE: DRIVE=1 USBINFO USBSTART=P1 SURE 
Accessing Drive 1 - "SanDisk SDDR-113" (2,002,255,872 bytes)
USB Sector 63
First FAT begins at LBA 558
Second FAT begins at LBA 4371
Root Directory begins at LBA 8184
First file data (cluster 0) begins at LBA 8192

 

So the "aligned" partition created with Partition Guru 4.20 starts at sector 4096 - but the first file cluster starts at 11752 which is unaligned.

 
This finding means that ALL current partition alignment advice currently discussed all over the internet on numerous forums is ALL WRONG. This means that "First file data cluster" alignment should also be performed for SSDs then also, correct? So all the current SSDs out there which are being created with partitions starting at 1MB boundaries must also be unaligned according to their "First file data cluster". This means that even Gparted is creating non-optimally aligned partitions because it is taking into account "partition start" and not "first file data cluster" start.

Edited by robertcollier4, 23 January 2013 - 12:29 PM.

  • Motasem likes this

#45 steve6375

steve6375

    Platinum Member

  • Developer
  • 6461 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars
  •  
    United Kingdom

Posted 23 January 2013 - 12:22 PM

The proof of the pudding is in the eating.

I tested using Windows Xcopy to write lots of small files to the target flash drive (and repeated 10 times and then repeated this each 3 times after a fresh partition/format each 10 times - with AV off).

i.e.

 

wipe/partition/format

Xcopy > target SSD

(repeat Xcopy 9 times thus overwriting files)

 

wipe/partition/format

Xcopy > target SSD

(repeat Xcopy 9 times thus overwriting files)

 

wipe/partition/format

Xcopy > target SSD

(repeat Xcopy 9 times thus overwriting files)

 

Maybe you can try some SSD tests and post results?



#46 robertcollier4

robertcollier4

    Member

  • Members
  • 32 posts
  •  
    United States

Posted 23 January 2013 - 12:42 PM

NTFS optimisation is already done by Windows (the Windows DLL is used to format the drive) - speed optimisation in RMPrepUSB is done for FAT32 only - it is not needed for NTFS as it should be already fairly optimum.

Windows Vista and later create partitions at starting sector 2048 (at 1 MB boundaries). However - then you have the partition metadata for NTFS.

 

Do you know of a way to check "First file data sector" for a NTFS drive? I tried RMPrepUSB > Drive Info > P1 and got:

USB Sector 4096

 

NTFS
000B Bytes Per Sector = 512 (0200h)
000D Sectors Per Cluster = 8 (08h)
0015 Media Descriptor = 248 (F8h) HDD
0020 Total Log. Sectors (big) = 0 (00000000h)
0028 Total Sectors on HDD = 0000000010D9DE9Fh
0030 Log. Cluster no. for $MFT file = 00000000000C0000h
0038 Log. Cluster no. for $MFTMirr file = 00000000010D9DE9h
0040 Clusters per MFT Record = 246 (F6h)
0044 Clusters per Index Buffer = 229 (E5h)
 
Don't see any way to get "First file data sector"? If a partition starts at sector X, then how many sectors Y does the NTFS metadata take up - to give us X+Y sector location where actual data clusters start getting written?

Edited by robertcollier4, 23 January 2013 - 12:56 PM.


#47 robertcollier4

robertcollier4

    Member

  • Members
  • 32 posts
  •  
    United States

Posted 23 January 2013 - 01:09 PM

Also, just thought I would point out, Partition Guru 4.20 does display an entry for "Data start sector" for FAT32 drives but it is a different value of 8121 than the RMPrepUSB>"Drive Info">P1 "First file data" of 8192.

 

For the same drive, RMPrepUSB says:

COMMAND LINE: DRIVE=1 USBINFO USBSTART=P1 SURE
Accessing Drive 1 - "SanDisk SDDR-113" (2,002,255,872 bytes)
USB Sector 63
First FAT begins at LBA 558
Second FAT begins at LBA 4371
Root Directory begins at LBA 8184
First file data (cluster 0) begins at LBA 8192


and Partition Guru 4.20 says:

fat32rmprepusb.gif


Edited by robertcollier4, 23 January 2013 - 01:13 PM.


#48 steve6375

steve6375

    Platinum Member

  • Developer
  • 6461 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars
  •  
    United Kingdom

Posted 23 January 2013 - 01:15 PM

Root Directory seems to be 63 out - perhaps they forgot to add on the partition offset?

They use a Data Start Sector = Root Directory, however the first two clusters are unused, so the first file data sector will be two clusters past the Root Dir sector



#49 steve6375

steve6375

    Platinum Member

  • Developer
  • 6461 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars
  •  
    United Kingdom

Posted 23 January 2013 - 01:42 PM

The MFT is usually not at the beginning of the volume, on a large volume it is at the 3GB position. NTFS will use any free cluster for a file, so it could be before the MFT or after.

The main thing would be to ensure that any file starts at a sector number (LBA number) that is divisible by 8.

If you install grub4dos to the NTFS formatted drive and then go to the grub4dos console and type

 

blocklist /$MFT

 

you can see where it starts and how many sectors it takes up. Or use it on any file on the drive (so you can just put one file on the drive and use blocklist on that?).

Note that blocklist seems to have a problem giving a result with some files (may be their sector position is many GB onto the drive and the number causes an overflow?).

 

for blocklist /$mft I get 6291456+39936, so 6291456 is divisible by 8.

 

 



#50 Clemens Ratte-Polle

Clemens Ratte-Polle
  • Members
  • 1 posts
  •  
    Germany

Posted 23 January 2013 - 01:55 PM

.

Firefox Lazarus Form Recovery Extension :)

.

https://addons.mozil...-form-recovery/

 

(damn I just hit backspace and my focus was apparently off editing my post and Firefox went back to the previous page, then when I went forward all my typing was lost :ranting2:)

Edited by vocivoci, 23 January 2013 - 01:56 PM.





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users