10 replies to this topic

[kodiak567]

Hello

 

Using IMDisk on Windows XP machine with 2GB DDR1 RAM.

 

File copy/read is very slow, directory with 2300 *.png files that is 12MB total takes like 2 minutes to copy from RAMDisk to RAMDisk. The transfer rate is below 120KB/s. To compare similar operation on my IDE HDD with 8MB Cache is done at 170KB/s speed.

 

I am not sure if it's IMDisk related, it could be my old hardware or CPU speed/bandwidth limiting the time to transfer those files. I have tested RAMDisk and according to CrystalDiskMark transfer rates for 4K files should be around 250-300MB/s. To compare for my HDD it's 0.5-2MB/s.

 

IMDisk settings were: 384MB size; FAT32; default cluster size; no TEMP; no AWE. Changing cluster size/TEMP/filesystem did not make huge difference, enabling AWE caused 4K CrystalDiskMark to slow down to 50-60MB/s.

 

Is there anything I can do to speed up such operations on small files? I need especially read to make certain applications run smoother.

 

Cheers.

[kodiak567]

Small update, it seems that using DataRAM RAMDisk 4.3 RC1 shows same speeds, or perhaps they are even slower. But if it's not ImDisk issue I'd like to know what could be causing it and is there any way of getting around.

 

Also I have checked exact same file folder on SATA2 SSD on different, faster machine with Windows 7 and the very same folder copy speed from SSD to SSD was around 700KB/s.

[Wonko the Sane]

How EXACTLY are you copynig the files?

I mean, like selecting them in Explorer than copy/paste, using a file manager like 7-zip, using the COPY command, using the XCOPY command, using Robocopy or Strarc, etc,?

It i possible that *somehow* the issue lies with the number of files and some "directory listing" activity that happens in the specific command/tool.

If this happens with one method and not the other it may help in pinpoint the cause of the behaviour, though if an alternate tool/method is faster, just using it will workaround the problem easily.

As a side note whilst the rule of the thumb is that the simpler the filesystem is the faster *anything* should be in it (i.e. for a 384 Mb sized volume try using FAT16 instead) there are (seemingly unrelated) reports about FAT32 being extremely slow on XP (i.e. try using NTFS instead) see:
http://www.msfn.org/...d-on-usb-stick/

 

Wonko

[kodiak567]

Hey, thanks for answers.

First I want to clarify that it's not only copy command that I need. In fact what I tried to do was to install application that reads those small picture files 'on-the-fly' but due to their size and numbers it slows down and stutters horribly when doing so. I found out that just by dragging file folders in Windows Explorer I get similar slow performance and delay to that caused by mentioned application.

Now how do I copy? Beside Windows Explorer drag and drop, I've tried TeraCopy software and Total Commander. But I copied also using command copy and xcopy to find out differences. The sample is 12MB folder with 2300 *.png files using NTFS filesystem. Results are:

copy = NTFSHDD->NTFSHDD=008s(same disk)
copy = NTFSHDD->NTFSRAM=076s
copy = NTFSRAM->NTFSRAM=098s(same disk)

xcopy = NTFSHDD->NTFSHDD=007s+000s read(same disk, read is extra time taken to open directory)
xcopy = NTFSHDD->NTFSRAM=015s+025s read(read is extra time taken to open directory)
xcopy = NTFSRAM->NTFSRAM=011s+022s read(same disk, read is extra time taken to open directory)

For drag and drop and TeraCopy copy times were bit less than 120s to perform operations(including copying files on same disk), always bit faster for TeraCopy and always bit faster when done on HDD than on RAMDisk. Total Commander copied files on HDD with similar speed but when copying files on RAMDisk it slowed down to just 30KB/s, that is like 5 times slower than drag and drop.

Again, the copy using commands seem to be faster and snappier but the application that I am trying to run reads(caches) those picture files at speeds similar to basic drag and drop. I don't know about about "directory listing" in such scenario, there is just single folder with files inside there and Indexing Service is disabled on my XP machine. I do believe FAT32/NTFS/FAT doesn't matter in my case as I've tried them all and CrystalDiskMark gave almost identical results. I used FAT32 to be sure that application will be installed properly.

 

Perhaps it would be possible to simply cache those files somehow manually and then run application without need for RAMDisk?

Edited by kodiak567, 22 April 2016 - 10:37 PM.

[Wonko the Sane]

when you say "NTFSHDD->NTFSHDD" do you mean, two different volumes on the same hard disk or two different directories on the same volume?

If the latter most probably there is some particular (faster) algrithm in use, or if you prefer it is not NTFSHDD->NTFSRAM that is slower, but rather NTFSHDD->NTFSHDD that is faster.

 

More or less a copy operation might need to list (like DIR) all files in the source folder and in the target folder each time a file is copied, i.e. you might have 2x2300 additional operations, particularly when using a GUI program that *needs* to update the target window/view while an XCOPY like operation is more "in bulk" and is usually faster, this seems like confirmed by your results.

 

What about compressing the source?

I.e. making a .zip file containing the 2.300 .png's and extracting it to "target" or copy the .zip to the target and then expanding it?

 

Wonko 

[Uneitohr]

I.e. making a .zip file containing the 2.300 .png's and extracting it to "target" or copy the .zip to the target and then expanding it?

 

That would me my suggestion as well. That's what I do when having thousands of images.

 

But I would be more curious in finding an optimal way to transfer them without having to make an archive. The issue with the archive sollution is:

• takes time to archive, even if using no-compression zip archive
• takes double the space (archive + extracted files)
• is a 3-way operation (archive files, copy/paste archive to destination, unpack on destination)

[kodiak567]

when you say "NTFSHDD->NTFSHDD" do you mean, two different volumes on the same hard disk or two different directories on the same volume?

If the latter most probably there is some particular (faster) algrithm in use, or if you prefer it is not NTFSHDD->NTFSRAM that is slower, but rather NTFSHDD->NTFSHDD that is faster.

 

More or less a copy operation might need to list (like DIR) all files in the source folder and in the target folder each time a file is copied, i.e. you might have 2x2300 additional operations, particularly when using a GUI program that *needs* to update the target window/view while an XCOPY like operation is more "in bulk" and is usually faster, this seems like confirmed by your results.

 

What about compressing the source?

I.e. making a .zip file containing the 2.300 .png's and extracting it to "target" or copy the .zip to the target and then expanding it?

 

Wonko 

 

I've tried to just duplicate directory and content. First test was to copy on same partition, second was to copy to different partition on same drive. I do believe that beside some internal algorithm, those small files were most likely somehow cached hence the instant transfer speed.

 

As for compression, I do believe amount of time to perform decompression was same on HDD and RAMDisk.

[Wonko the Sane]

Well, if you explain with some more details what the ACTUAL issue is and the EXACT context, maybe we can give you some more ideas.

 

You could store the thousands of .png's in a volume image, that would be mounted almost immediately by IMDISK or similar software, or you could use the same volume image as a "repository" and access it through a third party tool only when needed, or use (I believe those .png's are read only) use a .zip container and use OS built-in (or again third party) libraries to access the single .png's when actually needed.

 

Wonko 

[kodiak567]

The real problem is with using an application that loads(caches I assume) loads of those small png files "on the fly" and horribly slows down/stutters application when doing so. It also takes some time to do it. I hoped that using RAMDisk would aid me with the problem(ie. let that caching be much faster) but it had no effect.

 

I found out that similar problem happens when copying those small files(exactly same ones application uses), over RAMDisk - no difference in file transfer speed. On the other hand copying bigger files is few times faster on the RAMDisk and all synthetic bechmarks like CrystalDisk or Atto are showing massive speed difference.

 

BTW, is it fine to temporary turn ImDisk service to manual in Windows Services panel? I am trying to find out what is causing very occasional BSOD of my machine so I need to disable as much as possible when doing so.

[Olof Lagerkvist]

BTW, is it fine to temporary turn ImDisk service to manual in Windows Services panel? I am trying to find out what is causing very occasional BSOD of my machine so I need to disable as much as possible when doing so.

 
You cannot disable/enable drivers in Windows Services. You only see user-mode services there so what you have seen there is just a helper service that is used when ImDisk connects to other machines over network.
 
What you are looking for here is probably rather to stop drivers related to ImDisk.

net stop imdisk
net stop awealloc

Drivers will automatically load again at next reboot. If you want to disable that auto-load too (completely disable the drivers), use the following commands:

sc config imdisk start= disabled
sc config awealloc start= disabled

[kodiak567]

Thanks, found out BSODs are not related to ImDisk.

 

Anyway I can run this Flash application through website and then it content gets cached to HDD just like normal browser cache. I have tested it on portable Firefox 3.6.28 version but without any success. When Firefox caches those small *.pngs it makes it own files in which *.pngs are contained. They are bit bigger and can't be opened directly. AFAIK other browsers don't do it this way and save files like they were on found server. Anyhow application takes even longer to open those files - matter of seconds but tested with timer.

 

I might wonder, did anyone do any tests of browser performance(site loading speed, esp. one with many images) when they get cached from RAMDisk compared to HDD?

Edited by kodiak567, 04 May 2016 - 01:43 PM.