33 replies to this topic

[psewar]

Hello Olof,

first I want to thank you very much for your great tool/driver.

As the topic title already states I want to propose a feature which is stated often here in this forum as a problem, but it seems there is no real solution to this so far.
Sadly I have no knowledge about driver development so I can't really say if it will work like this but from Windows application programming I know that an Application will normally get a message from Windows if it has to be closed. So I'd guess that a driver also gets a notification to do some work before getting closed / shutdown.

What would be the benefit of such a feature you may ask?
I recent bought a SSD disc because the old hard disk was getting slower and provoked freezes during my work.
One of the biggest problems with SSD disks is still that many writes will cause the lifetime of the disk to decrease, so I thought it would be a good idea to use a ramdisk to store the temp folders like Windows temp, temporary Internet files etc in it, as those folders are used very often and would cause many writes to the SSD.
For real temporary files it is no problem if they are getting deleted during a shutdown, but there are many cases in my opinion where files are some sort of "half-temporary". Let's take the temporary internet files as an example:
They are stored on the disk because it will boost the performance during browsing the web.
If they are deleted during a reboot it is not a real problem, because they are only temporary as the name already states. But normally you will visit the same pages over and over again. So that means that after every reboot all the files which a browser wrote on the disk for performance reasons are gone and have to be loaded again from the internet vanishing the reason why those files where created.
There are probably even better examples of why a ramdisk loaded from an image should write back the changes to that image file, but I think my example also showed why there is a reason to do this.
Of course this should not be done for every ramdisk loaded from an image file, as there are also cases where you want the files to get deleted during a reboot.

So please find here my proposal:
- Add a new registry key: SaveDriveX (where X represents the number system already used to load the drives)

The driver will check on the close/shutdown notification the presence of the mentioned registry key, and as a faulty check also checks if the mentioned drive is really a memory disk loaded from an image file. On success it writes back to contents of the ramdisk to the file mentioned in the registry for that drive.

So kind of a pseudo code for that feature would be like:

OnShutdownNotification(object sender, EventArgs e)
{
foreach(Drive drv in RegisteredDrives)
{
if(drv.IsMemoryDisk && !string.IsNullOrEmpty(drv.FileName))
{
File imageFile = GetFileReference(drv.FileName);
// perhaps add here some Code to remove all references for that file to prevent locking issues
lock(imageFile)
{
SaveImageDisk(imageFile);
}
}
}
}

As I already wrote I don't know if this is possible on a driver level, but as it seems to be possible to load an image file from a disk during the boot process I'd guess that it should also be possible to write back changes to that image file during a shutdown.
I think many people which state similar problems would be very thankful if you would consider adding this feature.

With best regards,
psewar

[amalux]

Hi psewar,

I agree, such a feature (if possible) would be highly useful to many users. Currently, settings and registry changes need to be handled on a per program basis using scripts, saving data to a writable drive and reloading at startup. If all changes of this nature could be handled globally, this would be a great improvement! Thanks for making this suggestion.

[sbaeder]

Hi psewar,

I agree, such a feature (if possible) would be highly useful to many users. Currently, settings and registry changes need to be handled on a per program basis using scripts, saving data to a writable drive and reloading at startup. If all changes of this nature could be handled globally, this would be a great improvement! Thanks for making this suggestion.

I think there are a couple of issues here...

First, an SSD only has issues if the same CELL is written repeatedly too many times. This is why they have very sophisticated controllers, so even if a lot of files are written and deleted, it makes sure to NOT re-use the same cells over and over if at all possible. Now if you are 99% full of static content and keep beating on the remaining 1%, not a lot it can do...But this is why there is TRIM, and performance of an SSD can change as more content is "stored" (see recent article at HardOCP on this.

But let's put the reason why you think it is needed aside...There is all the issues with what sort of "guarantee" to you make. Since "hibernation" of the system writes the ram disk to the drive, can you just use that? Notice, it doesn't make any real guarantees and has easy way to bypass restore and reboot "from scratch", and that is the issue for any sort of "ram" disk...How do you manage it? What happens if the power goes out? Or I just hold in the power key until the machine shuts down.

Yes, as you outline above, you COULD probably put in a process to save the contents (either on shutdown or on command), and it might be worth doing...as long as the users understand it is still "temporary" and that "stuff happens"

[psewar]

Hey sbaeder,

I agree with you that the SSD example has it's flaws also, but yeah my thoughts of somehow circumvent the "flaw" of rewriting the same cells again and again with data lead me to think about using a ramdisk to store temporary and "half-temporary" files. So I used it as an example why it can be useful.

And you are also right that there is no "guarantee" that it will always be safed, else you would need some kind of logic which writes the data buffered back to the disk image which is nearly impossible to make, as the ways in which a ramdisk is used are just to numerous to make it work for everyone.
But yeah I do not want any guarantee that the data there is always saved.
I mean not even Microsoft gives me any guarantee that Windows will still work after a power outage.

As amalux already pointed out we currently have to do workarounds with scripts and whatever to solve this issue, so why not solve it there where it somehow logically should be made possible?
I mean the first thing I searched in Imdisk after creating a ramdisk from an image file is where I can click a check box that the content should be written back on a shutdown. And I think many other people were thinking the same, at least that was my impression after reading through many threads here about writing back the content of an ramdisk.

So I think this would really be a big pro for imdisk if it would allow such a thing.
I also know from many gamers which would love to have the possibility to move all the temporary files from their game to a ramdisk and do not have to care with scripts that they are being written back after they stop playing or rebooting.
So if imdisk would offer such a feature I can imagine that a lot of gamers will install and use it, leading to a bigger "target group" for imdisk.

Cheers,
psewar

[Olof Lagerkvist]

Thanks for your ideas and suggestions!

I prefer to do things like this in user mode and not in the driver. Even though there are some kind of shutdown notifications for drivers they are a little bit tricky to use for other things than the driver itself. So, once the driver gets a notification it could very well be already impossible to save anything because the disk driver may already have shut down, etc.

So, I would suggest that you either do like I personally do in almost all cases like this, schedule a script to run at system shutdown that saves the disk image. More specifically I almost never save a disk image but rather mount the original disk image as another virtual disk and use robocopy or something to synchronize them. This usually saves lots of time and causes fewer disk writes. Or, you can write your own service and respond to shutdown events with something like your pseudo code and use ImDisk .NET API to save the image file.

[thorc]

Hi!

Imdisk looks great, however I would be great with save function.
Save features:
* Save an ramdisk that is in use in an intervall (only changes)
* Save an ramdisk in real time (look at vsuite ramdisk)

Real time scenario is really good if you have files that really never change, but when they do you have the comfort to change it easily.

[dog]

@thorc
Robocopy has options for interval copying:
/MON:n :: MONitor source; run again when more than n changes seen.
/MOT:m :: MOnitor source; run again in m minutes Time, if changed.

@psewar
If gamers don't start with a bit of scripting, where will the next generation of programmers come from?

[thorc]

@thorc
Robocopy has options for interval copying:
/MON:n :: MONitor source; run again when more than n changes seen.
/MOT:m :: MOnitor source; run again in m minutes Time, if changed.

thanks. wasn't aware of those switches. how do you end a monitoring job then?

I was thinking perhaps using powershell for the sync. like this:
http://blog.usepower...rom-powershell/

[psewar]

Hey there,

it took a while until I had everything set up.
Currently it works more or less, sadly it does not always save the image during the shutdown process.
Probably a timing problem, that the SaveImageFile function is interrupted by the shutdown, as it seems to do the save somehow asynchroniously.

So I tried to create my own save process so that I can also give a progress about it.
But sadly the GetRawDiscStream() does not return the whole image.
As the image file is exactly 512.000.000 bytes but the methods filestream misses 4096 bytes.
I also tried the SafeFileHandle Property to get the stream with the same result.

Do you know what the problem there could be?

Cheers,
psewar

[Olof Lagerkvist]

So I tried to create my own save process so that I can also give a progress about it.
But sadly the GetRawDiscStream() does not return the whole image.
As the image file is exactly 512.000.000 bytes but the methods filestream misses 4096 bytes.
I also tried the SafeFileHandle Property to get the stream with the same result.

Do you know what the problem there could be?

Yes, this sounds familiar. SaveImageFile() used to have that problem as well, but after a discussion in this forum we added a call to FSCTL_ALLOW_EXTENDED_DASD_IO to make filesystem drivers skip filtering direct I/O but rather sending requests directly down to disk driver level. I think I need to add that to ImDiskDevice objects too.

Will fix that soon!

Edit: You can try it now. http://www.ltr-data....es/ImDiskNet.7z

[psewar]

Cool thanks for the fast reply Olof, will try it out tommorrow after work.

Do you have an Idea where the missing 4096 bytes may come from when accessing the filestream directly?
I thought it might be that the Filestream I recieve from GetRawDiscStream may not be returning the NTFS table or something like that.
But yeah that is just a guess. Perhaps this 4mb are also some kind of overhead from the imagefile itself which of course is not part of the Filestream then.

Will keep you updated about the service.

[Olof Lagerkvist]

Do you have an Idea where the missing 4096 bytes may come from when accessing the filestream directly?

I think it had to do with different geometry calculation i filesystem driver vs disk driver, or something like that. There is different number of missing bytes for different disk sizes. I think I found some articles about it when I searched for a solution but no solid solution was found until someone posted the suggestion to use FSCTL_ALLOW_EXTENDED_DASD_IO.

[pscEx]

Edit: You can try it now. http://www.ltr-data....es/ImDiskNet.7z

@Olof:
I would VERY appreciate, when you publish imdisk 1.4.3 or 1.5.0 containing this feature. I could use it (better said: I need it) in one of my projects.

Source code access does not help me here.

Peter

[Olof Lagerkvist]

I would VERY appreciate, when you publish imdisk 1.4.3 or 1.5.0 containing this feature. I could use it (better said: I need it) in one of my projects.

Next version will absolutely include this fix.

Source code access does not help me here.

It is not source code, it is a compiled DLL with XML documentation! Have you tried it?

[psewar]

Hey Olof,

I just downloaded and installed the new version you uploaded, now I get the following error message when trying to query a device:
Unable to find an entry point named 'ImDiskQueryDevice' in DLL 'imdisk.cpl'.

So I downloaded again the full imdisk and reinstalled it but without success.
Perhaps you compiled the .Net dll with a newer version of the imdisk.cpl so the entry points are not machting correctly?

Edited by psewar, 28 September 2011 - 06:28 PM.

[Olof Lagerkvist]

Hey Olof,

I just downloaded and installed the new version you uploaded, now I get the following error message when trying to query a device:
Unable to find an entry point named 'ImDiskQueryDevice' in DLL 'imdisk.cpl'.

So I downloaded again the full imdisk and reinstalled it but without success.
Perhaps you compiled the .Net dll with a newer version of the imdisk.cpl so the entry points are not machting correctly?

Sorry, you also need to replace imdisk.cpl with latest beta: http://www.ltr-data....i386/imdisk.cpl or http://www.ltr-data....md64/imdisk.cpl or http://www.ltr-data....ia64/imdisk.cpl depending on your architecture.

You are right, dll is no longer compatible with older cpl versions.

[psewar]

Hello again

So I'm back with an other problem.
I downloaded all the beta cpl you mentioned and copied the 32 bit one to the system32 folder, the intel64 to the systemWOW64 folder but recieved then the following error:
An attempt was made to load a program with an incorrect format. (Exception from HRESULT: 0x8007000B)

If I understand that error right it means that a 32bit program wanted to start a 64bit dll, or the opposite.
So I tried to copy the 64bit dll to the sys32 folder, the 32bi cpl version to the wow64 folder and the same with the amd 64 bit version, but with no success. I always kept getting the message mentioned above.

Do you have an idea what the problem could be?

[Olof Lagerkvist]

Hello again

So I'm back with an other problem.
I downloaded all the beta cpl you mentioned and copied the 32 bit one to the system32 folder, the intel64 to the systemWOW64 folder but recieved then the following error:
An attempt was made to load a program with an incorrect format. (Exception from HRESULT: 0x8007000B)

It should be the other way around. 64 bit binaries should always be in system32 and 32 bit binaries should always be in syswow64.

So I tried to copy the 64bit dll to the sys32 folder, the 32bi cpl version to the wow64 folder and the same with the amd 64 bit version, but with no success. I always kept getting the message mentioned above.

Do you have an idea what the problem could be?

DId you still get the same exception "... incorrect format..." when you put 64 bit cpl in system32 and 32 bit cpl in syswow64? Surprises me in that case. I don't know what it could be. Could you just double check that you have put everything in correct places?

[psewar]

Hey Olof,

you were right of course right.
64bit binary in System32 and 32bit binary in SysWOW64.
After I restarted my machine it worked like a charm, perhaps I played around a bit to much.

Thank you very much for your kind assistance

Edited by psewar, 29 September 2011 - 12:05 AM.

[thorc]

Using above method, am I right at you will save the whole thing to image on disk?
so if you allready have a image, you are writing everything even though perhaps only a small percentage is changed on the image file.

I feel you save more IO by using a differential sync by robocopy or syncframework.

[Olof Lagerkvist]

Using above method, am I right at you will save the whole thing to image on disk?
so if you allready have a image, you are writing everything even though perhaps only a small percentage is changed on the image file.

I feel you save more IO by using a differential sync by robocopy or syncframework.

I agree. I have built a few similar solutions running in various places and I have used a script that mounts original image file as a separate drive and synchronize between RAM disk and mounted original image. After that is done it just dismounts original, now modified, image file again and upon next startup RAM disk is loaded from the modified image file. This feels quick and robust and something I would like to recommend as well but some people just prefer image copying instead.

[psewar]

Using above method, am I right at you will save the whole thing to image on disk?
so if you allready have a image, you are writing everything even though perhaps only a small percentage is changed on the image file.

I feel you save more IO by using a differential sync by robocopy or syncframework.

That was also a concern of me, that's why I write my own Save method which only writes back the changed bytes to the image file.
This already works quit well.

The only Problem I still have no solution for is the strict Shutdown Process since Windows 7 which just ignores requests for more time and will kill my process if not shut down in around 5 seconds. -.-

[sambul61]

Sorry for interrupting this nice conversation, but as a "relative novice" to just about everything, I get a bit lost in this topic (and possibly many other readers):

1. The topic is started in ImDisk section. Does it imply that ImDisk currently has a feature "Create Memory Disk"? OR, the Memory Disk is expected to be auto created or read from a HD image file on startup by other means - which ones?
2. Does the topic started ask to add "Save Memory Disk to Image File" feature and option to RMC ImDisk Menu?
3. What this feature has to do with ImDisk? From Olof's reply, a link can be found if ImDisk upon user selecting the above option will mount previously saved Memory Disk Image, then synchronize it with current Memory Disk (with a 3-d party sych tool) and then dismount the image - is that right?
4. OR, instead of user selection, ImDisk would perform the synch automatically on system shutdown, if Win 7 allows enough delay to finish the job - correct?

That would be a nice ImDisk feature to have, if my assumptions are right. Btw, doesn't eBoostr do similar job in terms of updating its RAM Disk mirror on a hard drive at Win shutdown?

[psewar]

Hey Sambul,

no problem I'll try to answer your questions:

1. Yes ImDisk provides a feature to create Memory Disks. It can create a blank one or create one from an image file.

2. ImDisk already provides an option to save back the memory disc to the image file. I created the topic to ask Olof if it is possible to automate this process during shutdown.

3. The reply from Olof was to the poster before him which asked if it would not make more sense to just replace the files which changed instead of writing down the whole image again to the harddisc.

4. Yes the Windows Service which I'm currently working on will automate the process of saving back the ramdisc to the image during shutdown.

[sambul61]

Thanks. So, I assume ImDisk would need to be called from a batch file at startup with certain parameters to auto create a Memory Disk from a previously saved image file?