334 replies to this topic

[v77]

Version 20140124
- dynamic ramdisks: write speed improvement
- fix: value 0 to disable dynamic ramdisk cleanup was not working
 

 

Code of dynamic ramdisks is now improved and cleaned. I also made a few more tests and it seems reliable enough to be considered as stable.

 

However, because of the inherent issue of this feature, that is, the fact we have to be careful about the content of the ramdisk in order to keep enough memory for the system, I think to keep the option unchecked by default.

 

About the available memory, if memory cannot be allocated or if there is less than 100 MB of memory available (swap file included), an error message is displayed. This value may seem a bit high but tests on my current main system (8.1) showed issues even with 50 MB free.

[v77]

Hi!

 

Can You add feature for RAMdisk UI to create them as mountpoints?

 

This would have been interesting, but unfortunately, it will be difficult to format the volume without a drive letter.

However, on Windows Vista and later, you can redirect a folder content into a ramdisk by defining a symbolic link to the ramdisk volume with the command mklink /d (type mklink /? in an elevated command prompt for details).
So, for example, if you want to redirect a folder like D:\my_folder to a ramdisk R:, you can try in an elevated command prompt (D:\my_folder must not exist):
mklink /d D:\my_folder R:
You can even hide the drive letter of the ramdisk in Explorer by using the "General Settings" of ImDisk Toolkit available in the Start menu.

[Olof Lagerkvist]

This would have been interesting, but unfortunately, it will be difficult to format the volume without a drive letter.

However, on Windows Vista and later, you can redirect a folder content into a ramdisk by defining a symbolic link to the ramdisk volume with the command mklink /d (type mklink /? in an elevated command prompt for details).
So, for example, if you want to redirect a folder like D:\my_folder to a ramdisk R:, you can try in an elevated command prompt (D:\my_folder must not exist):
mklink /d D:\my_folder R:
You can even hide the drive letter of the ramdisk in Explorer by using the "General Settings" of ImDisk Toolkit available in the Start menu.

 

Just FYI, on 2000, XP and Server 2003, where mklink is not included, you can use my "junc.exe" tool to do the same:

junc D:\my_folder \??\R:

 

or, directly to the root directory on the ImDisk device:

junc D:\my_folder \Device\ImDisk0\

 

Junc.exe is available here:

http://ltr-data.se/files/junc.zip

 

Another idea. I have found that format.com, chkdsk.exe and a few others accept a few more things than A-Z drive letters. For example, I have a script where I create an ImDisk drive, mount it to a subdirectory and then formats it using a temporary drive letter. I create and remove this temporary drive letter with my dosdev.exe tool before and after the script calls format.com to format the virtual disk. As drive letter, to make sure I don't interfere with any existing drive letter, I use Ö: which, for some yet unknown reason, actually works with format.com. But Ä: does not work, for example.

[v77]

Another idea. I have found that format.com, chkdsk.exe and a few others accept a few more things than A-Z drive letters. For example, I have a script where I create an ImDisk drive, mount it to a subdirectory and then formats it using a temporary drive letter. I create and remove this temporary drive letter with my dosdev.exe tool before and after the script calls format.com to format the virtual disk. As drive letter, to make sure I don't interfere with any existing drive letter, I use Ö: which, for some yet unknown reason, actually works with format.com. But Ä: does not work, for example.

 

Yes, I could indeed use DefineDosDevice... Thanks for the tip.
About the drive letter Ö: and some others, this does not look very official...   I can find no information about that.

[Olof Lagerkvist]

About the drive letter Ö: and some others, this does not look very official...   I can find no information about that.

Of course. I guess that's just one of those "happens to work now but could change at any time". But after all, it has been around for many Windows versions now.

[rebootpro****you]

v77, thank you for your awesome utility, I've been waiting for a FOSS ramDisk that would support dynamical memory occupation. But could you, please, add 1 more tiny feature? The thing is that your utility works by timer and when the timer hits 0 - it automatically frees up some memory if possible. Always. And that process eats some CPU resources, but the thing is that it's not necessary to do in most cases, when it frees up very little RAM. Could you, please, add an option that would make your utile not free up RAM in case the amount of freed up RAM is insignificant (and let user choose the amount which would be regarded as the maximum insignificant either in mb or in percents)? I have 4Gb ram disk, but usually it's occupied by only ~350Mb (firefox's cache files + temporary downloads, etc.), and the amount of data usually vary, like this: 0Mb > 150Mb > 300Mb > 350Mb > 300Mb > 280Mb > 320Mb > ... and ramDisk will always try to fit the size of occupied RAM to the used disk, but in general, in the above example - I don't mind if it remains 350Mb even when the amount of data stored on that disk later decreased to 300Mb or 280Mb or 320Mb.

[v77]

v77, thank you for your awesome utility, I've been waiting for a FOSS ramDisk that would support dynamical memory occupation. But could you, please, add 1 more tiny feature? The thing is that your utility works by timer and when the timer hits 0 - it automatically frees up some memory if possible. Always. And that process eats some CPU resources, but the thing is that it's not necessary to do in most cases, when it frees up very little RAM. Could you, please, add an option that would make your utile not free up RAM in case the amount of freed up RAM is insignificant (and let user choose the amount which would be regarded as the maximum insignificant either in mb or in percents)? I have 4Gb ram disk, but usually it's occupied by only ~350Mb (firefox's cache files + temporary downloads, etc.), and the amount of data usually vary, like this: 0Mb > 150Mb > 300Mb > 350Mb > 300Mb > 280Mb > 320Mb > ... and ramDisk will always try to fit the size of occupied RAM to the used disk, but in general, in the above example - I don't mind if it remains 350Mb even when the amount of data stored on that disk later decreased to 300Mb or 280Mb or 320Mb.

 

I would like, but I don't know in advance how many RAM will be freed. When you copy and delete a file, the system will say that you have as much free space than at the beginning. So I cannot rely on this to start the cleaning.
An interesting experiment could be to disable the cleanup (with the value 0 in the Advanced tab) and measure the amount of memory allocated by RamDyn.exe after a given time. If you put your browser cache into the ramdisk, it should not take long before getting a high value.

That said, I could try to make the thing smarter by looking at the amount of written data... For instance, it is obvious that if there is no written data after a cleanup, the next one will be useless.
Anyway, thanks for your feedback.

[rebootpro****you]

I would like, but I don't know in advance how many RAM will be freed.

Uhm, sorry, but I don't get why.
That seems quite trivial to me, since the amount of occupied disk's space is the accessible data, so when a timer fires up your utility should store the value of current data stored on RAM disk.
And further, whenever the timer fires up - the utility should get get the new value and then compare it with the old one:

If (oldValue < currentValue) ; If old value is lower than the current value
	oldValue := currentValue ; assign currentValue to oldValue.
Else If (oldValue == currentValue) ; If values are equal
	return ; do nothing, a cleanup is not needed.
Else ; The current value is lower than the old value, thus there is some memory to free up, but maybe that's not necessary.
{ ; and here goes the code where the decision gets taken: whether to initiate memory freeing up or not.
; That can be done either by subtraction (in case you'd make your setting accept absolute values)
; or by division (in case you'd make your setting accept percents, instead of absolute values).
	If (oldValue - currentValue) > 300Mb ; 300Mb is the absolute value specified by user in the settings.
		cleanUp()
	Else
		Return
}

[v77]

the amount of occupied disk's space is the accessible data

 

Yes, but this is not the amount of allocated RAM. If you create a file and then delete it, the occupied disk space remains the same, but RAM have been allocated for the space required by the file.

Memory is allocated by the proxy when it receives a write request at a position where memory is not already allocated. But I don't know if the written bytes are for a file that is created, deleted, or if this is an operation on a folder, on the file system, etc.

Of course, I can use the fact that, between two cleanups, if there is a lot of new allocated blocks of memory and the free disk space is almost the same, there is a high probability that files have been deleted and so, a cleanup is required. But this is not necessarily the case, because of the file fragmentation, because of the fact that memory is allocated by block of 1 MB, because some little files can be moved, etc.

[rebootpro****you]

Yes, but this is not the amount of allocated RAM. If you create a file and then delete it, the occupied disk space remains the same, but RAM have been allocated for the space required by the file.

Yeah, you are right, I forgot that. So there should be another loop/timer that checks the current occupied disk space more frequently (like once per second) and overwrites the variable "oldValue" in the example above if the current value is greater than the one stored in "oldValue" variable.
So in case something like this happens:

[250Mb]		>	[200Mb]		>	[500Mb]		>	[248Mb]
   ^									   ^
1st fire up of the main timer					2nd fire up of the main timer

then the more frequent timer will correct the main timer's data about the previous state and will actually let it know that there was some leap in the amount of saved data, so that in the above case the cleanup would actually free up 252Mb, not just 2Mb.

[v77]

Another idea. I have found that format.com, chkdsk.exe and a few others accept a few more things than A-Z drive letters. For example, I have a script where I create an ImDisk drive, mount it to a subdirectory and then formats it using a temporary drive letter. I create and remove this temporary drive letter with my dosdev.exe tool before and after the script calls format.com to format the virtual disk. As drive letter, to make sure I don't interfere with any existing drive letter, I use Ö: which, for some yet unknown reason, actually works with format.com. But Ä: does not work, for example.

 

One more thing about that. In fact, the main issue may be not the fact that it is not documented. I just found out that the available "special" drive letters change depending on the language of the system. For instance, the letter you are using is usable on a french version of Windows XP, but not on an english one. I also tested on a chinese version of Windows 8 and it does not work either.

I can just take the first available drive letter, but at the system boot, this seems a bit hazardous.

As a result, I think to let the user choose the drive letter he want.  But I really dislike this: making the interface heavier because of a technical issue...

[Olof Lagerkvist]

One more thing about that. In fact, the main issue may be not the fact that it is not documented. I just found out that the available "special" drive letters change depending on the language of the system. For instance, the letter you are using is usable on a french version of Windows XP, but not on an english one. I also tested on a chinese version of Windows 8 and it does not work either.

I can just take the first available drive letter, but at the system boot, this seems a bit hazardous.

As a result, I think to let the user choose the drive letter he want.  But I really dislike this: making the interface heavier because of a technical issue...

 

Yes, I have seen that too about such characters as drive letters. But from I can see, there seems to be a few ones that could be used with any language. For example UTF16 character 255, at least if it is passed as Unicode to format.com. I think it needs to be tested with a few more languages though, but this far I am optimistic about it.

 

Another and more urgent reason to implement this in a reasonably safe way in imdisk.exe is the problem with wrong drive formatted. That has been reported now and then both in this forum and in e-mails to me. The problem occurs when people use imdisk.exe to "overwrite" an existing drive letter with an ImDisk one and having the ImDisk drive formatted directly through imdisk.exe. In that case imdisk.exe nowadays tends to format the old existing drive with this letter instead of the new ImDisk one. There have been quite some loss of data in some places where people have by mistake formatted their external 1 TB USB HD or similar instead of their newly created ImDisk memory disk.

 

This became an issue as a kind of side-effect when we solved the drive letter appearance problems a few versions back. Since only one drive letter link is created nowadays and the rest handled by shell notifications, the correct drive letter association might not work early enough for format.com to pick up the new link, Or something similar, I have not yet figured out the exact reason why this happens. But anyway. If we could run format.com with a temporary drive letter that is never in use in any other case, this issue would be solved as well.

[v77]

Another and more urgent reason to implement this in a reasonably safe way in imdisk.exe is the problem with wrong drive formatted. That has been reported now and then both in this forum and in e-mails to me. The problem occurs when people use imdisk.exe to "overwrite" an existing drive letter with an ImDisk one and having the ImDisk drive formatted directly through imdisk.exe. In that case imdisk.exe nowadays tends to format the old existing drive with this letter instead of the new ImDisk one. There have been quite some loss of data in some places where people have by mistake formatted their external 1 TB USB HD or similar instead of their newly created ImDisk memory disk.

 

Yes, I too have experienced this with a bad test... But there is a simple solution that I just implemented: checking before formatting that there is no valid file system in the volume. I use GetVolumeInformation without information to retrieve. If this fails with ERROR_UNRECOGNIZED_VOLUME, this means that it should be okay for formatting in most cases.

 

This became an issue as a kind of side-effect when we solved the drive letter appearance problems a few versions back. Since only one drive letter link is created nowadays and the rest handled by shell notifications, the correct drive letter association might not work early enough for format.com to pick up the new link, Or something similar, I have not yet figured out the exact reason why this happens. But anyway. If we could run format.com with a temporary drive letter that is never in use in any other case, this issue would be solved as well.

 

In fact, the notifications do not work at all when the drive letter already belong to a physical drive. However, the notifications work in the case of a device removal. It seems to be a simple protection of the disk manager...
As it could be difficult to make a difference between the protected drive letters and the others, my suggestion to check the existence of a file system could be interesting.
Anyhow, it seems to me that using a weird drive letter will give us more problems than solutions: the issue of the system language, the fact that the drive letter you will choose could be used by another software... I recently read some things about a software that use $:...

[Olof Lagerkvist]

Yes, I too have experienced this with a bad test... But there is a simple solution that I just implemented: checking before formatting that there is no valid file system in the volume. I use GetVolumeInformation without information to retrieve. If this fails with ERROR_UNRECOGNIZED_VOLUME, this means that it should be okay for formatting in most cases.

Yes, that ensures that we don't format an existing volume. But it does not format the new one either.

In fact, the notifications do not work at all when the drive letter already belong to a physical drive. However, the notifications work in the case of a device removal. It seems to be a simple protection of the disk manager...
As it could be difficult to make a difference between the protected drive letters and the others, my suggestion to check the existence of a file system could be interesting.
Anyhow, it seems to me that using a weird drive letter will give us more problems than solutions: the issue of the system language, the fact that the drive letter you will choose could be used by another software... I recently read some things about a software that use $:...

Of course this is not an ideal solution. Specifically, detection of usage collision with other software is a complex part of it. It is more of an idea so far, but I will continue spending some work on it, to see if it could lead to a reasonably robust solution.

[v77]

Yes, that ensures that we don't format an existing volume. But it does not format the new one either.

 

Yes, but in this case, formatting is not very useful, since the volume will be unreachable for the user... So, the user has first to change his settings. But at least, his data will not be erased.

By the way, my test on an english XP with all the weird possible drive letters has shown interesting things:

 

E:\>format ╣:
The type of the file system is NTFS.

WARNING, ALL DATA ON NON-REMOVABLE DISK
DRIVE ╣: WILL BE LOST!
Proceed with Format (Y/N)? y
Verifying 20465M

 

E:\>format ╤:
The type of the file system is RAW.
Please use the /FS switch to specify the file system
you wish to use on this volume.

E:\>format ≤:
The type of the file system is RAW.
Please use the /FS switch to specify the file system
you wish to use on this volume.

This is a simple virtual machine with a CD drive and a network drive, and the C: drive is 20 GB... So yes, using these weird drive letters is really dangerous.

This is demoralizing... For now, I keep the ugly solution: add an option to let the user choose the temporary drive letter.

Edited by v77, 06 February 2014 - 01:32 AM.

[v77]

If someone is looking for informations about that:
a strange bug occurs when a dynamic ramdisk is defined on Windows XP or Vista: in some cases, the system shutdown is blocked.
This occurs when the user creates the ramdisk manually. When the proxy is started by the service at startup, the bug does not occur. It is invariable, and only with XP and Vista.

Even if it is reproducible, circumstances are so weird that it might take time to fix it.

However, my ProxyCrypt tool, which uses the same mechanism to close the volume at shutdown, does not seem to be affected by this bug. So, there is probably something to do.

[Olof Lagerkvist]

Yes, but in this case, formatting is not very useful, since the volume will be unreachable for the user... So, the user has first to change his settings. But at least, his data will not be erased.

But this would also make it impossible to use imdisk.exe to re-format an image that already has a filesystem. I have for example in some of my build&test-scripts a few cases where I mount an image a few GB in size. Some of the iterations first format this image to create a clean filesystem to work with. That would not have worked if imdisk.exe would have first checked if the virtual disk already contains a filesystem. Alright, I could easily re-write my scripts to call format.com afterwards or simply delete the existing image file before, but anyway, it is a case where it will be incompatible with existing behaviour.

[v77]

But this would also make it impossible to use imdisk.exe to re-format an image that already has a filesystem. I have for example in some of my build&test-scripts a few cases where I mount an image a few GB in size. Some of the iterations first format this image to create a clean filesystem to work with. That would not have worked if imdisk.exe would have first checked if the virtual disk already contains a filesystem. Alright, I could easily re-write my scripts to call format.com afterwards or simply delete the existing image file before, but anyway, it is a case where it will be incompatible with existing behaviour.

 

Yes, I forgot this case. We can indeed assume that a ramdisk contains nothing when it is created, but we know nothing about the content of image files.
But how about simply check the existence of the drive letter before mounting the imdisk one? Even with the issue of the different namespaces, this should not be a problem because a drive letter that is not visible to imdisk.exe will not be visible to format.com.

 

By the way, I found a fix for the bug described just above. It seems that XP and Vista send no message at shutdown to the windows created by console applications. But unfortunately, I compiled RamDyn.exe as a console application to avoid some false alerts of antivirus. When compiled as a GUI application, the only way I found to remove the false alerts is to put back the CRT library of MinGW, but the size of executable goes from 8.5 KB to 18 KB. This will be probably my "fix", until I find something better.

[v77]

Version 20140207
- added the ability to define ramdisks on mount points
- changed a tooltip in RamDiskUI.exe to tell that image files are not compatible with dynamic ramdisks
- fix: when a dynamic ramdisk was defined on XP or Vista, system shutdown could be blocked in some cases
- minor fixes and improvements

 

 

This new version brings the ramdisks on mount points. Up to 10 mount points can be registered to be mounted at Windows startup, I hope this will be enough...
They can be defined in the Advanced tab of the RamDisk Configuration Tool.

Previously discussed improvements about dynamic ramdisk cleanup are still planned for a future release.

 

 

Edit: Forgot to update the version displayed by the installer, this is now fixed.

Edited by v77, 07 February 2014 - 08:23 PM.

[v77]

Version 20140221
- dynamic ramdisk improvement: release of allocated memory does no longer rely only on a timer but can now be tuned by several parameters
- fix in RamDyn.exe: cleanup of dynamic ramdisk with FAT32 file system bigger than 4GB could take an infinite time
- RamDyn.exe: reduced process shutdown priority for applications that need to access a dynamic ramdisk at shutdown
- minor fixes and improvements

 

 

Here is the planned improvements about the dynamic ramdisk cleanup, in order to avoid some unwanted or unnecessary CPU loads.

The cleanup function now checks the variation of the difference between the allocated space and the space taken by the files and the file system. This takes into account several things, including the new files and the partially used memory blocks. However, the actually freed space can be slightly higher or lower than expected because I still don't know if a partially used memory block is used for deleted files, non-deleted files or both.

That said, the behavior now seems very correct, and the function can be tuned if needed by 4 parameters, available with a new dialog box in the Advanced tab of the RamDisk Configuration Tool.

[rebootpro****you]

v77, thanks a lot!
A few questions though:
1. how to properly update your utility? Am I right that last 2 updates didn't require the driver update?
2. What does this mean? https://i.imgur.com/WmVPXMy.png What are the units there or is it supposed to mean thousandths (i.e. 1 = 0.1%)? Or is it in megabytes? Then what's the 2nd number is for? Or is it 20mb per 1000 mb, thus if you have a 4000mb disk then those 20/1000 will mean that the minimal amount would be 80Mb?
Sorry, it's just a bit confusing since units are not mentioned.
3. What is "Security margin"?

Maybe it's worth adding the descriptions of those new features right into the GUI?

EDIT: oops, disregard the above, seems like there are the details in the GUI, they are in the popups that appear when input field is hovered.

Edited by rebootprofuckyou, 24 February 2014 - 07:31 PM.

[v77]

1. how to properly update your utility? Am I right that last 2 updates didn't require the driver update?

 

You just have to run the installer. If the driver does not need to be updated, it is automatically unchecked (this speeds up a bit the install).
However, if the driver needs to be updated, all ImDisk volumes must be unmounted. And if you are using a dynamic ramdisk, it must also be unmounted to be updated. This is not handy if you have defined your TEMP folder onto this ramdisk, because the installer uses this folder to uncompress itself, so you have first to redefine the environment variables.
I would like to improve that, but there is no obvious solution, at least with the decompression module that I am using (7zS2.sfx).
Anyway, in most cases, you will get an error message if something cannot be updated.

 

By the way, if you need to downgrade to a previous version, in some cases there might be some issues, therefore I would recommend to do first a complete uninstall.

 

 

EDIT: oops, disregard the above, seems like there are the details in the GUI, they are in the popups that appear when input field is hovered.

 

Yes, like for some fields in the other dialog boxes. I hope the explanations are clear enough.

[ericgl]

V77,

We need to find a way to update the ImDisk Toolkit in an elegant way.

Having my TEMP folders in RamDisk is a problem when driver needs to be updated.

Also, now that I have set the RamDisk to be dynamic, it's also a problem to update it, even if the driver itself doesn't need updating.

 

Maybe during the update process we can point the TEMP folders to another temporary location (C:\temp), and then point them back to the RamDisk.

[v77]

V77,

We need to find a way to update the ImDisk Toolkit in an elegant way.

Having my TEMP folders in RamDisk is a problem when driver needs to be updated.

Also, now that I have set the RamDisk to be dynamic, it's also a problem to update it, even if the driver itself doesn't need updating.

 

Maybe during the update process we can point the TEMP folders to another temporary location (C:\temp), and then point them back to the RamDisk.

 

I fully agree with that. I am using the 7-SFX module 7zS2.sfx because of its size, but it is the first thing executed, and it uses only the system TEMP folder (it cannot be tuned). This is perfect for most installers, but not for something that needs to remove the physical location of the defined TEMP folder.
So, I think it is time to search for another module, even if it adds some KBs to the final package.
Or perhaps it would be cleaner to ask for a reboot of Windows, because in any case, the volume of the TEMP folder needs to be unmounted, therefore you need to stop all the softwares that require this folder. With a reboot, we can use the TEMP folder of the ramdisk, but we have to ensure that the files will be updated before the service that runs the ramdisk.
Anyway, it will take some time, but I hope to find something better.