334 replies to this topic

[ericgl]

v77,

 

I think a system reboot makes sense.

I have no problem with this solution.

[Olof Lagerkvist]

I fully agree with that. I am using the 7-SFX module 7zS2.sfx because of its size, but it is the first thing executed, and it uses only the system TEMP folder (it cannot be tuned). This is perfect for most installers, but not for something that needs to remove the physical location of the defined TEMP folder.

Most installers use the TEMP folder in one way or another. In many cases there is not much you can do about that, because the installer may run several processes and have quite complex relationships between all components. In the case of 7zSFX things are reasonably simple, though. You could create an application that copies the install directory from the TEMP folder to another location and restart setup there. You need to wait until the 7zSFX terminates. You get into about the same kind of solution with most self-extracting based installers, with various complexity.

But anyway, it does solve only parts of the problem. There are most probably other applications using the TEMP directory as well, like you write later...

So, I think it is time to search for another module, even if it adds some KBs to the final package.
Or perhaps it would be cleaner to ask for a reboot of Windows, because in any case, the volume of the TEMP folder needs to be unmounted, therefore you need to stop all the softwares that require this folder. With a reboot, we can use the TEMP folder of the ramdisk, but we have to ensure that the files will be updated before the service that runs the ramdisk.
Anyway, it will take some time, but I hope to find something better.

I suppose all files involved can be replaced so that the new versions will run on next reboot. I don't think you even need to use the FileRenameOperations registry key or anything like that in this case. Driver .sys files can be updated immediately. The kernel does not keep .sys files locked in any way. User-mode running binaries can be updated in-place by renaming the running file (.exe, .dll or similar) and then write a new file in its place. You could of course use FileRenameOperations anyway to cleanup the old files on next reboot, or having your application automatically removing the renamed old file next time it runs.

[v77]

About the driver, there is still imdsksvc.exe that cannot be directly replaced, like any .exe.

 

User-mode running binaries can be updated in-place by renaming the running file (.exe, .dll or similar) and then write a new file in its place.

 

Yes, I remembered that yesterday. Finally, it will be very simple to implement an update with reboot.
In order to solve the issue of the applications that are using the TEMP folder, I think to keep this solution. Of course, the reboot will be suggested only if needed, and the user will have the possibility to refuse.

[rebootpro****you]

I hate to restart my PC, I do it about once a month: when winupdate requires that.
If the only reason to require a restart is the problem of updating %TEMP% and %TMP% EnvVars - there's a way out to force update them without a system restart: it is described in that bulletin.
I think you'd better add a prompt to run a restart, but also provide an option to decline it. In case the prompt gets declined by the user - user should receive an alert that tells it to restart all the necessary running processes (including the shell), in case they are not able to track EnvVars' changes on the fly.

[Olof Lagerkvist]

I hate to restart my PC, I do it about once a month: when winupdate requires that.
If the only reason to require a restart is the problem of updating %TEMP% and %TMP% EnvVars - there's a way out to force update them without a system restart: it is described in that bulletin.
I think you'd better add a prompt to run a restart, but also provide an option to decline it. In case the prompt gets declined by the user - user should receive an alert that tells it to restart all the necessary running processes (including the shell), in case they are not able to track EnvVars' changes on the fly.

Thanks, but the information on that link does not solve the problem. Even if running processes could sometimes track changes to global environment variables, I would say that they generally do not, maybe except for the shell. The shell actually do track such changes through the mentioned window message in order to use the new settings for new processes created by the shell. But anyway, applications that use the temp directory frequently keep files open there for quite some time. In many cases for the lifetime of the process. It could be for example memory mapped files they are currently working with or similar and it is not always limited to current user processes, so it does not always help to log off and on again either.

In general, changes to global environment variables only affect new processes and not running ones. So, if you need them to change for all processes in order to move a directory or similar, a reboot is required because there is no method available to force running processes to honor new settings in any way.

[v77]

I hate to restart my PC, I do it about once a month: when winupdate requires that.
If the only reason to require a restart is the problem of updating %TEMP% and %TMP% EnvVars - there's a way out to force update them without a system restart: it is described in that bulletin.

 

I am aware that some people dislike this, but please understand that I cannot let the installer in the current state.
The information you are providing is interesting, but as Olof explains, the problem is more the current open files, that cannot be moved, than the new files that the softwares could create in the TEMP folder (and with the unlikely assumption that they are checking the current TEMP folder each time they need to create a new file).

This also does not fix the problem of the decompression module that uses only the TEMP folder. I can use another module, which lets to specify a directory, but what directory? There will always be issues for some users. So, should I let the user to specify a directory? But this makes the installation less simple.

 

I think you'd better add a prompt to run a restart, but also provide an option to decline it. In case the prompt gets declined by the user - user should receive an alert that tells it to restart all the necessary running processes (including the shell), in case they are not able to track EnvVars' changes on the fly.

 

Like I said in my previous message, the user will have the possibility to refuse the reboot. In this case, all the current open ImDisk volumes will continue to work without any issue. Of course, it will not be recommended to create a new one until the reboot.

If the driver does not need to be updated, and if you are not using a dynamic ramdisk, then the reboot will probably not be required, therefore you will not be prompted to reboot. This means you can still unmount your ramdisk, update the software and then restart the ramdisk, like before.

I cannot even add an option to restart the ramdisk automatically, because you can use ramdisk that are not registered to be launched at Windows startup. There is also the issue of the mounted image files, the used ImDisk proxies, etc.
And I do not want to take the responsability to create bugs in the used softwares (and some Windows services that are also using the TEMP folder) because some users are not aware of all the consequences of unmounting a used volume.

For this kind of tool, a reboot is the more clean and simple solution. This is also often the fastest one, because changing environment variables, closing and restarting a ramdisk, closing all the softwares that could cause issues, can take more time than a simple clic on a "reboot now" button and the time required for a reboot that should not nowadays exceed 30 seconds, or one minute if you are using an old system (like Windows 7 ) on a slow machine.

[rebootpro****you]

Uhm, can't the installer before updating ramdisk freeze all the other processes, make a per-byte copy of RAM disk, save it locally, unmount RAM disk, update the driver and utility,then re-create the RAM disk byte per byte is it was before the unmount, and then un-freeze the previously frozen processes?
Won't the programs run just fine after that?

I can use another module, which lets to specify a directory, but what directory? There will always be issues for some users.

Uhm, why would it cause any issues if you'd create a temporary "temp" folder somewhere in %APPDATA% for the very short period of time of your util's self-update?

And I do not want to take the responsability to create bugs in the used softwares (and some Windows services that are also using the TEMP folder) because some users are not aware of all the consequences of unmounting a used volume.

For this kind of tool, a reboot is the more clean and simple solution. This is also often the fastest one, because changing environment variables, closing and restarting a ramdisk, closing all the softwares that could cause issues, can take more time than a simple clic on a "reboot now" button and the time required for a reboot that should not nowadays exceed 30 seconds, or one minute if you are using an old system (like Windows 7 ) on a slow machine.

Wait a minute, but you don't need to restart all the programs! You don't even need to let them know you've updated an EnvVar at all!
Why?
Because you're going to change it back after the update finishes!

You have an old version of RAM-Disk and would like to update, but your %TEMP%'s and %TMP%'s paths are already pointing to the 'TEMP' folder on your existing RAM-Disk and all the programs are using it with that path. Then you freeze all the processes, make a per-byte copy of the RAM-Disk, then unmount-update-remount-restore_backup-unfreeze... done: %TEMP% and %TMP% are still pointing to your "r:\temp" that all the running programs already used before.

Won't that work? The only weak side of what I suggest is freezing processes: although it's something that a user can carry-out manually from the process manager, I suppose some shitty programs may crash after getting unfrozen. But it's their bug, not yours

Edited by rebootprofuckyou, 28 February 2014 - 01:43 PM.

[v77]

 The only weak side of what I suggest is freezing processes: although it's something that a user can carry-out manually from the process manager, I suppose some shitty programs may crash after getting unfrozen. But it's their bug, not yours

 

Thanks for your enthusiasm but unfortunately, there is no way to do such a thing... Theoretically, we can set the priority of our own process to "real-time", which means that it will be the only one executed by the system, but this creates a lot of other problems.

 

And this is not the only issue: when we dismount a volume, all open "handles" of the files used by the processes are broken. Recreating a drive with the same drive letter and the same content will change nothing to that, because the system is not supposed to know that we create a new volume with the same content. So, it will always consider the new volume as something different than the previous one, and therefore does no longer consider the current open files as valid.

[rebootpro****you]

I was not talking about setting "real-time" to your own process. I was talking about suspending other processes: there are a few ways of how to do that, but usually only one of them is the most popular.
While processes are suspended - they do not interact with all their "handles", thus, you may easily copy and then dismount a disk, change the %tmp% and %temp% paths to some temporary folder in %appdata%, uninstall your driver, do everything you need to update your utility, re-create the ramdisk, backup from the copy, switch %tmp% and %temp% paths back to ramdisk, then resume all the processes and since they won't notice all the actions you did while they were suspended - they won't notice anything at all.

[Olof Lagerkvist]

I was not talking about setting "real-time" to your own process. I was talking about suspending other processes: there are a few ways of how to do that, but usually only one of them is the most popular.
While processes are suspended - they do not interact with all their "handles", thus, you may easily copy and then dismount a disk, change the %tmp% and %temp% paths to some temporary folder in %appdata%, uninstall your driver, do everything you need to update your utility, re-create the ramdisk, backup from the copy, switch %tmp% and %temp% paths back to ramdisk, then resume all the processes and since they won't notice all the actions you did while they were suspended - they won't notice anything at all.

I see your idea, but there is actually a problem that you do not notice. When you suspend processes and remove the ramdisk they have open handles to, the handles to the files on the old ramdisk will be invalid. Then, you re-create the ramdisk with exactly the same contents and same drive letter and unfreeze the suspended processes. The processes will then wake up and use their existing handles to the old ramdisk, which are now invalid. In practice, they will get a "handle is invalid" error whatever they try to do with the handles they have. If they have memory-mapped files, the process will crash the next time they touch the memory that was mapped to the file on the old ramdisk.

 

The only way around this that I can see, would be to make heavy use of undocumented functions such as NtQuerySystemInformation to enumerate handles in the frozen processes and replace the handles in the handle table for those processes with new ones pointing to the new files. But this requires lots of work and is extremely risky without knowing exactly how the handles are used in the processes.

[rebootpro****you]

Wait a minute, RAM disk is not used by the programs as RAM: it is used as disk, so as far as I understand the programs don't store handles (memory addresses), instead, they use just the paths to the file. In other words - the handles are used only when programs put something to store in RAM, not on a disk. I'm not 100% sure, but that's how I thought programs actually work.

[Olof Lagerkvist]

Wait a minute, RAM disk is not used by the programs as RAM: it is used as disk, so as far as I understand the programs don't store handles (memory addresses), instead, they use just the paths to the file.
In other words - the handles are used only when programs put something to store in RAM, not on a disk.
I'm not 100% sure, but that's how I thought programs actually work.

No, handles are just numbers that represent open files. An application that wants to read from or write to a file first use a path to get a handle. Then, that handle is used for read/write operations. When the application is done with file, it closes the handle.

[rebootpro****you]

Oh, so it works this way... Well, thanks for at least this piece of knowledge, Olof.

[rebootpro****you]

One question though: is it possible to teach the installer somehow "lock" the space occupied by the ramdrive before unmounting that drive (in order to update the utility and/or the driver) and then re-mount the drive at exactly same space? In that case all the firstly suspended and then resumed processes will have their handles pointing to the very same addresses. Won't that trick actually work? The question is just hypothetic, of course I realize that even if that was possible - it would be very hard to code such a "memory space locker" and that probably won't ever be done, I'm just curious whether my idea would work at least in theory.

[rebootpro****you]

Also, another question to you, v77: could you please add some contacts (like, at least your email address) anywhere right into the utility? Because currently this shitty forum is the only place where users can contact you. But the problem is that this forum is actually quite shitty: first of all, it sucks as a forum, because it lacks all the necessary tags (like spoiler, code, quote, etc) full support, but what bothered me most is the registration: the forum closed it's own registration and now new users can only register if they are registered in twitter (and I had to create an account there first, which I didn't actually want to do). Also, since you've chosen this forum for official support - that's also the only place for all the issue reports. It would be way better if you'd host your utility on some site like github or at least sourceforge: sites like those create an issue tracker for each project you host there, also, there are quite a lot of other software projects already, so some active users like me, who like to try different software and report issues to the authors - they already probably have accounts there. So, please, consider moving your project to, say, github.

[Olof Lagerkvist]

One question though: is it possible to teach the installer somehow "lock" the space occupied by the ramdrive before unmounting that drive (in order to update the utility and/or the driver) and then re-mount the drive at exactly same space? In that case all the firstly suspended and then resumed processes will have their handles pointing to the very same addresses.
Won't that trick actually work?
The question is just hypothetic, of course I realize that even if that was possible - it would be very hard to code such a "memory space locker" and that probably won't ever be done, I'm just curious whether my idea would work at least in theory.

It does not work that way. Again, a handle is not a pointer. It represents an open file. Technically, it points to a device object that in turn points to a driver object. That driver is in this case imdisk.sys which in turn redirects requests to this dynamic ram disk application.

When you replace imdisk.sys and want to use the updated one, you need to restart this driver. This means that all handles connected to the driver object of the old driver in one way or another needs to close. It does not matter if you create a new driver object with exactly the same name and such, it will not have correct initialization for servicing requests made for the old driver.

[v77]

One question though: is it possible to teach the installer somehow "lock" the space occupied by the ramdrive before unmounting that drive (in order to update the utility and/or the driver) and then re-mount the drive at exactly same space? In that case all the firstly suspended and then resumed processes will have their handles pointing to the very same addresses. Won't that trick actually work? The question is just hypothetic, of course I realize that even if that was possible - it would be very hard to code such a "memory space locker" and that probably won't ever be done, I'm just curious whether my idea would work at least in theory.

 

A drive can be locked only if there is no active handle on it. And as Olof has explained, a handle is not an address but something like an identifier. It is not managed by the process but by the system: this is why an unmounted volume automatically invalidates all related handles. And if we recreate a new volume, it is just another volume, not the previous one, even if the content is the same, therefore the handles of the open files remain invalid.

 

Also, another question to you, v77: could you please add some contacts (like, at least your email address) anywhere right into the utility? Because currently this shitty forum is the only place where users can contact you. But the problem is that this forum is actually quite shitty: first of all, it sucks as a forum, because it lacks all the necessary tags (like spoiler, code, quote, etc) full support, but what bothered me most is the registration: the forum closed it's own registration and now new users can only register if they are registered in twitter (and I had to create an account there first, which I didn't actually want to do). Also, since you've chosen this forum for official support - that's also the only place for all the issue reports. It would be way better if you'd host your utility on some site like github or at least sourceforge: sites like those create an issue tracker for each project you host there, also, there are quite a lot of other software projects already, so some active users like me, who like to try different software and report issues to the authors - they already probably have accounts there. So, please, consider moving your project to, say, github.

 

All the tags you are talking are available here, even "spoiler" (e.g. here). And the Twitter account does not seem to be required but only a way to register faster...
I have another tool hosted on SourceForge (Whirlpool File Checker). It was at a moment where the server of reboot.pro was unreachable for me without proxies. Fortunately, these problems are gone and I prefer reboot.pro for several reasons.
GitHub? Probably great for the collaborative development, but not really designed to be used as a download server... Not suited at all for something like the ImDisk Toolkit.
About an email, well... this is something to consider, but it is a bit difficult for me to speak in english, and I don't like the idea to ignore much messages because I would not have the time to say exactly what I want.

[rebootpro****you]

Is that possible to create a RAM disk in GPU's memory?

[v77]

Is that possible to create a RAM disk in GPU's memory?

 

Theoretically, yes. For instance, we can imagine to use the content of loaded textures as memory blocks...
But even with the speed of the graphic memory, I am not very optimistic about the final speed, especially because of all the hardware latencies to add, and because of the communication between the proxy and the imdisk driver. So, if you are looking for speed, a classic static ramdisk will be always the best.

This could be funny to implement, but even with some experiments, I am not very familiar with the Direct3D API, and to speak frankly, my personal interest in such a feature is rather limited.

 

 

Edit: by the way, a new version is available, with the discussed changes of the installer and the ability to save and restore the configuration.

Edited by v77, 11 March 2014 - 02:32 AM.

[tjwasiak]

I would like to report strange issue I faced with ImDisk Toolkit on freshly installed (but fully updated) Polish XP Pro SP3:

 - both imdisk and imdsksvc are started and running

 - there is one DVD image mounted as drive H:

 - I am trying to create new hard disk image (I put filename including path as image file, drive letter is set to unused one, size = 25600 MB is smaller then free space on target drive, image file offset set to 0 bytes, device type set to hard disk, image file access set to "Virtual disk drive accesses image file directly") and constantly getting mixed English and Polish error stating  Error creating virtual disk: (this is in English) Access denied (this is in Polish).

 

I had been using ImDisk Toolkit many times in the past without such errors. I first faced similar error yesterday on different XP installation (same hardware, and also freshly installed fully updated XP Pro) when I was trying to mount existing DVD image. Restart cured that error but is not helping today.

 

I have to admit I first tried with previous (20140221) version, then I found there is newer version available and after update still have same issue.

 

BTW. I would suggest using a switch for silent install mode as more standard solution then environment variable...

[v77]

I would like to report strange issue I faced with ImDisk Toolkit on freshly installed (but fully updated) Polish XP Pro SP3:

 - both imdisk and imdsksvc are started and running

 - there is one DVD image mounted as drive H:

 - I am trying to create new hard disk image (I put filename including path as image file, drive letter is set to unused one, size = 25600 MB is smaller then free space on target drive, image file offset set to 0 bytes, device type set to hard disk, image file access set to "Virtual disk drive accesses image file directly") and constantly getting mixed English and Polish error stating  Error creating virtual disk: (this is in English) Access denied (this is in Polish).

 

I had been using ImDisk Toolkit many times in the past without such errors. I first faced similar error yesterday on different XP installation (same hardware, and also freshly installed fully updated XP Pro) when I was trying to mount existing DVD image. Restart cured that error but is not helping today.

 

I have to admit I first tried with previous (20140221) version, then I found there is newer version available and after update still have same issue.

 

According to your description, I assume that you are using the "ImDisk Virtual Disk driver" dialog box (the one of the control panel). On Vista and later, you can get an access denied error with that because this dialog box does not require admin rights, so if you put your file at a location where the file requires admin rights to be created, you get this error. Consequently, on XP, you are not supposed to get this error, unless the folder where the file is created does not have the permissions for file creation or the drive is locked.
Anyway, given the randomness of this error, I would think that another software is interfering.

 

 

BTW. I would suggest using a switch for silent install mode as more standard solution then environment variable...

 

I don't know why, but I was thinking that the decompression module (7zS2.sfx) does not transmit the command line parameters. It seems that is wrong, so I think to remove the use of the environment variable and replace it by command line parameters.

[v77]

Version 20140323
- installer: changed silent installation method: a command line switch is now used instead of environment variable (use any argument to get informations)
 

 

There is now two command line switches that can be used directly with ImDiskTk.exe:

/silent: Silent installation. Error messages and reboot prompt are still displayed.
/fullsilent: Silent installation, without error message or prompt.

Any other parameter displays a description of the available switches.
Since the processing of the environment variable IMDISK_SILENT_SETUP should no longer be useful for this installer, I removed it.

[rebootpro****you]

v77, could you, please, add "removable drive" disk type? I need this for mounting ram disk to VMs (in VMWare vSphere Client) as flash sticks.

[v77]

v77, could you, please, add "removable drive" disk type? I need this for mounting ram disk to VMs (in VMWare vSphere Client) as flash sticks.

 

When you put the mouse pointer on "Additional Parameters" in the Advanced tab of the RamDisk Configuration Tool, the tooltip tell you exactly what you need, that is, to add -o rem in this field.

Of course, I could add a check box for that, but I am not sure that much users could be interested by such a feature.

[rebootpro****you]

Oh, thx and sorry, didn't notice that. But unfortunately, VMware vSphere Client 5.1 doesn't see a disk created by your tool