10 replies to this topic

[v77]

I have a big issue, that I sort of fixed after 2 days of tests, with my ProxyCrypt tool: when the system shuts down, the proxy is killed and so, the driver is waiting indefinitely, blocking this way the system.

I could ask Olof to add a timeout in the driver, but this will also add several other issues.
So, I need to unmount the volume myself, by checking the system shutdown. Here, the difficulties start...

Starting a process at the shutdown is really not recommanded. So I could call the functions of imdisk.cpl, but with all the notifications, this is too slow. So, I finally made my own dismount function, with only 2 calls to DeviceIoControl, with a handle opened to my volume: FSCTL_DISMOUNT_VOLUME and IOCTL_STORAGE_EJECT_MEDIA (strangely, without the second one, the dismount take more time).

I first tested this by defining an handler with SetConsoleCtrlHandler. I don't have tested on previous versions of Windows, but with 8.1, I simply get nothing at the shutdown. Or at least, I can do nothing, not even creating and closing an empty file. But this handler works for all other events, for an amount of time that not only depends of the event, but also strangely of what I am doing.
In all cases, this no longer reliable.
So, I tried to create a dummy window to check for the WM_QUERYENDSESSION message. It works, but if I create a thread to start something long and answer FALSE to this message, my process is killed about one second later. However, if I take that time directly while responding to the message, I get about 5 seconds.
In all cases, processing this message does no longer work to prevent the shutdown. I cannot even get the fullscreen message that says that an application is not responding.
Finally, I think to directly use WM_ENDSESSION. I doubt (and I don't hope) that Microsoft change its behavior before a long time.

It also seems that, at least at the shutdown of Windows 8.1, the FlushFileBuffers function is not necessary: even if start the shutdown just after a copy of large files, Windows flushes everything itself while several seconds, without FlushFileBuffers, and I have no data lost. That said, to be sure, maybe using it can be useful.

 

Anyhow, I wonder what we can do to fix this issue reliably.

[Sha0]

Can your user-land process receive a shut-down notification, cancel the pending shutdown, do work, then re-initiate a shut-down? shutdown -a manages to abort a shut-down.

[v77]

Can your user-land process receive a shut-down notification, cancel the pending shutdown, do work, then re-initiate a shut-down? shutdown -a manages to abort a shut-down.

 

As far as I know, this command can prevent a scheduled shutdown, like shutdown.exe can do, not a running shutdown.

Finally, it seems still possible to block a running shutdown by processing WM_QUERYENDSESSION (and then getting a fullscreen message that asks for user approval), but like in Windows Vista/7/8, the window has to be visible. For a console application, this is not really appropriate... And if the window is not visible, I also get WM_QUERYENDSESSION, but with a limited time. For now, it seems to be enough, but I would like something more reliable.

I also tried to make the window visible when I receive WM_QUERYENDSESSION: I get the fullscreen message, and so, the shutdown is halted. But this does not work with Vista.

[Olof Lagerkvist]

Note that calling FSCTL_DISMOUNT_VOLUME without a successful FSCTL_LOCK_VOLUME call first, will cause any open files to be forcefully closed. That is, applications that are about to save data through these file handles will not be able to complete file I/O operations.

FSCTL_LOCK_VOLUME will not succeed as long as there are open handles to files on the volume, so it is not always possible. But I think it would be a good idea if you at least call FlushFileBuffers on the volume handle before FSCTL_DISMOUNT_VOLUME. Even if you happen to find now that seems unnecessary, different situations on other machines could require it.

For "emergency" cases, there is also IOCTL_IMDISK_REMOVE_DEVICE. This is useful for example when proxy backend communication hangs. This IOCTL will cancel any waiting for proxy service and immediately remove the device. It could be used to create some timeout mechanism in the way you suggest. Just spawn another thread that monitors what happens and calls IOCTL_IMDISK_REMOVE_DEVICE in cases when normal FSCTL_nnn/IOCTL_nnn operations hang.

[v77]

Note that calling FSCTL_DISMOUNT_VOLUME without a successful FSCTL_LOCK_VOLUME call first, will cause any open files to be forcefully closed. That is, applications that are about to save data through these file handles will not be able to complete file I/O operations.

FSCTL_LOCK_VOLUME will not succeed as long as there are open handles to files on the volume, so it is not always possible. But I think it would be a good idea if you at least call FlushFileBuffers on the volume handle before FSCTL_DISMOUNT_VOLUME. Even if you happen to find now that seems unnecessary, different situations on other machines could require it.

For "emergency" cases, there is also IOCTL_IMDISK_REMOVE_DEVICE. This is useful for example when proxy backend communication hangs. This IOCTL will cancel any waiting for proxy service and immediately remove the device. It could be used to create some timeout mechanism in the way you suggest. Just spawn another thread that monitors what happens and calls IOCTL_IMDISK_REMOVE_DEVICE in cases when normal FSCTL_nnn/IOCTL_nnn operations hang.

 

About FSCTL_LOCK_VOLUME, I wonder what is the use of it in a case like this one: if there is open handles, it does not work, and if there is not, there will be no file that will be forcefully closed...
If the volume is not locked by another process, it can be useful to get an exclusive access, but if it is the case, why not directly dismount the volume?

Thanks for the hint about IOCTL_IMDISK_REMOVE_DEVICE. A bit complicated to use without the device number...   I will probably use your ImDiskForceRemoveDevice function instead. I prefer to not add too much code for such an improbable case.

[Olof Lagerkvist]

About FSCTL_LOCK_VOLUME, I wonder what is the use of it in a case like this one: if there is open handles, it does not work, and if there is not, there will be no file that will be forcefully closed...
If the volume is not locked by another process, it can be useful to get an exclusive access, but if it is the case, why not directly dismount the volume?

I understand how you mean, there is not much you could do in the case FSCTL_LOCK_VOLUME does not succeed. But you could at least write a note about it to some log somewhere. The general idea is otherwise that you are supposed to wait until FSCTL_LOCK_VOLUME succeeds or some kind of timeout (in which case a warning should be logged to event log). A service that is notified about a pre-shutdown will be able to delay a system shutdown until it is reported as stopped, so it is generally possible to wait for at least some reasonable time. Though, if your application is not running as a service, I am not sure it is possible to delay a system shutdown in a similar way.

[Olof Lagerkvist]

More here:
http://msdn.microsof...p/ms683241.aspx
And a discussion about pre-shutdown notifications here:
http://stackoverflow...fication-from-c

[v77]

What you are saying about FSCTL_LOCK_VOLUME is true in the general case. But now, Microsoft seems to want that we consider the shutdown as an emergency situation, supposedly for the "user experience"...
And about writing a log somewhere, this is something I prefer to avoid for a tool like ProxyCrypt.

About writing a service, unfortunately, the default "WaitToKillServiceTimeout" registry value is now quite low:
- XP/Vista: 20000
- 7: 12000
- 8/8.1: 5000
So, I am afraid that writing a service to get some additionnal time is not a good idea...
This is also why I did not yet respond to the request about ImDisk Toolkit to save the content of a ramdisk at the system shutdown... I found no way to bypass this limit.

Anyway, I now have an "correct" solution for ImDisk. But for Arsenal Image Mounter, which have the same issue... I know nothing about its API. That said, the removal of the device with ArsenalImageMounterControl.exe is quite fast, and WM_QUERYENDSESSION is sent before the process are killed... A bit dirty in my toast, though. But if you have a minimalist solution in C...

[Olof Lagerkvist]

What you are saying about FSCTL_LOCK_VOLUME is true in the general case. But now, Microsoft seems to want that we consider the shutdown as an emergency situation, supposedly for the "user experience"...
And about writing a log somewhere, this is something I prefer to avoid for a tool like ProxyCrypt.

Of course, but it could be useful to at least log somewhere that the dismount was unsafe and that there could possibly be lost data. It is at least better to log this scenario than to simply ignore it.
 

About writing a service, unfortunately, the default "WaitToKillServiceTimeout" registry value is now quite low:
- XP/Vista: 20000
- 7: 12000
- 8/8.1: 5000
So, I am afraid that writing a service to get some additionnal time is not a good idea...
This is also why I did not yet respond to the request about ImDisk Toolkit to save the content of a ramdisk at the system shutdown... I found no way to bypass this limit.

You have misunderstood how this works. The timeout you mention is for the "shutdown" notification, the same notification that existed before Vista. This does not in any way affect the new "pre-shutdown" notification. This is how for example Hyper-V gets enough time to save state of virtual machines before system shutdown. Once again, read the StackOverflow discussion I linked to! http://stackoverflow...fication-from-c
 

Anyway, I now have an "correct" solution for ImDisk. But for Arsenal Image Mounter, which have the same issue... I know nothing about its API. That said, the removal of the device with ArsenalImageMounterControl.exe is quite fast, and WM_QUERYENDSESSION is sent before the process are killed... A bit dirty in my toast, though. But if you have a minimalist solution in C...

In the case of Arsenal Image Mounter, this will be exactly like removing a physical SCSI disk from a hotswap bay without first dismounting the filesystems mounted on it. It causes a "surprise removal" up the device stack and the filesystem driver may or may not respond by writing an event log entry about lost data. When the filesystem is remounted, the filesystem driver may or may not write another event log entry about possibly corrupt filesystem. It is generally a good idea to attempt to offline a disk before removing in this way. This will cause Windows to automatically attempt to dismount filesystems and then setting the disk in offline mode. Unfortunately, this also means that the disk will be offline in Disk Management and no filesystems automatically mounted the next time you attach this disk too, which might not be what you want. But I don't know of any better solutions, currently.

[v77]

Of course, but it could be useful to at least log somewhere that the dismount was unsafe and that there could possibly be lost data. It is at least better to log this scenario than to simply ignore it.

 

Sorry but I will not follow you on this point. If I write somewhere informations about the use of ProxyCrypt, an attacker will just have to "ask" the user for the password, and in this case, the cryptographic tool is useless. Like with TrueCrypt. I consider the security of the user as more important than the security of the data.
That said, I hope to avoid all risk of data lost with the new version, and like always, this tool is provided without any warranty etc...

 

 

You have misunderstood how this works. The timeout you mention is for the "shutdown" notification, the same notification that existed before Vista. This does not in any way affect the new "pre-shutdown" notification. This is how for example Hyper-V gets enough time to save state of virtual machines before system shutdown. Once again, read the StackOverflow discussion I linked to! http://stackoverflow...fication-from-c

 

Yes, it seems I missed something. But I wonder how many time Microsoft will keep this possibility without adding a timeout...
Furthermore, this means to make extra code for XP, or to break the compatibility with it... But this is interesting.

[Olof Lagerkvist]

Sorry but I will not follow you on this point. If I write somewhere informations about the use of ProxyCrypt, an attacker will just have to "ask" the user for the password, and in this case, the cryptographic tool is useless. Like with TrueCrypt. I consider the security of the user as more important than the security of the data.
That said, I hope to avoid all risk of data lost with the new version, and like always, this tool is provided without any warranty etc...

Of course I understand such concerns too. It just feel that for myself for example, I would like to know when I am at risk of losing data even if it is about encrypted data. Encrypted data I have could be just as important as unencrypted data in that way. But it is your choice of course. I just share my thoughts on the matter.  
 

Yes, it seems I missed something. But I wonder how many time Microsoft will keep this possibility without adding a timeout...

Well, this possibility has been added for a couple of reasons that include design-breaking demands. One example is the Hyper-V platform. There are other similar needs in modern database engines. It is of course possible that they would change the way you could use this feature in some ways, including for example requirements to report remaining time needed or similar things. But a simple timeout would just take data integrity issues back to the situation on XP and prior which I would guess is out of question. Especially considering the kind of server/cluster/virtualization infrastructure environments Windows is used for in these days.
 

Furthermore, this means to make extra code for XP, or to break the compatibility with it... But this is interesting.

Yes, that is a problem of course. But it is generally possible to at least keep most of the code the same even if you make some extra code for XP compatibility.