Shared Memory feature

Olof, I would like to sincerely thank you for listening to the request for the "Shared Memory" feature and implementing it. I know it is hardly the most useful feature for something like a boot disk (looks like a lot of ImDisk users do this), but I hope I am not alone using ImDisk as a reliable and high-performance virtual disk driver to build something on top of it, without having to focus on writing a driver myself. I should've said this 3 days ago, but I only noticed the update tonight Once again, huge thanks for making ImDisk even better (and it was already superb before) - I know it has been a lot of effort and I, and I think everyone else using ImDisk, really appreciate it.

I can't wait to try it out and re-write my proxy from using IP to use SHM instead.

You got me really curious with the "although performance gain compared to TCP/IP is most notable on old Windows versions, especially prior to Windows XP" part. Any chance you could share your findings in a bit more detail on how TCP/IP performance compares to SHM?

Olof, I would like to sincerely thank you for listening to the request for the "Shared Memory" feature and implementing it. I know it is hardly the most useful feature for something like a boot disk (looks like a lot of ImDisk users do this), but I hope I am not alone using ImDisk as a reliable and high-performance virtual disk driver to build something on top of it, without having to focus on writing a driver myself. I should've said this 3 days ago, but I only noticed the update tonight Once again, huge thanks for making ImDisk even better (and it was already superb before) - I know it has been a lot of effort and I, and I think everyone else using ImDisk, really appreciate it.

Thanks a lot!

I can't wait to try it out and re-write my proxy from using IP to use SHM instead.

You got me really curious with the "although performance gain compared to TCP/IP is most notable on old Windows versions, especially prior to Windows XP" part. Any chance you could share your findings in a bit more detail on how TCP/IP performance compares to SHM?

Well, the TCP/IP performance in Windows XP and later is quite good and there is not much difference actually in what is done (number of memory copies, context switches etc) when you use TCP/IP vs. SHM in that case. On older versions of Windows though, there were a little less effective IP stack, especially for loopback communication, so the difference is more notable there. That's all.

I was getting close to having implemented the SHM communication and decided to test it out, but I couldn't persuade ImDisk to connect. I decided to see if devio can use SHM in the first place, since I am using it as an example, and I didn't manage to persuade devio to work either.

Here's what I did:

devio.exe shm:test1 C:\TEST\x.img

Successfully opened 'C:\TEST\x.img'.
Image size used: 366421872 bytes.
No master boot record detected. Using entire image.
Total size: 366421872 bytes. Using 366421872 bytes from offset 0.
Required alignment: 512 bytes.
Shared memory operation.
Waiting for connection on object test1. Press Ctrl+C to cancel.

imdisk.exe -a -t proxy -o shm -f test1
Creating device...
Error creating virtual disk: The system cannot find the file specified.

My guess was that the Shared Memory (and events) get created in the User's context, whilst imdisk service is executing in LOCAL SYSTEM context, so the driver cannot see it.

Consequently, I changed the name from test1 to Global\test1 and ... it did work just fine.

However, there is an example for VHD in FAQ topic which uses -f vhd1 and not Global\vhd1. Is this a typo, or am I missing something entirely?

However, there is an example for VHD in FAQ topic which uses -f vhd1 and not Globalvhd1. Is this a typo, or am I missing something entirely?

No, I have never actually seen that "Global" prefix was needed, it has always worked for me without that. But it will never harm to specify it either, so I will just change that in the FAQ!

At the moment when the SHM Server application "dies", meaning that it crashes or simply gets closed, it is impossible to remove the virtual drive using imdisk -D -u 0. This was not the case however with TCP/IP.

Can it be fixed, so that ImDisk can remove the virtual drive, even if the Server application does not respond?

At the moment when the SHM Server application "dies", meaning that it crashes or simply gets closed, it is impossible to remove the virtual drive using imdisk -D -u 0. This was not the case however with TCP/IP.

Can it be fixed, so that ImDisk can remove the virtual drive, even if the Server application does not respond?

I have tried to figure out a way to do that, but it is not entirely simple. If you (or anyone else) have an idea about how to do that it would be great.

Usually, with user mode applications you solve that problem with a mutex that gets "abandoned" when a process terminates. But this is a kernel mode driver and there is a very limited API set available for using this particular kind of mutex objects. There are other really simple objects for mutex synchronization in kernel though, but they cannot be used in a user mode application. Therefore I need to use event objects instead but then we get this problem where we cannot tell for sure if there is something "at the other end" or not.

I can think of at least two ways:

1. When removing the drive, put the IMDPROXY_REQ_CLOSE into the Shared Memory, set the "ShmName_Request" Event and then, without waiting for a Response, proceed with clean up. If the Server is alive, it will receive the event and can process it. If the Server is dead, the Events and Shared Memory will get cleaned up anyway.

2. Client creates an additional "ShmName_Client" Event. ImDisk before removing the drive attempts to open this event using ZwOpenEvent. If it fails, ImDisk knows that the client is gone.

I can think of at least two ways:

1. When removing the drive, put the IMDPROXY_REQ_CLOSE into the Shared Memory, set the "ShmName_Request" Event and then, without waiting for a Response, proceed with clean up. If the Server is alive, it will receive the event and can process it. If the Server is dead, the Events and Shared Memory will get cleaned up anyway.

Well, this is how it already works. The problem though is that if the device thread is stuck waiting for an I/O response from the server, it will not notice that server has died and therefore does not get out to the service loop again so that it will not get the notification that it should terminate.

2. Client creates an additional "ShmName_Client" Event. ImDisk before removing the drive attempts to open this event using ZwOpenEvent. If it fails, ImDisk knows that the client is gone.

Yes, that is a good way to know if the server has died, but what to do from there? How do you tell the device thread that in a reasonably safe way?

Oh ok, now I understand what the real issue is.

But what is wrong with your original abandoned mutex suggestion? According to the documentation I don't see what would prevent us from using KeWaitForMultipleObjects and check for the STATUS_ABANDONED_WAIT_X wait result, as a sign that the server is gone.

Oh ok, now I understand what the real issue is.

But what is wrong with your original abandoned mutex suggestion? According to the documentation I don't see what would prevent us from using KeWaitForMultipleObjects and check for the STATUS_ABANDONED_WAIT_X wait result, as a sign that the server is gone.

Like I said, it is a different kind of mutex object. If you find a (safe) way to open a mutex that the user mode server has created or how to get an object reference, this would not be a problem.

Sorry, I am new to kernel level programming, or it would be more accurate to say that I haven't ever done it yet

I didn't realize that Events are "the same" for user and kernel mode, but Mutexes are different.

Why don't we approach it from a different angle... could imdisk.exe itself, or the driver, when it receives a "remove drive" request, open the "ShmName_Response" event and then signal the event?

Sorry, I am new to kernel level programming, or it would be more accurate to say that I haven't ever done it yet

I didn't realize that Events are "the same" for user and kernel mode, but Mutexes are different.

Why don't we approach it from a different angle... could imdisk.exe itself, or the driver, when it receives a "remove drive" request, open the "ShmName_Response" event and then signal the event?

Something like that would be good. Although I would like to have a separate event object for that so that the driver easily can understand that this is a "remove request" and not an "I/O finished" request. Just to avoid all sorts of possible half-done I/O requests reported back to caller as complete.

But, we are already there. There is a cancel/remove event for each device thread that could be set to ask it to terminate. They also wait for this event with WaitForMultipleObjects or similar, so once that is fired all requests will be cancelled and the device will be removed. But the problem is that imdisk.exe will not reach that point because when it first tries to delete the device in normal safe way, it will just hang there. This is because when the filesystem is dismounted, the filesystem driver will request some information in the disk driver and this is where it will hang. Therefore imdisk.exe does not get any chance to move on to the point where it attempts to forcefully remove the device.

But when I think about it, we could probably add some kind of "emergency remove" switch to imdisk.exe where it does not even attempt to dismount the filesystem or in any other way attempt a "safe" removal of the virtual disk but instead directly request the driver to forcefully remove it. That would work. I just tested that now with a separe imdiskforceremove.exe that does exactly this. Link: http://www.ltr-data....forceremove.exe

Regards,

Yes, I like the idea of an "emergency remove" switch. Thanks Olof

I tested out the app and it works like a charm. Can't wait for that new switch

One more question:

In devio.c, am I correct in assuming that shm_server_mutex is not actually used at the moment and was put there for testing, so I don't need to even create it?

I tested out the app and it works like a charm. Can't wait for that new switch

One more question:

In devio.c, am I correct in assuming that shm_server_mutex is not actually used at the moment and was put there for testing, so I don't need to even create it?

It is there to make sure that you never start another server with same object name. It will normally work without it, but there is a risk that a simple mistake would seriously break things if a user starts two sessions with same object name.

That's what I thought initially, but in devio.c, CreateFileMapping comes before CreateMutex, so if it used as a uniqueness check, I think the order should be changed.

However, if the name is already taken,CreateFileMapping will result in GetLastError = ERROR_ALREADY_EXISTS. The same goes for the two events. Isn't this a good enough check? It might be even better, because it checks the real cause of a potential problem, rather than a flag. For example, if one day ImDisk creates for some technical purpose an "ImDisk" Shared Memory location, and then I attempt to create a virtual drive with a name "ImDisk", the mutex check will be fine, because "ImDisk_Server" mutex is not taken. Even worse, CreateFileMapping will not report any error if it already exists, so as a result it will corrupt the data in the "ImDisk" Shared Memory location, because now it is being used for two purposes.

That's what I thought initially, but in devio.c, CreateFileMapping comes before CreateMutex, so if it used as a uniqueness check, I think the order should be changed.

However, if the name is already taken,CreateFileMapping will result in GetLastError = ERROR_ALREADY_EXISTS. The same goes for the two events. Isn't this a good enough check? It might be even better, because it checks the real cause of a potential problem, rather than a flag. For example, if one day ImDisk creates for some technical purpose an "ImDisk" Shared Memory location, and then I attempt to create a virtual drive with a name "ImDisk", the mutex check will be fine, because "ImDisk_Server" mutex is not taken. Even worse, CreateFileMapping will not report any error if it already exists, so as a result it will corrupt the data in the "ImDisk" Shared Memory location, because now it is being used for two purposes.

Yes, of course you have a point here. But I think we should keep this mutex object anyway, because if we find a way for the driver to open it and check if it gets abandoned, we have a robust solution to the initial problem discussed here today. But it is probably a good thing to change devio etc to check for ERROR_ALREADY_EXISTS too, particularly for the reasons you mention here. I think I will change that right away!

One more thing... do we actually need the two Request and Reponse events? Can we instead maybe have just one, say DataReady event, which is set as soon as the data in the Shared Memory is ready for consumption by the other party? I realize it makes virtually no difference in terms of performance... just wondering if we can make things simpler

One more thing... do we actually need the two Request and Reponse events? Can we instead maybe have just one, say DataReady event, which is set as soon as the data in the Shared Memory is ready for consumption by the other party? I realize it makes virtually no difference in terms of performance... just wondering if we can make things simpler

Over the years I have heard many developers suggest things like that. Every time it has in one way or another required at least one extra synchronization object to synchronize the first synchronization object, which then have ended up with a simple two-object design. So, the reason behind this design is pure experience from real life (sort of).

More specifically, the problem is that if you just have one event and set that event as signaled, there is a risk that this signaled event might satisfy the next wait call for the same thread. This risk can of course be avoided in various ways and if everyone is extremely careful, but it makes life a lot easier to not even introduce this risk in the first place.

I understand your reasons and "there is a risk that this signaled event might satisfy the next wait call for the same thread" is something I didn't think about.

I finally got my Server to work properly with SHM - hooray! Actually, it took me about 3 hours of debugging to realize that the data is located at a 4KB offset. It is obvious from the imdisk.c code, but somehow in the devio.c I missed it entirely - probably due to my rather limited C++ experience.

One more thing I would like to check...

Is it correct that the OS read/write request size should not exceed 16MB, and therefore the SHM buffer can have a static 16MB + 4KB size? I got this idea from the #define DEF_BUFFER_SIZE (16 << 20) line.

I just have never seen a Disk IO size limit mentioned, but then again I probably never looked in the right section of MSDN

I understand your reasons and "there is a risk that this signaled event might satisfy the next wait call for the same thread" is something I didn't think about.

I finally got my Server to work properly with SHM - hooray! Actually, it took me about 3 hours of debugging to realize that the data is located at a 4KB offset. It is obvious from the imdisk.c code, but somehow in the devio.c I missed it entirely - probably due to my rather limited C++ experience.

Sorry, I missed to mention this difference. Reason is mostly page efficiency since this means that the I/O data will always start at a page base which means that it can be used directly in unbuffered I/O calls at the server end and also that the number of in-page operations will be kept to an absolute minimum at both ends.

One more thing I would like to check...

Is it correct that the OS read/write request size should not exceed 16MB, and therefore the SHM buffer can have a static 16MB + 4KB size? I got this idea from the #define DEF_BUFFER_SIZE (16 << 20) line.

I just have never seen a Disk IO size limit mentioned, but then again I probably never looked in the right section of MSDN

Yes, you can assume that this is safely beyond any reasonable I/O buffer size. If not, the driver checks of how large the shared memory block is and never write outside it nor request the I/O server to do so. If the calling filesystem driver (or something else) has requested more data, it will practically just be sent an error return. This should quite emulate the behaviour in physical disk drivers where for example SCSI block sizes may be limited.

This is nice. I was always concerned that when working with an SSD, the OS might go for requests larger than 16MB, in order to reach that peak throughput. I actually thought that returning an error would result in an IO error, but I suppose this is not the case - the OS will retry with a smaller sized request?

Anyway, I have been testing SHM and at least for me everything works absolutely wonderfully. There are no problems that I noticed on Windows 7 x64, with both UAC on and off.

If you say that adding "Global" prefix should never do any harm, I would suggest adding it by default in both ImDisk and DevIO. I am in fact curious why it works for you without it - I thought the driver cannot (meaning it doesn't look there) access events in the user's object namespace.

This is nice. I was always concerned that when working with an SSD, the OS might go for requests larger than 16MB, in order to reach that peak throughput. I actually thought that returning an error would result in an IO error, but I suppose this is not the case - the OS will retry with a smaller sized request?

Yes, this is how it is supposed to work. Filesystem drivers must count on the possibility that buffer sizes may be limited on underlying hardware.

Anyway, I have been testing SHM and at least for me everything works absolutely wonderfully. There are no problems that I noticed on Windows 7 x64, with both UAC on and off.

Good to hear that! What is your impression on performance?

If you say that adding "Global" prefix should never do any harm, I would suggest adding it by default in both ImDisk and DevIO. I am in fact curious why it works for you without it - I thought the driver cannot (meaning it doesn't look there) access events in the user's object namespace.

It can do that because the device thread in the driver impersonates the user that requests the drive to be created. This is needed for a couple of other reasons as well, for example creating drive letter links in correct namespace and using the thread client security token to access files on network servers through UNC paths and similar. This also means that if devio runs in same security context as the thread calling ImDisk to create a virtual disk, they will use the same object namespace. This breaks on Vista/7/2008 etc with UAC when one of them run elevated and one with reduced privileges.

But we could of course make Global a default prefix. It just needs to retry without it in case it will get an error when Global directory does not exist. But that would be pretty simple anyway.

What is your impression on performance?

As soon as I have some proper quantifiable results, I will share them. For now all I can say is that it seems to work quite a bit faster when testing with a 600MB/s SSD. This probably due to the fact that there is much less memory copying involved. One thing is pretty noticeable - much less CPU load at full speed. Also, I love the simplicity. The TCP/IP libraries that I was using were quite annoying, with thread pools and lots of performance degrading features and excessive memory copying. Now all I need is 1 thread, 1 Shared Memory location and two Events (and a Mutex ). All in all, I am very pleased and grateful for this new feature. Once again, thank you very much!

This breaks on Vista/7/2008 etc with UAC when one of them run elevated and one with reduced privileges.

That must be the reason why it didn't work for me. I was never a big fan of UAC so I always deactivated it after installing Windows. However, I got tired of finding out from someone else that my apps don't work because of UAC, so I decided to run it at all times, so that I can experience the problems first hand.

will get an error when Global\ directory does not exist

I didn't realize that was even possible... is this the case for pre-NT Windows versions or is there some special case that I am missing?

I didn't realize that was even possible... is this the case for pre-NT Windows versions or is there some special case that I am missing?

The Global\ and Local\ prefixes were introduced when Windows NT kernel started supporting multiple user sessions. There were a special version of Windows NT 4.0 called "Terminal Server Edition" that supported this, but the normal version of NT 4.0 did not. In Windows 2000 kernel object namespace behaved differently depending on whether Terminal Services was running or not. With Terminal Services it was pretty much like in later versions of Windows except that it did not automatically search global namespace in case an object was not found in local namespace. Without Terminal Services running it behaved close to how it worked on NT 4.0, except that it provided two "dummy" symbolic links called Global and Local that pointed to the same \BaseNamedObjects directory to just support the prefixes although not actually used for anything.

There is a note about compatibility issues with Global\ and Local\ prefixes in MSDN documents for example about CreateEvent() and similar functions.