31 replies to this topic

[Sophia]

So I guessed right... well almost

One can tell that you have been programming for quite some time!

Thanks for the info, and all you support, Olof.

[Olof Lagerkvist]

Thanks a lot for all your ideas and all useful feedback!

[v77]

As I am using shared memory feature for a new project, I made some tests about the required buffer size. Here are my results:
Windows 2000/XP: 64 KB
Windows Vista: 8 MB
Windows 7/8/8.1: 2 MB

By the way, it seems that the current default value of devio (2 MB) is not enough for Vista...

So, is there a way to know for sure the size of this buffer, in order to not lose requests or datas?
Olof, I assume that you have made some researches, but would you have something new about this?

[Olof Lagerkvist]

As I am using shared memory feature for a new project, I made some tests about the required buffer size. Here are my results:
Windows 2000/XP: 64 KB
Windows Vista: 8 MB
Windows 7/8/8.1: 2 MB

By the way, it seems that the current default value of devio (2 MB) is not enough for Vista...

So, is there a way to know for sure the size of this buffer, in order to not lose requests or datas?
Olof, I assume that you have made some researches, but would you have something new about this?

The driver will use the buffer size your proxy service provides. It does this by querying the virtual memory size of the section object. Then, if the filesystem requests more data at one time than could fit in this buffer, the request is denied with STATUS_INVALID_BUFFER_SIZE. Not very nice and not really how a disk driver is supposed to handle things like this, but anyway. You have seen scenarios where filesystem drivers on Vista request more than 2 MB in one chunk? 32 or 64 bit? Steps to reproduce? I would very much like to see it and how it is currently handled!

[v77]

You have seen scenarios where filesystem drivers on Vista request more than 2 MB in one chunk? 32 or 64 bit? Steps to reproduce? I would very much like to see it and how it is currently handled!

 

All my tests were made on virtual machines (VirtualBox). The Vista machine is 64-bit with 2 GB of RAM. It was a copy of a big file (about 600 MB) with the "copy" command in a command prompt, from an image file. And to be sure, I setted the buffer size to 100 MB.
With that, almost all requests have a size of 8 MB.

But the most strange is that I don't get the same results on my real machine, Windows 8 x64, in comparaison to to a virtual machine with the same system.
The requests don't go beyond 1 MB, but in the virtual machine, about 10-20 % of the requests have a size of 2 MB. Yes, bigger requests on a slower machine with less RAM...

For now, I can limit the errors by sending an error code in case of a too big request: with that, Windows halts the copy with an error message. Currently, devio does not send back an error code in case of insufficient buffer, and so, Windows tries to continue the copy without warning the user. Maybe you could add this in devio...

But anyway, I am rewriting almost all to get something more suited to my new tool. So, for now, I would just like to know the exact maximum size needed for this buffer. Of course, I can simply allocate a huge buffer like 32 or 64 MB, but this will not look very pretty.
Unfortunately, this information seems to be quite hard to find.

[Olof Lagerkvist]

All my tests were made on virtual machines (VirtualBox). The Vista machine is 64-bit with 2 GB of RAM. It was a copy of a big file (about 600 MB) with the "copy" command in a command prompt, from an image file. And to be sure, I setted the buffer size to 100 MB.
With that, almost all requests have a size of 8 MB.

But the most strange is that I don't get the same results on my real machine, Windows 8 x64, in comparaison to to a virtual machine with the same system.
The requests don't go beyond 1 MB, but in the virtual machine, about 10-20 % of the requests have a size of 2 MB. Yes, bigger requests on a slower machine with less RAM...

For now, I can limit the errors by sending an error code in case of a too big request: with that, Windows halts the copy with an error message. Currently, devio does not send back an error code in case of insufficient buffer, and so, Windows tries to continue the copy without warning the user. Maybe you could add this in devio...

I don't follow. An I/O request larger than the shared memory block should never reach devio. The driver protects the shared memory block by never request more data than could fit there, nor (of course) fills more data to be written either. Like I said, it simply fails such requests and returns STATUS_INVALID_BUFFER_SIZE to the calling filesystem.

A more "correct" way to handle this, would be to split the request into smaller pieces, each fitting within the limits of the shared memory block.

 

Edit: I looked at the source code now. It appears I had either forgotten how this was designed, or there is some fatal mistake down there somewhere. The size protection I talked about, it appears it is only there for write requests. Not for read requests. This is bad in many ways, but particularly because it hands over the responsibility for verifying buffer sizes to the service end. I will try to figure out the best way to fix this once and for all. I feel that I should really get some split request logic implemented. It would at least mean some kind of following design guidelines for this kind of driver... I will be back on this matter soon!

 

[v77]

I feel that I should really get some split request logic implemented. It would at least mean some kind of following design guidelines for this kind of driver...

 

Yes, this would be perfect and would settle the issue once and for all.