16 replies to this topic

[TheK]

-- Release 1.2.6
* When contents of a disk volume is saved as an image file from Control Panel applet or using Explorer context menu option it is now ensured that the resultning image file has the same size as the saved disk volume. This means that even if the underlying filesystem driver does not return all data at the end of the disk volume the image file will still be padded and filled up to the matching size so that everything works correctly when the image file is later mounted as a virtual disk.

You can use the FSCTL_ALLOW_EXTENDED_DASD_IO control code to get access to the whole disk volume.

The FSCTL_ALLOW_EXTENDED_DASD_IO control code is used to signal the file system driver not to perform any I/O boundary checks on read or write calls made with the specified handle. FSCTL_ALLOW_EXTENDED_DASD_IO allows access to hidden sectors, a part of the partition that might exist between the first sector of the partition (the boot parameter block) and the first useful sector of the partition. FSCTL_ALLOW_EXTENDED_DASD_IO also allows access to lost clusters, which might exist between the last useful cluster and the end of the partition.

[Olof Lagerkvist]

You can use the FSCTL_ALLOW_EXTENDED_DASD_IO control code to get access to the whole disk volume.

THAT seems to be what we have been looking for all the time. I have changed that now, and 1.2.7 is out!

Thank you very much for this!

[TheK]

I'm gald that I was able to help to improve this wonderful software.

[TheK]

hmmm, I tested v 1.2.7 and now there seems to be another problem

500 MB image -> saved to 500 MB, image is complete
501 MB image -> saved to 500 MB, 1 MB missing
502 MB image -> saved to 502 MB, image is complete
503 MB image -> saved to 502 MB, 1 MB missing
504 MB image -> saved to 504 MB, image is complete
505 MB image -> saved to 504 MB, 1 MB missing

Looks like we have a (not so) nice odd/even pattern

[Olof Lagerkvist]

hmmm, I tested v 1.2.7 and now there seems to be another problem

500 MB image -> saved to 500 MB, image is complete
501 MB image -> saved to 500 MB, 1 MB missing
502 MB image -> saved to 502 MB, image is complete
503 MB image -> saved to 502 MB, 1 MB missing
504 MB image -> saved to 504 MB, image is complete
505 MB image -> saved to 504 MB, 1 MB missing

Looks like we have a (not so) nice odd/even pattern

Verified. Strange, really. This is with FSCTL_ALLOW_EXTENDED_DASD_IO turned on. On the other hand, there should be no problem adjusting the image file size afterwards like I introduced in 1.2.6. But I am starting to believe that the problem could be something else, or that there are many problems involved and that we have solved just one or a few of them...

The problem reported by jaclaz some time ago with saving small floppy images seem to have been solved now. Good so far. On the other hand, there is still a problem saving large disk images and it seems that it depends on the buffer size used when reading disk -> writing image file. If disk size is a multiple of the buffer size everything works as expected, but otherwise various lengths are missed at the end of the disk. The buffer size is currently hard-coded to 2 MB which is the reason why odd number of MB sized disks fail in your tests. But this only happens for large disks. With smaller disks, it does not matter if the disk size is a multiple of the buffer size.

Now, an idea would be to auto-select buffer size so that the disk size will always be a multiple of it, up to, say max 2 MB buffer size.

But I start feeling that we are only guessing and that this could fail at some other unpredictable point in the future.

[MedEvil]

Seems like the buffer is only written to disk when it is full. Else it is skiped now. (After a time out i suspect.)




edit: Could it be that you've forgot to flush it at the end?

[TheK]

And once again MSDN has the answer.

When a synchronous read operation reaches the end of a file, ReadFile returns TRUE and sets *lpNumberOfBytesRead to zero.

Really nasty

So if you reach EOF you have the request the exact number of bytes left instead of a full buffer.

[Olof Lagerkvist]

And once again MSDN has the answer.

Really nasty

So if you reach EOF you have the request the exact number of bytes left instead of a full buffer.

No, that means that ReadFile will return TRUE and report zero bytes read when the file pointer is at EOF or beyond EOF. A simple test program verifies this. This also works for small disk sizes. But for some reason it does not work for large disk sizes and I do not see a clear way to solve it except choosing a buffer size for which the disk size is an even multiple.

[Olof Lagerkvist]

Seems like the buffer is only written to disk when it is full. Else it is skiped now. (After a time out i suspect.)




edit: Could it be that you've forgot to flush it at the end?

No, the problem is not regarding the output image file, it is reading the disk device that stops before the "physical end". But after a few tests now I start to believe that an automatic selection of buffer sizes based on disk size seems to fix it. I think I will go for that.

I will also soon publish a new version of the rawcopy tool with new switches for selecting read and write without (intermediate) buffering and "write-through" options. I added this while testing and it could be useful for others to have too.

[MedEvil]

Can't wait!

[TheK]

No, that means that ReadFile will return TRUE and report zero bytes read when the file pointer is at EOF or beyond EOF. A simple test program verifies this. This also works for small disk sizes. But for some reason it does not work for large disk sizes and I do not see a clear way to solve it except choosing a buffer size for which the disk size is an even multiple.

I get the following results with my own test program:

Using FSCTL_ALLOW_EXTENDED_DASD_IO on a volume handle seems to change the way ReadFile() works.
If nNumberOfBytesToRead is larger than bytes left to read from the volume, then ReadFile returns TRUE but lpNumberOfBytesRead is set to zero.

Without FSCTL_ALLOW_EXTENDED_DASD_IO everything works as usual.

Here is some pseudocode that always works for me:

...

	LONGLONG numLoop = disk_size / BufferSize;

	LONGLONG BytesLeft= disk_size % BufferSize;



  for &#40;int j=0;j<numLoop;j++&#41;

	{

		  ReadFile&#40;...&#41;;

		  WriteFile&#40;...&#41;;

	 }

	if&#40;BytesLeft != 0&#41;

	 {

		  ReadFile&#40;DeviceHandle, buffer,BytesLeft, &dwReadSize, NULL&#41;;

		  WriteFile&#40;...&#41;;

	   }

	}

....

Basically I only adjust the 'nNumberOfBytesToRead' for the last ReadFile() operation.

edit:

btw, I still can't save floppy images.

Attached Thumbnails

• 

[Olof Lagerkvist]

I get the following results with my own test program:

Using FSCTL_ALLOW_EXTENDED_DASD_IO on a volume handle seems to change the way ReadFile() works.
If nNumberOfBytesToRead is larger than bytes left to read from the volume, then ReadFile returns TRUE but lpNumberOfBytesRead is set to zero.

Without FSCTL_ALLOW_EXTENDED_DASD_IO everything works as usual.

Here is some pseudocode that always works for me:

...

Basically I only adjust the 'nNumberOfBytesToRead' for the last ReadFile() operation.

Yes, I saw this too. A deeper examination shows that it is actually the disk driver that fails the last read because it is (partially) outside the partition. When there is a filesystem driver in between that one has the option to adjust the length of the I/O operation. Would it be allowed by the disk driver it would possibly have read a piece of the next partition on disk (or whatever is there).

The solution I am implementing right now is to select a buffer size of which disk size is a multiple so that there is no "odd-sized" read operation left at the end. Either that or your solution would work if we have understood everything correctly.

[TheK]

The solution I am implementing right now is to select a buffer size of which disk size is a multiple so that there is no "odd-sized" read operation left at the end. Either that or your solution would work if we have understood everything correctly.

How do you calculate the buffer size if the total number of sectors of the volume is a prime number?

[MedEvil]

How do you calculate the buffer size if the total number of sectors of the volume is a prime number?

Easy! Buffer size = volume size!

[Olof Lagerkvist]

How do you calculate the buffer size if the total number of sectors of the volume is a prime number?

I bitshift by one (divide by two) as long as the result is even, up to max buffer size = 2 MB. So, in that particular case the buffersize would be equal to sector size. This could be inefficient so your method is probably better. I have experimented with some different buffer sizes lately and found that 2 MB seems somewhat too large to be optimal on most machines. I think I will go for something like 512 KB or 1 MB instead.

[TheK]

Once the buffer exceeds a certain size there are usually no more noticable performance gains. An exception is probably when you perform read and write operations on the same physical drive where a larger buffer results in less seek and more sequential I/O operations.

But I guess as long as it works correctly and you don't use a sector size buffer, most people will be happy

[TheK]

v 1.2.8 works!!!

Thank you very much Olof