Jump to content











Photo
- - - - -

Issue with FSCTL_MOVE_FILE


  • Please log in to reply
7 replies to this topic

#1 v77

v77

    Silver Member

  • Team Reboot
  • 602 posts
  •  
    France

Posted 05 February 2019 - 06:20 PM

I am experiencing a very strange bug with FSCTL_MOVE_FILE.
In ImDisk Toolkit, in the cleanup function of RamDyn, I use it to move a small file filled with 0 several times over the volume.
The file is open one time, with FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH (CreateFile), and then FSCTL_MOVE_FILE is called several times to move the file.

All that works pretty well when the volume is a drive letter. But when the volume is mounted on a folder, things become really bad.
The first call to FSCTL_MOVE_FILE takes at least 30 seconds to be processed. The following calls are processed without big wait, and they work as expected, but at least 100 times slower than with a drive letter. And it becomes gradually slower.
Eventually, the code get stuck at some point.
With this bug, Explorer freezes and cannot be killed. And the system cannot even shutdown.

The only difference is that I open a volume mounted on a folder instead of a drive letter.
The folder is open with FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_NO_BUFFERING (and the drive letter with only FILE_FLAG_NO_BUFFERING).
I tried to add FILE_FLAG_WRITE_THROUGH, without success.

The bug occurs on Windows 7 and 10, with both FAT32 and NTFS, and I already seen that these file systems process FSCTL_MOVE_FILE very differently. But here, the symptoms are exactly the same. So the culprit does not seem to be the file system driver.



#2 Olof Lagerkvist

Olof Lagerkvist

    Gold Member

  • Developer
  • 1448 posts
  • Location:Borås, Sweden
  •  
    Sweden

Posted 06 February 2019 - 03:48 PM

Strange. Maybe we have not seen that many use cases involving FSCTL_MOVE_FILE with ImDisk before. It is usually only used by defragmentation tools and probably not many people use them with ImDisk volumes.

 

It could take some time to find out what is going on here though. I need to hook up a kernel debugger and trigger this phenomenon and see if I can find out what really takes that much time. In general, file system drivers implement FSCTL_MOVE_FILE by sending a IOCTL_DISK_COPY_DATA request and if that fails (with an error code indicating that it is not supported), it will not try IOCTL_DISK_COPY_DATA anymore but instead issue read and write requests with a buffer. But this should be fine, there is nothing that indicates that IOCTL_DISK_COPY_DATA is required by a disk driver, it should be fine to just return failure to such calls.



#3 Olof Lagerkvist

Olof Lagerkvist

    Gold Member

  • Developer
  • 1448 posts
  • Location:Borås, Sweden
  •  
    Sweden

Posted 17 February 2019 - 05:20 AM

Tried the following:

Created 100 MB virtual disks, tried both with an image file and with just virtual memory.

Tried mounting both as drive letter D: and as a subdirectory on C:

Quick formatted, tried both FAT32 and NTFS.

Created a file called test.bin, 2 MB size in root directory. Wrote zeroes explicitly, not just setting file size.

Checked with fsutil file queryextents that clusters for the file were actually allocated.

Opened volume handle for \\?\D: in drive letter case and \\?\ImDisk0 in subdirectory case.

Opened file to be moved and filled out a MOVE_FILE_DATA structure and called FSCTL_MOVE_FILE on the volume handle.

Everything looks fine in both cases, no notable differences between drive letter and subdirectory mount points.

 

Volume handle opened in this way:

CreateFile(vol_name, FILE_READ_DATA | FILE_WRITE_DATA |
            FILE_APPEND_DATA | FILE_EXECUTE | FILE_READ_ATTRIBUTES,
            FILE_SHARE_READ | FILE_SHARE_WRITE, NULL,
            OPEN_EXISTING, 0, NULL)

File to be moved opened in this way:

        move.FileHandle = CreateFile(file_name,
            FILE_READ_DATA | FILE_WRITE_DATA | FILE_APPEND_DATA |
            FILE_EXECUTE | FILE_READ_ATTRIBUTES,
            FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
            NULL, OPEN_EXISTING, 0, NULL);

 

I also tried now to use a directory handle (opened with FILE_FLAG_BACKUP_SEMANTICS) instead of volume handle in the call to DeviceIoControl. Just like the documentation says that works correctly for NTFS but not for FAT32. But in the case of FAT32 it just fails immediately, there are no delays there either.

 

Tested on Windows 10 Enterprise x64 Build 17763.rs5_release.180914-1434. I could try on some other versions as well of course. What architectures, versions etc have you tested this on?



#4 v77

v77

    Silver Member

  • Team Reboot
  • 602 posts
  •  
    France

Posted 17 February 2019 - 10:54 AM

Thanks for testing. The way I open both the file and the volume is quite different.
I don't use the imdisk device but rather open the directory itself (that is, the mount point), with:
CreateFile(mount_point, GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_NO_BUFFERING, NULL);

For the file:
mfd.FileHandle = CreateFile(file_name, GENERIC_READ | GENERIC_WRITE, 0, NULL, CREATE_NEW, FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM | FILE_FLAG_DELETE_ON_CLOSE | FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH, NULL);

I will try with your flags.

#5 v77

v77

    Silver Member

  • Team Reboot
  • 602 posts
  •  
    France

Posted 17 February 2019 - 05:57 PM

I used your exact command for opening the volume: changed nothing.
For the file, I absolutely need FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH. This is for a dirty hack I had to write because of FSCTL_GET_RETRIEVAL_POINTERS that does not work as it should.

There is another difference: my imdisk volume is a proxy (RamDyn). But it would be surprising that it changes something.

The system is a virtual machine (VirtualBox) with Windows 7 SP1 x64, with 2GB of RAM and 2 CPU cores.



#6 Olof Lagerkvist

Olof Lagerkvist

    Gold Member

  • Developer
  • 1448 posts
  • Location:Borås, Sweden
  •  
    Sweden

Posted 18 February 2019 - 01:08 AM

I tried with FILE_FLAG_NO_BUFFERING now but it does not seem to make any difference here either. Still works perfectly. And looking at the source code for FAT file system driver it does not look like FILE_FLAG_NO_BUFFERING should make any difference for this FSCTL call anyway.

 

I'll retry my tests on Windows 7!



#7 Olof Lagerkvist

Olof Lagerkvist

    Gold Member

  • Developer
  • 1448 posts
  • Location:Borås, Sweden
  •  
    Sweden

Posted 18 February 2019 - 02:49 AM

Tested on Windows 7 x64 now. I still cannot reproduce the problem. Also tried with ImDisk in proxy mode just to rule out any differences this could have.

 

Just a random thought, since this is a difference between different kinds of mount points and since things nowadays frequently expect all volumes to be reachable through Volume Mount Manager, could there be some other components that interfere? Like some filter driver that waits for Volume Mount Manager registration before letting certain requests through? Like antimalware scanners etc? (I don't know if I am on the right track here, but if you have anything else installed that could possibly, even remotely possibly, have anything to do with this it could be useful for me to try to run that too and see what happens.)



#8 v77

v77

    Silver Member

  • Team Reboot
  • 602 posts
  •  
    France

Posted 18 February 2019 - 03:57 PM

The only thing installed is the guest additions of VirtualBox.

I think I will leave that for later. I come back if I find something.


  • Olof Lagerkvist likes this




1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users