Jump to content











Photo
- - - - -

Implement ramdisk with "lazy-write" of actively written files

imdisk ramdisk backup open-source strarc

  • Please log in to reply
11 replies to this topic

#1 AndyL

AndyL
  • Members
  • 3 posts
  •  
    United States

Posted 04 May 2013 - 03:59 PM

Thank you so much for writing imdisk and so many other open source windows applications.

 

I am wondering if imdisk would be suitable for a very particular usage I have in mind.

 

I wrote a large C code to acquire and process images from a scientific instrument I built (all open source.. see https://github.com/b...-frickin-lasers)

 

I use OpenCV's computer vision library to generate a video by adding a new frame to an open AVI file many times a second in a loop as the images arrive from the instrument. Writing to ramdisk speeds up my looping time, however, because everything is in ram, now if the computer crashes or if power goes out I will lose the entire AVI file.

 

One option would be to rewrite my code to change how I write to file. Another option though, ought to be to do periodic backups, every few seconds or so, using robocopy or strarc. This is what people seem to be calling "lazy-write." Robocopy, though, can't copy files that are currently being written, or "locked". Apparently windows shadow copy will do this, but it doesn't work with a commercial RAMDisk utility I tried from Dataram.

 

Would imdisk allow me to copy files  from ramdisk to disk using shadow copy or strarc and even files that are currently being written?

 

 

I can afford to lose the last few frames of video in my AVI file, and I have tools that can open a corrupted AVI file and recover everything else.

 

Thank you again.

 

Best,

Andy


  • Brito likes this

#2 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 04 May 2013 - 04:41 PM

Independently from whatever specifically IMDISK may or may not allow, you can always backup/copy a single file (as long as it is contiguous, and your file on ramdisk should always be so if I got right your setup :unsure:) by using direct access, i.e. using any dd-like tool and imaging the sectors the file occupies (since you are OK with losing a few frames).

 

:cheers:

Wonko 



#3 v77

v77

    Silver Member

  • Team Reboot
  • 602 posts
  •  
    France

Posted 04 May 2013 - 05:30 PM

This is quite weird... If you are adding only one frame at a time, the write caching of Windows should make the write almost instantly, and as fast as on a ramdisk...

 

If you didn't do this, I think you should check the flags if you are using the CreateFile function... especially FILE_FLAG_RANDOM_ACCESS and FILE_FLAG_SEQUENTIAL_SCAN:

http://msdn.microsof...8(v=vs.85).aspx

 

 

@Wonko:

There is no warranty, even on a ramdisk, that his file is not fragmented. And if this is the case, a copy by direct access will fail.



#4 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 04 May 2013 - 05:52 PM

@Wonko:

There is no warranty, even on a ramdisk, that his file is not fragmented. And if this is the case, a copy by direct access will fail.

Since when? :dubbio:

If the ramdisk is "brand new" (please read as freshly formatted) and you only write a single file on it, and that single file "grows" in time the file won't be fragmented on any FAT filesystem (on NTFS it may become fragmented because it may grow to a size overlapping the filesystem structures - which are also files).

 

In any case, IF it becomes fragmented you can use it's file extents and dd the corresponding chunks into a single output file.

Try getfileextents.exe or myfragmenter.exe (links in no apparent order in this thread):

http://www.msfn.org/...essful-unbrick/

http://www.msfn.org/...ost__p__1017553

Normally, if I can do something in batch :w00t: using third party small utilities a "real" programmer can manage to do it both "faster" and "better".... ;).

 

And yes, just for the record there is still the issue IF the file starts smaller than around 700 bytes and needs to be imaged before it exceeds some 744 bytes (exact length depending on filename)

http://www.forensicf...wtopic/t=10403/

but if it's a video capture, unless the resolution is something like 1x1 pixels :w00t: just the first frame will be bigger than 744 bytes.... :)

 

:cheers:

Wonko



#5 AndyL

AndyL
  • Members
  • 3 posts
  •  
    United States

Posted 04 May 2013 - 06:13 PM

@Wonky & @v77 Thank you both for these comments.

 

@v77 I also have been wondering about the windows write cache.  I would have thought that the windows write cache would have made writing just as fast as a RAMdisk If the windows write cache was working like this, then that would entirely solve my problem. All i know is that emprically, when I write to the C:\ drive i average 2.4 ms and when I write to ramdisk I average 1.1ms. (I measure elapsed time using some in-house routines that rely on Time.h).

 

The actual writing is handled by cvCreateVideoWriter and associated functions from the openCV library:

http://cgi.cs.indian...m#highgui_video

 

OpenCV abstracts away the low level part of actually writing the file. OpenCV is cross-platform and developed primarily with the GCC toolkit, so its not clear to me what sort of file access flags its using under the hood or if its even using any windows file writing flags at all.   (I compile with MINGW-w64, not Microsoft Visual Studio).

 

Perhaps I should get in touch with the OpenCV folks and learn a little more about their implementation under the hood. Maybe I could include the appropriate header files and use the MS specific createFile() function with some of the flags you suggest.



#6 v77

v77

    Silver Member

  • Team Reboot
  • 602 posts
  •  
    France

Posted 04 May 2013 - 06:16 PM

Since when? :dubbio:

If the ramdisk is "brand new" (please read as freshly formatted) and you only write a single file on it, and that single file "grows" in time the file won't be fragmented on any FAT filesystem (on NTFS it may become fragmented because it may grow to a size overlapping the filesystem structures - which are also files).

 

You are here assuming he is using a FAT filesystem, and no software/system service can write a hidden temporary file on that volume. This seems to be a dangerous practice...

 

Anyway, the answer is probably a lot more simple than using a ramdisk and tools for direct access to the volume. By the way, if the file is locked, this could be a bad use of the share flags of CreateFile.



#7 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 04 May 2013 - 06:31 PM

You are here assuming he is using a FAT filesystem, and no software/system service can write a hidden temporary file on that volume. This seems to be a dangerous practice...

Naah, not assuming anything, I would (personally and) intentionally use a FAT filesystem on a ramdisk, for the described use, since most of the "features" of NTFS seem to me like unneeded in this case.
And yes, before you additionally say anything about FAT32 having a 4 Gb file size limit ;), I would additionally divide the video file in "chunks" otherwise the "backup" process would, at each iteration, take more and more time, and casually   :whistling: I would choose for the "chunk" a size of 4,294,965,248 bytes:

http://reboot.pro/to...stick/?p=162473

but as said - at the most - I can put together some half-@§§ed batches :ph34r: and have much greater expectations from a "real" programmer.   :worship:

 

Anyway, the answer is probably a lot more simple than using a ramdisk and tools for direct access to the volume.

Sure :thumbup:.

 

:cheers:

Wonko



#8 v77

v77

    Silver Member

  • Team Reboot
  • 602 posts
  •  
    France

Posted 04 May 2013 - 07:29 PM

Naah, not assuming anything, I would (personally and) intentionally use a FAT filesystem on a ramdisk, for the described use, since most of the "features" of NTFS seem to me like unneeded in this case.
And yes, before you additionally say anything about FAT32 having a 4 Gb file size limit ;), I would additionally divide the video file in "chunks" otherwise the "backup" process would, at each iteration, take more and more time, and casually   :whistling: I would choose for the "chunk" a size of 4,294,965,248 bytes:

http://reboot.pro/to...stick/?p=162473

but as said - at the most - I can put together some half-@§§ed batches :ph34r: and have much greater expectations from a "real" programmer.   :worship:

 

This can work for you because, by luck, you have nothing that interferes, but this will not be the case for everyone. I have seen so much installers who are writing their files on a totally unexpected volume...

So, no need to be so sarcastic...



#9 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 04 May 2013 - 07:38 PM

This can work for you because, by luck, you have nothing that interferes, but this will not be the case for everyone. I have seen so much installers who are writing their files on a totally unexpected volume...

So, no need to be so sarcastic...

I wasn't attempting being sarcastic, sorry if I seemed to be so :(.

 

:cheers:

Wonko



#10 MedEvil

MedEvil

    Platinum Member

  • .script developer
  • 7771 posts

Posted 04 May 2013 - 09:54 PM

No idea how bad the powerlines are in the US, here in Germany a power failure is rather the exception.

But if the data is that precious, invest into a uninterrupted power supply. Then you won't loose any frames and can even continue with the data gathering, if the power supply is sufficently dimensioned.

:cheers:

#11 AndyL

AndyL
  • Members
  • 3 posts
  •  
    United States

Posted 04 May 2013 - 10:44 PM

Thanks @MedEvil, thats certainly not a bad idea. Still seems like there ought to be a way to make occasional on the fly backups in software, though.



#12 MedEvil

MedEvil

    Platinum Member

  • .script developer
  • 7771 posts

Posted 05 May 2013 - 12:12 PM

You maybe could do it.

But i think you've overlooked something.
You wanna use the ramdisk, because it is faster, than using the HDD. However, if you use the ramdisk for reading and writing at the same time, you basicly cut it's write percormance in half.

Accoring to your data, this would make it almost as slow as writing directly to HDD.

If you don't want to invest into an uninterrupted power supply. How about a small SSD?

:cheers:





Also tagged with one or more of these keywords: imdisk, ramdisk, backup, open-source, strarc

1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users