42 replies to this topic

[ispy]

Hi all ,

Here's what I think is an intersting find?

Ever drop on a file or collection of files that you want to copy, that spits back an error message sumfin like "File in use" unable to copy which can be a bit of a nuisance!

I like the commandline quite a lot so this little gizmo is right up my alley & maybe up yours, well how do we achieve this then, "Copying locked files"?!
Since XP, Windows has supported a technology called Volume Shadow Copy, which is used to power the previous versions feature in Vista as well as System Restore and backups. What it does is take a temporary snapshot of the file or drive, and then allow an application to read from the snapshot even while other applications are accessing or modifying the file.

What we can do is use a command line utility called HoboCopy that utilizes this service to copy the file.

Well maybe just maybe it could be used to say back up your active registry why not give it a go & find out, an interesting topic for a homework project LOL?

And where can one find out details about this Gizmo:?

Try Here:

http://www.howtogeek...ked-in-windows/

Noteably it is open source 2

http://sourceforge.n...ckage_id=204974

There are dependancies of course but isn't there always, kinda looks nifty to me?

Tis also avaialable for Vista also

N-Joy!

Regards & Best Wishes,

ispy

Addendum UMMmmm Maybe its the case that some people would prefer a Gui interface so as an alternative to HoboCopy what about this:

http://en.kioskea.ne...712-shadow-copy

Maybe this would suffice

N-Joy!

R&R,

ispy

[was_jaclaz]

Good find (hobocopy).



The idea is nice, I guess that Olof could include the feature in strarc.

Right now he is using a workaround that might be less efficient

[quote name='http://www.ltr-data.se/files/strarc.txt']3.6 How to backup a complete running Windows system.

Complete running Windows systems may be backed up in the ways described in the
3.4 section. Special care must however be taken when it comes to files that may
be open while the backup is running. This is always the case of the registry
database files in the %SystemRoot%\System32\config directory and the profile
directories of logged on users. This means that strarc cannot backup the
registry database files the way normal files are backed up while the system is
running.

With release 0.1.3 of strarc it is now possible to backup snapshots of the
registry database files using the -r switch to strarc.

The snapshot files are extracted from the registry keys of the running system
before the actual backup routine of strarc begins and will therefore be a
momentary copy of the contents of the registry when strarc starts. The snapshot
files are temporary files created with the same name as the loaded registry
files with the special extension '.$sards', "Stream Archiver Registry Database
Snapshot". When the backup routine later in the strarc backup run finds the
snapshot files they are stored in the backup archive with the name of the actual
loaded registry file, not the name of the temporary snapshot file. This way a
later restore of the backup archive will recreate the snapshots with the name
and path of registry database files Windows expects to find.

The temporary snapshot files with the special extension '.$sards' are deleted
automatically by the backup routine after they have been stored in the backup
archive. However, if the backup operation is cancelled before the snapshot files
are backed up or if a directory with some of the snapshots are not included in
the backup run, the snapshot files will be left on the disk. You can safely
delete them after a strarc backup operation is complete.[/quote]

For "normal" use, I've always used unlocker:
http://ccollomb.free.fr/unlocker/

What about the overall speed of hobocopy when compared to other tools?

There can be big differences in speed of copying apps, expecially under Vista (did I say en passant that Vista sucks IMNSHO ):
http://www.msfn.org/...howtopic=125248
http://www.msfn.org/...howtopic=120677

For the record, Shadow Copy is from Runtime.org:
http://www.runtime.org/
http://www.runtime.org/shadow-copy.htm

Not really "a new kid on the block" ... ....you may remember an app called DriveimageXML that uses VSS to copy partitions....

jaclaz

[ispy]

Hi Jaclaz ,

Always a pleasure doing business with you! The author has written the project as a hobby project but to me has potential. Improvements will be 4thcoming in the future.

The additional info in reply is also illuminating from you, Many thanks!

The speed at which it accomplishes the copying process is anyones guess at this stage & I suppose is dependant on the hardware spec of the host PC?

Regards & Best Wishes,

ispy

[paraglider]

ShadowCopy only works on 32 bit windows OS. Hobocopy has a version for 64 bit OS.

[j7n]

Shadow Copy is an interesting and occasionally useful tool. I've encountered a few programs that keep files open for writing as a protection. However, I am not able to use Shadow Copy due to absence of the actual Volume Shadow Copy service which I removed several years ago during installation. I've read about restoring it on MSFN but I don't believe I can do it without drastic measures such as reinstallation, since VSC isn't working well for me on another nLited computer where the service was left alone, but others have been removed.

 

VideoCacheView still manages to copy locked Flash videos on this machine. I wonder if it could be possible to somehow extend this functionality to copy any type of file. Or rather that there was an universal copyer like VCV, without dependencies on the system.

[Wonko the Sane]

I wonder if it could be possible to somehow extend this functionality to copy any type of file. Or rather that there was an universal copyer like VCV, without dependencies on the system.

If your issue is copying files that are in use or somehow "locked", as long as the file is not in a situation in which it is dynamically updated/changed/written to - you can use direct disk access.

Get the location of the file on disk.

dd (or dsfo, etc.) the corresponding sectors.

re-assemble the file to a new path/new name.

 

The procedure and some suitable tools are described in this thread:

http://www.msfn.org/...essful-unbrick/

the tests there resulted in myfragmenter.exe being more reliable than getFileExtents.exe, so start from here:
http://www.msfn.org/...ost__p__1017553

 

 

Wonko

[j7n]

This gave me an idea. I wrongly assumed that direct access to disk wouldn't be possible without leaving Windows or unmounting the file system (and stopping the program which created the protected file).

 

I used WinHex to successfully copy the intended file. The program opens a logical partition and parses the file allocation table itself. It was also possible to copy the windows registry, which is not very useful because the file of course is dynamically updated.

The versatility of WinHex impresses me. It's not free though.

[Wonko the Sane]

The versatility of WinHex impresses me. It's not free though.

Yep, that's why i proposed a way to do the same through completely free tools, any disk editor will do, the "trick" is knowing the area(s) where the file is placed.

 

Among the free tools, DMDE can do it easily (if you prefer GUI to command line):

http://dmde.com/

 

Wonko

[joakim]

Basically it all boils down to resolving the filesystem system structures yourself (or the tool used does it for you). All tools mentioned so far (except last 2) takes shortcuts, meaning they depend on the filesystem somehow one way or another. Even using winapi functions like CreateFile, ReadFile, WriteFile, SetFilePointerEx etc like some of my tools does will depend on Windows obviously, however at least not on the filesystem. Removing the depency on those functions essentially makes your tool ready for porting to linux (think C). But if Windows is preferred/required, then utilizing those functions mentioned above are the core minimum of permissions you need (like CreateFile on volume or PhysicalDrive. The rest is up to you. And then nothing can stop you from reading sectors otherwise protected by measures through filesystem.

 

Take WinHex and DMDE as an example. Yes it can backup protected files for you, because it does what I mentioned above. However it is limited to the $DATA attribute (I assume NTFS based on the information provided so far, like shadow copy service etc), meaning they don't support extraction of any of the other 14 attributes. So even those has limitations. Just to mention DMDE, a quick test here reveals it can not even extract alternate data streams ($DATA). Not sure if that is a limitation in the free edition only. But if not, then charging money for it is a rip off. On the other hand, a nice GUI counts, but come on..

[v77]

Recuva is also able to copy locked files, by checking the "Scan for non-deleted files" option.

[joakim]

Recuva is also able to copy locked files, by checking the "Scan for non-deleted files" option.

Just as DMDE, it can only recover the first un-named data stream. But for the average user that is ok I guess.

[Wonko the Sane]

Just to mention DMDE, a quick test here reveals it can not even extract alternate data streams ($DATA). Not sure if that is a limitation in the free edition only. But if not, then charging money for it is a rip off. On the other hand, a nice GUI counts, but come on..

 

Well, I have the utter respect for DMDE Author .
 
He provides for free an exceptionally good app with the only limitation that it can recover one file at the time and a few other limitations, that are not commonly a problem.
 
Accusing him of a rip off is - as I see it - very UNfair.
 
The intended scope of DMDE is to recover files, and UNLIKE Recuva (which like many automated tools is pretty much unuseful if you need anything more that "standard"), it is very good at providing a whole lot of "side features" that allow to recover manually also bits and pieces.
 
I personally find DMDE one of the best pieces of software you can have for free , but noone is forced to use it, and noone, specifically, is forced to buy the Commercial version of it.
 
In any case DMDE is not a competitor for WinHex, the most expensive Commercial version of DMDE is 60 Euros (and BTW can recover ADS alright):
http://dmde.com/editions.html
whilst the "real thing" in Winhex is the Forensic Edition:
http://www.x-ways.ne...comparison.html
which is around 750 Euro's (including the 25% discount for "no maintenance"), and the "entry level" with some advanced feature is anyway the "specialist" at around 200 Euro.
 
The OP asked a question, a suitable (freeware) command line solution was given to him, then a suitable GUI one (actually two, including Recuva) were added.
 

Wonko

[erwan.l]

Hi Guys,

 

Based on this discussion, I decided to gave a quick look at FSCTL_GET_RETRIEVAL_POINTERS  .

 

I then coded this quick GUI : http://reboot.pro/fi...le/316-extents/ .

 

The idea is to list all clusters used by a file on a logical drive.

Once we have that list, we will translate it to offsets on the drive and dump these clusters on a destination file.

 

Doing so, one can dump in used files.

I tested it with \boot\bcd and \windows\system32\config\sam.

 

This is only a proof of concept.

 

if of any use, I could make it a command line tool and add extra features like manage damaged clusters, etc.

 

Regards,

Erwan

Attached Thumbnails

• 

[Wonko the Sane]

if of any use, I could make it a command line tool...

There is always some use for something.

 

Yes, this would avoid the need of use two tools (like mydefragmenter.exe and dsfo.exe/dsfi.exe).

It would be nice if the tool would also print out the actual clusters/sectors involved in the file.

 

Please also note how what is also missing (if wishes are allowed ) is getting files that being smaller than some 744 bytes are stored directly in the $MFT (though it is unlikely that they will be "locked", adding the possibility of getting them may be an added feature, a typical file that resided entirely in the $MFT is BOOT.INI ):

http://www.forensicf...wtopic/t=10403/

 

Wonko

[erwan.l]

It would be nice if the tool would also print out the actual clusters/sectors involved in the file.

 

 

Wonko

 

Done

now need to work on the command line and eventually the MFT thingy.

 

/erwan

Attached Thumbnails

• 

[erwan.l]

i have added a poor man's command line :

 

extents -details x:\any.file

extents -backup x:\source.file x:\destination.file

 

/erwan

[joakim]

Sorry, but the "extents" utility also takes shortcuts instead of resolving the relevant clusters from the datarun list in the file's $MFT record.

 

If it can't copy the registry hives that are in use by the system (for instance C:\WINDOWS\system32\config\SAM), then it takes shortcuts. That is a very simpel test.

[erwan.l]

Hi Joakim,

 

I dont understand what you mean about shortcuts?

From MS web page about FSCTL_GET_RETRIEVAL_POINTERS : "Given a file handle, retrieves a data structure that describes the allocation and location on disk of a specific file" .

 

Once you have the list of cluster's offset for a file, it is very easy to open a handle to the logical drive, and dump each block/clusters from source drive to a destination file.

 

Can you tell me more about these shortcuts? Do you mean the cluster offset I get for my SAM file are actually not the real ones?

I'd be surprised thus as I use these to dump clusters from a handle belonging to the logical drive and my dumped file looks identical to the original file.

 

Regards,

Erwan

[joakim]

OK maybe I wrongly assumed something because the program returned "invalid handle,2" when trying it on C:\WINDOWS\system32\config\system. Anyways, the program was tested on a virtual Windows 7 x64 SP1.

 

Edit: and of course since I did not reverse engineer the program in any way, I can't really backup the statement of shortcuts. Are the source code available?

[Wonko the Sane]

The extents utility  created a copy of C:\WINDOWS\system32\config\SAM on my XP SP2 (obviously 32 bit), maybe it's a Win 7 or 64 bit connected issue.

(and as well one of C:\WINDOWS\system32\config\system though how "valid" this latter can be is most probably debatable)

 

Wonko

[erwan.l]

code error 2 means : file not found.

on a x64 system, is the registry under system32? dont know.

note that I performed my test on win 7 32 bits.

 

About source code, it will be part of the package once cleanup.

 

the big lines :

 

-to retrieve the clusters used by a file :

//not the read_attributes flag, the only one that will work on a in used file

//this flag is enough for the retrieval_pointers, but not enough to read it directly from the file

hFile := CreateFile(lpFileName,FILE_READ_ATTRIBUTES, FILE_SHARE_READ,nil,OPEN_EXISTING,   FILE_FLAG_NO_BUFFERING, 0);

...

DeviceIoControl(hFile, FSCTL_GET_RETRIEVAL_POINTERS, @InBuf, SizeOf(InBuf), OutBuf, OutSize, Bytes, nil)

...

 

-to back it up

hDrive := CreateFile(Name, GENERIC_READ, FILE_SHARE_READ or FILE_SHARE_WRITE, nil, OPEN_EXISTING, 0, 0);

...

hFile := CreateFile(lpDstName, GENERIC_WRITE, 0, nil, CREATE_NEW, 0, 0);

...

SetFilePointer(hDrive, Offset.LowPart, @Offset.HighPart, FILE_BEGIN);

ReadFile(hDrive, Buff^, ClusterSize, Bytes, nil);

BlockSize := Min(FileSize, ClusterSize);

WriteFile(hFile, Buff^, BlockSize, Bytes, nil);

...

 

/erwan

 

[erwan.l]

Note : about dumping an in used file this way as opposed to snapshot (vss), one need to be careful.

 

As Wonko mentionned above, the resulting file could be corrupted if there was changes made during the dump.

A good safety measure could be to dump the file twice and to compare both files.

I am thinking about adding a md5 checksum on top of the backup.

 

/erwan

[Wonko the Sane]

Note : about dumping an in used file this way as opposed to snapshot (vss), one need to be careful.

 

As Wonko mentionned above, the resulting file could be corrupted if there was changes made during the dump.

A good safety measure could be to dump the file twice and to compare both files.

I am thinking about adding a md5 checksum on top of the backup.

 

/erwan

 

Both ways (double backup OR MD5 hash) are however only a way to "verify" that a backup with this method worked, i.e. if the two saved files differ (or their MD5 hash differs) you know that you have different files, but you don' t know which one of the two is "good" (if any) then you do it again, and you need to compare the two files again, and again and again until by sheer luck you manage to copy it twice with the same result.

 

As long as the file is very small, you may be lucky  but, with a file like C:\Windows\system32\config\system which is read and written to continuously, the probabilities of having a "good" match are very, very, very low, for these VSS is possibly the only way.

 

Wonko

[erwan.l]

Hi Wonko,

 

Actually, VSS is not the only way.

Registry windows API also offer the possibility to backup your hives.

 

Regards,

Erwan

[Wonko the Sane]

Hi Wonko,

 

Actually, VSS is not the only way.

Registry windows API also offer the possibility to backup your hives.

 

Regards,

Erwan

Sure, I was thinking of *any* largish file which is continuously written, let's say then a .vhd with a VM booted and defrag running in it .

 

Wonko