603 replies to this topic

[Vortex]

Hello Erwan,

 

Could you check CCleaner?

 

http://www.piriform....free-disk-space

[Wonko the Sane]

If anyone got a tool name in mind which can zeroed out the unused clusters without having to format...

Sysinternals Sdelete (but it is not redistributable) and tests need to be made, depending on versions for the effect of the -z vs. -c switch.
http://www.forensicf...568809/#6568809
http://serverfault.c...ter-compression

But creating a zero-filled file as large as the free space and then deleting it, would be "good enough" IMHO.

I'll have a look if I can find something "better/easier/simpler".

As a programmer, you will likely find something of interest in defragger and/or DiskBuddy (sources available):
https://web.archive....ls/toolsen.html


Wonko

[erwan.l]

Hello Erwan,

 

Could you check CCleaner?

 

http://www.piriform....free-disk-space

 

Thanks, indeed having used that software in the past, I was vaguely remembering something like that !

Will test.

[erwan.l]

Sysinternals Sdelete (but it is not redistributable) and tests need to be made, depending on versions for the effect of the -z vs. -c switch.
http://www.forensicf...568809/#6568809
http://serverfault.c...ter-compression

But creating a zero-filled file as large as the free space and then deleting it, would be "good enough" IMHO.

I'll have a look if I can find something "better/easier/simpler".

As a programmer, you will likely find something of interest in defragger and/or DiskBuddy (sources available):
https://web.archive....ls/toolsen.html


Wonko

 

A user has also kindly mailed me that tip (sdelete).

I will try it but I dont feel super confident about filling up the whole drive first : defo something I would not do on production machines.

 

About the second link, I'll have a look for sure : any source is always welcome

 

About coding,  I was tempted to try something like the following : :

-use FSCTL_GET_VOLUME_BITMAP  to retrieve a list of (NTFS) clusters

-for each free cluster, fill it with zero's

 

Of course, this would be only to save space when backuping/compressing a disk/volume to an image.

Definitely not something to do if you want to clone a pure 1 to 1.

 

Regards,

Erwan

[erwan.l]

About used clusters / unused clusters, here below an experiment I performed using two different methods / tools :

 

1/ made a new drive 256MB (formatted with NTFS, cluster size is 4096)

->total_clusters:65279 / free_clusters:60597 / used_clusters:4682

 

2/ copied 128MB of files (about 5000 of them)

->total_clusters:65279 / free_clusters:33489 / used_clusters:31790

compressed the drive to an ewf file ->  result is 26MB

 

3/ (shift) deleted all files

total_clusters:65279 / free_clusters:59507 / used_clusters:5772

compressed the drive to an ewf -> result is 26MB (near 1000 extra clusters compared to step 1...)

 

4/ zeroed out all free clusters (with my home made tool)

total_clusters:65279 / free_clusters:59507 / used_clusters:5772

compressed the drive to an ewf file -> result is 18MB

 

Did 1,2,3 again and 

 

4/ ran sdelete drive (as suggested in previous post)

total_clusters:65279 / free_clusters:59507 / used_clusters:5772

compressed the drive to  an ewf -> result is 2MB

 

So obviously sdelete is quiet efficient in freeing space (despite number used clusters being identical between method 1 & 2).

"My" approach (zeroing free clusters) although achieving some result is not perfect yet...

 

Also, if anyone can explained me where did my about 1000 extra used clusters go between step 1 and 4, I'd be grateful

And while at it, explain me why I get less garbage datas with sdelete despite number of used clusters staying the same between method 1 & 2 ... (although that question I should be able to answer myself while looking closer at sdelete...).

 

That was my wandering of the day...

 

/Erwan

[Wonko the Sane]

Are you clearing - besides the actual space occupied by data - also the remnants in metadata ($MFT, $Bitmap, etc.)?


Wonko

[erwan.l]

Version 2.2.2 uploaded :

 

added : zero out unused (ntfs) clusters

added : CompactVirtualDisk

 

The Clean Volume (i.e zero unused clusters) has been optimised :

useful before backuping a disk to an EWF or else before using CompactVirtualDisk (dynamic VHD only).

 

Note that the Clean Volume will not touch the MFT (resident datas will stay there).

Also, the approach is different than sdelete which fills the disk with many temp files and then delete the clusters used by these temp files.

Sdelete will also remove the unused resident datas (i.e files smaller than a file record, 512 bytes max on my drive).

 

And last but not least I really enjoy the EWF format.

Note fort its forensic purpose (as it was meant to) but because it a great compromise between speed and compression and moreover it is a format you can mount like any raw image file.

 

/Erwan

[Wonko the Sane]

erwan.l, on 21 May 2014 - 9:11 PM, said:
Sdelete will also remove the unused resident datas (i.e files smaller than a file record, 512 bytes max on my drive).

No. (meaning "NOT 512 bytes")
JFYI:
http://www.forensicf...wtopic/t=10403/

The file mentioned in the above is not anymore available, and for whatever reasons I cannot attach the file to this post, I reuploaded the batches used and the (short) instructions to a temporary storage: http://www.filedropper.com/mftcap


Wonko

P.S.: Edit attached file

Attached Files

•  MFT_cap.zip   2.28KB   607 downloads

[erwan.l]

No. (meaning "NOT 512 bytes")
JFYI:
http://www.forensicf...wtopic/t=10403/

The file mentioned in the above is not anymore available, and for whatever reasons I cannot attach the file to this post, I reuploaded the batches used and the (short) instructions to a temporary storage: http://www.filedropper.com/mftcap


Wonko

 

Well, I am not sure there is a size limit writing in stone.

Sdelete source code states 4096 bytes (max).

 

I did some tests myself :

-4096 : not resident

-2048 : not resident

-1024 : not resident

-512 : not resident

-256 : yes (i.e i could find my datas in the MFT)

 

So on my system (windows 7 sp1, ntfs), anything above 512 bytes would not be stored in the MFT.

So I'll play the stubborn guy and will maintain for now that on my system, resident datas will be between 256 and 512 bytes

I also did not check it but I believe MFT records are 2 sectors i.e 1024 byes long on my system.

 

Now googling around, it seems that some systems do have 4k long MFT records.

I dont know what determines a MFT record length for now.

[Wonko the Sane]

Well, I am not sure there is a size limit writing in stone.

 

It may well be not written in stone, and that is EXACTLY why I provided the testing tools.

 

It is as an example well possible that transactional NTFS (introduced in Vista ) or *whatever* has introduced changes that occupy space in the $MFT record (and thus allow for only a smaller file on your Windows 7 volume), another reason why tests are needed, but the idea is/was to find out the EXACT number of bytes, the width of the "leap" between 256, 512 and 1024 make no sense.

 

A "normal" $MFT record is 1024 bytes in length, so, OBVIOUSLY it cannot store anything "near" to 1024 bytes.

 

To determine the length of a $MFT record, simply open the disk and - starting at the $MFT begin address - check the frequence with which you can find the "FILE0" string (it will be every 1024 bytes, as a $MFT record begins with "FILE0" or "FILE*", this latter in previous versions).

 

In any case, since on the good ol' XP the "possible length" is function of the actual "filename length", allow me to doubt that later OS's/NTFS versions have made this "fixed" and "fixed to 512 bytes", I stand by my "NOT 512 bytes".

 

Wonko

[erwan.l]

Indeed, a file record on my hard drive is 1024 bytes (from one FILE0 to another FILE0).

Resident datas start at offset 280 so max resident datas in my case would be 1024-280 = 804 bytes in theory which is not the case in reality : I cannot explain it why but that wont affect my sleep too much

 

I aggree that the leap 256/512/1024 was confusing : it was a way for me to find a range (lower/upper).

Finding the exact number of bytes is actually not so important for me.

 

More important for me was to realise that actually the resident datas in the MFT of deleted files (i.e unused space in the MFT but on used clusters) is actually not that big.

I have developped a piece of code that goes thru the MFT records, spots deleted files, spots if datas is resident or not, and if resident, then fills it with zeroes. So that piece of code will stay in my draft folder.

The idea being preparing a drive for better compression (as zeros obviously compress better than junk datas).

Although it was good exercice for me (to understand the MFT better), the benefit (in size) is rather small.

 

More important for me was to zero the unused clusters.

There, the benefit is very significant when compacting a (dynamic) VHD or using EWF.

There it was worth adding a piece of code in CloneDisk

 

EDIT:

Using your batch (make_string_file.cmd), I managed to stuff exactly 664 bytes as resident datas.

Then (above that size), the datas became non resident BUT the 664 first bytes remained in the MFT as lost junk...

One could wish that MS was clean enough to NOT leave that junk data behind... but I guess this is a blessing for forensic guys !

 

It is sad thus to realise there is not much a user can do to keep his hard drive safe.

Take the following example : 

Create a small file containing a few userids/passwords.

Shift delete it.

Well there is a great chance that the datas actually stays forever ony our hard drive, lost in the MFT.

[Wonko the Sane]

And again not exactly.

The length of the data that will be stored on the $MFT is not "fixed", as it depends on the length of the filename.

So is the 664 the max with the 3 letter filename? (i.e. corresponding to the 744 I can get on XP?)

 

And yes, this is one of the effects, if you start typing (say) plain text files and save frequently, your incipits http://en.wikipedia.org/wiki/Incipit will remain in the $MFT forever....

 

 

Wonko

[David Lynch]

Maybe Speed Test could check also network drives?...

 

&  

[Wonko the Sane]

Erwan.l

 

Version 2.2.3 does NOT open a choose dialog for "Convert RAW to VMDK/VHD" at least here.

 

Last I have that works is 2.1.6 (but any intermediate release may well work fine).

 

Usual caveman questions :

1. WHY (the heck) don't you name the file somewhat meaningfully (i.e. NOT "clonedisk.zip" fixed, no matter the version)?
2. WHY (the heck) you don't have a repository (or something similar) where older version can be got from?

 

Quick dictionary reference (JFYI):

• New means "new".
• Better means "better".
• Latest means "latest".
• BOTH new and latest do not mean "better". 

Wonko

[erwan.l]

Erwan.l

 

Version 2.2.3 does NOT open a choose dialog for "Convert RAW to VMDK/VHD" at least here.

 

Last I have that works is 2.1.6 (but any intermediate release may well work fine).

 

Usual caveman questions :

1. WHY (the heck) don't you name the file somewhat meaningfully (i.e. NOT "clonedisk.zip" fixed, no matter the version)?
2. WHY (the heck) you don't have a repository (or something similar) where older version can be got from?

 

Quick dictionary reference (JFYI):

• New means "new".
• Better means "better".
• Latest means "latest".
• BOTH new and latest do not mean "better". 

Wonko

 

Hi Wonko,

 

I believe you spot a bug indeed.

Will be fixed later today.

 

About why not renaming the zip file for each new version?

Because I can

Or more precisely because I have a script that uploads that zip everywhere needed (ftp, dropbox, etc) and that different external sources (starting with reboot.pro are linked to clonedisk.zip).

 

Why dont I store previous versions on a repository?

Because I prefer to avoid older versions in the wild.

 

Call me "Lazy bxxtard" and you would be right

[Wonko the Sane]

It's not about you being a lazy bastard (which probably you are ), that's fine with me (as I am also a bastard, though an old and grumpy but not lazy one ).
It is about the fact that I lost half an hour today because of this little bug AND that this issue could be solved in no time if a "previous" version was available.
And of course you can, point was only about letting you know how foolish this is, now you know and everything is cool.
The generic idea of having a hosting area on reboot.pro is to have (hopefully) a DUPLICATE storage (and you can have all versions on here and not a stupid link to the file on your host).

I will start a new thread to see if any of the other peeps is willing to contribute to rebuilding a §@ç#ing repository.


Wonko

[erwan.l]

Seems i had solved the bug already as button works fine over here.
I have re uploaded the zip file.

Give it a go and let me know.

Not related but a "new" version (did not say better ) should be coming soon that should include vmdk, qcow and vhd reading.

[Zoso]

FWIW I think what Wonko has suggested is absolutely proper and makes total sense. I dont upgrade because of this except when I need to try something a newer release might help with.

at the very least though, I wish you would put the version number in the file names. youre not the only lazy one ;-)


thanks

[David Lynch]

On my tests on Windows XP SP3, offline registry > delete currentcontrolset\enum should not delete sw and root subkeys as they are needed for audio and other basic functionalities. Those keys are not completely rebuilt upon hardware detection.

 

Could someone confirm this?

[Wonko the Sane]

On my tests on Windows XP SP3, offline registry > delete currentcontrolset\enum should not delete sw and root subkeys as they are needed for audio and other basic functionalities. Those keys are not completely rebuilt upon hardware detection.

Could someone confirm this?

Can you explain?
WHY (the heck) would you actually want to delete all contents of the currentcontrolset\enum\ hive?
And WHY should a plain, simple, tool have "safeguards" against doing something that the user asked for?



Wonko

[erwan.l]

On my tests on Windows XP SP3, offline registry > delete currentcontrolset\enum should not delete sw and root subkeys as they are needed for audio and other basic functionalities. Those keys are not completely rebuilt upon hardware detection.

 

Could someone confirm this?

 

delete currentcontrolset\enum will delete all keys and subkeys.

my tests showed that xp would redetect it all at next reboot.

thus i did not test it extensively.

 

to be used with extreme care thus and only if you know what you are doing (testing, last chance action, etc).

[David Lynch]

Tested here and needed to restore SW and Root keys for audio to work.

 

I've tried to find a MS or Technet KB regarding this and I had no success. Many times a enum key rebuild is wanted; scenarios for that goes from infections to being unable to open device manager, devices erratic behaviour, etc.

 

It is a great feature but maybe it needs some changes. I'm not completely sure of that, so I'm just asking here.

Edited by David Lynch, 16 November 2014 - 09:20 PM.

[erwan.l]

Tested here and needed to restore SW and Root keys for audio to work.

 

I've tried to find a MS or Technet KB regarding this and I had no success. Many times a enum key rebuild is wanted; scenarios for that goes from infections to being unable to open device manager, devices erratic behaviour, etc.

 

It is a great feature but maybe it needs some changes. I'm not completely sure of that, so I'm just asking here.

 

under root, i suspect that only the MEDIA key is needed for your audio.

and under sw, probably one subkey is needed for your audio.

 

still, why is not windows able to rebuild these i could not tell...

 

i usually delete the enum key when i have messed up my vm's and it did help me more than once.

never used it in real life thus.

[Wonko the Sane]

ONLY to cheer you guys up a little bit, as often happens OT , but not much :

http://rwmj.wordpres...ks-technically/

 

Wonko

[erwan.l]

Tested here and needed to restore SW and Root keys for audio to work.

 

I've tried to find a MS or Technet KB regarding this and I had no success. Many times a enum key rebuild is wanted; scenarios for that goes from infections to being unable to open device manager, devices erratic behaviour, etc.

 

It is a great feature but maybe it needs some changes. I'm not completely sure of that, so I'm just asking here.

 

If you are willing to perform tasks against an offline registry hive, I would recommend using OfflineReg .

 

There you could batch something like :

-save the root & sw key

-delete the enum key

-recreate it (empty)

-restore root & sw key

 

Regards,

Erwan