Jump to content











Photo
* * * * * 4 votes

CloneDisk


  • Please log in to reply
595 replies to this topic

#451 Vortex

Vortex

    Frequent Member

  • Advanced user
  • 239 posts

Posted 14 May 2014 - 06:38 AM

Hello Erwan,

 

Could you check CCleaner?

 

http://www.piriform....free-disk-space



#452 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15030 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 14 May 2014 - 08:53 AM



If anyone got a tool name in mind which can zeroed out the unused clusters without having to format...

Sysinternals Sdelete (but it is not redistributable) and tests need to be made, depending on versions for the effect of the -z vs. -c switch.
http://www.forensicf...568809/#6568809
http://serverfault.c...ter-compression

But creating a zero-filled file as large as the free space and then deleting it, would be "good enough" IMHO.

I'll have a look if I can find something "better/easier/simpler".

As a programmer, you will likely find something of interest in defragger and/or DiskBuddy (sources available):
https://web.archive....ls/toolsen.html

:duff:
Wonko

#453 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2783 posts
  • Location:Nantes - France
  •  
    France

Posted 14 May 2014 - 09:58 AM

Hello Erwan,

 

Could you check CCleaner?

 

http://www.piriform....free-disk-space

 

Thanks, indeed having used that software in the past, I was vaguely remembering something like that !

Will test.



#454 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2783 posts
  • Location:Nantes - France
  •  
    France

Posted 14 May 2014 - 10:06 AM

Sysinternals Sdelete (but it is not redistributable) and tests need to be made, depending on versions for the effect of the -z vs. -c switch.
http://www.forensicf...568809/#6568809
http://serverfault.c...ter-compression

But creating a zero-filled file as large as the free space and then deleting it, would be "good enough" IMHO.

I'll have a look if I can find something "better/easier/simpler".

As a programmer, you will likely find something of interest in defragger and/or DiskBuddy (sources available):
https://web.archive....ls/toolsen.html

:duff:
Wonko

 

A user has also kindly mailed me that tip (sdelete).

I will try it but I dont feel super confident about filling up the whole drive first : defo something I would not do on production machines.

 

About the second link, I'll have a look for sure : any source is always welcome :)

 

About coding,  I was tempted to try something like the following : :

-use FSCTL_GET_VOLUME_BITMAP  to retrieve a list of (NTFS) clusters

-for each free cluster, fill it with zero's

 

Of course, this would be only to save space when backuping/compressing a disk/volume to an image.

Definitely not something to do if you want to clone a pure 1 to 1.

 

Regards,

Erwan



#455 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2783 posts
  • Location:Nantes - France
  •  
    France

Posted 14 May 2014 - 08:42 PM

About used clusters / unused clusters, here below an experiment I performed using two different methods / tools :

 

1/ made a new drive 256MB (formatted with NTFS, cluster size is 4096)
->total_clusters:65279 / free_clusters:60597 / used_clusters:4682
 
2/ copied 128MB of files (about 5000 of them)
->total_clusters:65279 / free_clusters:33489 / used_clusters:31790
compressed the drive to an ewf file ->  result is 26MB
 
3/ (shift) deleted all files
total_clusters:65279 / free_clusters:59507 / used_clusters:5772
compressed the drive to an ewf -> result is 26MB (near 1000 extra clusters compared to step 1...)
 
4/ zeroed out all free clusters (with my home made tool)
total_clusters:65279 / free_clusters:59507 / used_clusters:5772
compressed the drive to an ewf file -> result is 18MB
 
Did 1,2,3 again and 
 
4/ ran sdelete drive (as suggested in previous post)
total_clusters:65279 / free_clusters:59507 / used_clusters:5772
compressed the drive to  an ewf -> result is 2MB
 
So obviously sdelete is quiet efficient in freeing space (despite number used clusters being identical between method 1 & 2).
"My" approach (zeroing free clusters) although achieving some result is not perfect yet...
 
Also, if anyone can explained me where did my about 1000 extra used clusters go between step 1 and 4, I'd be grateful :)
And while at it, explain me why I get less garbage datas with sdelete despite number of used clusters staying the same between method 1 & 2 ... (although that question I should be able to answer myself while looking closer at sdelete...).
 
That was my wandering of the day...
 
/Erwan


#456 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15030 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 15 May 2014 - 08:41 AM

Are you clearing - besides the actual space occupied by data - also the remnants in metadata ($MFT, $Bitmap, etc.)? :dubbio:

:duff:
Wonko

#457 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2783 posts
  • Location:Nantes - France
  •  
    France

Posted 21 May 2014 - 08:11 PM

Version 2.2.2 uploaded :

 

added : zero out unused (ntfs) clusters
added : CompactVirtualDisk
 
The Clean Volume (i.e zero unused clusters) has been optimised :
useful before backuping a disk to an EWF or else before using CompactVirtualDisk (dynamic VHD only).
 
Note that the Clean Volume will not touch the MFT (resident datas will stay there).
Also, the approach is different than sdelete which fills the disk with many temp files and then delete the clusters used by these temp files.
Sdelete will also remove the unused resident datas (i.e files smaller than a file record, 512 bytes max on my drive).
 
And last but not least I really enjoy the EWF format.
Note fort its forensic purpose (as it was meant to) but because it a great compromise between speed and compression and moreover it is a format you can mount like any raw image file.
 
/Erwan


#458 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15030 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 22 May 2014 - 10:28 AM

erwan.l, on 21 May 2014 - 9:11 PM, said:
Sdelete will also remove the unused resident datas (i.e files smaller than a file record, 512 bytes max on my drive).

No. (meaning "NOT 512 bytes")
JFYI:
http://www.forensicf...wtopic/t=10403/

The file mentioned in the above is not anymore available, and for whatever reasons I cannot attach the file to this post, I reuploaded the batches used and the (short) instructions to a temporary storage: http://www.filedropper.com/mftcap

:duff:
Wonko

P.S.: Edit attached file

Attached Files



#459 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2783 posts
  • Location:Nantes - France
  •  
    France

Posted 22 May 2014 - 01:44 PM

No. (meaning "NOT 512 bytes")
JFYI:
http://www.forensicf...wtopic/t=10403/

The file mentioned in the above is not anymore available, and for whatever reasons I cannot attach the file to this post, I reuploaded the batches used and the (short) instructions to a temporary storage: http://www.filedropper.com/mftcap

:duff:
Wonko

 

Well, I am not sure there is a size limit writing in stone.

Sdelete source code states 4096 bytes (max).

 

I did some tests myself :

-4096 : not resident

-2048 : not resident

-1024 : not resident

-512 : not resident

-256 : yes (i.e i could find my datas in the MFT)

 

So on my system (windows 7 sp1, ntfs), anything above 512 bytes would not be stored in the MFT.

So I'll play the stubborn guy and will maintain for now that on my system, resident datas will be between 256 and 512 bytes :)

I also did not check it but I believe MFT records are 2 sectors i.e 1024 byes long on my system.

 

Now googling around, it seems that some systems do have 4k long MFT records.

I dont know what determines a MFT record length for now.



#460 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15030 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 22 May 2014 - 02:17 PM

Well, I am not sure there is a size limit writing in stone.

 

It may well be not written in stone, and that is EXACTLY why I provided the testing tools.

 

It is as an example well possible that transactional NTFS (introduced in Vista :ph34r:) or *whatever* has introduced changes that occupy space in the $MFT record (and thus allow for only a smaller file on your Windows 7 volume), another reason why tests are needed, but the idea is/was to find out the EXACT number of bytes, the width of the "leap" between 256, 512 and 1024 make no sense.

 

A "normal" $MFT record is 1024 bytes in length, so, OBVIOUSLY it cannot store anything "near" to 1024 bytes.

 

To determine the length of a $MFT record, simply open the disk and - starting at the $MFT begin address - check the frequence with which you can find the "FILE0" string (it will be every 1024 bytes, as a $MFT record begins with "FILE0" or "FILE*", this latter in previous versions).

 

In any case, since on the good ol' XP the "possible length" is function of the actual "filename length", allow me to doubt that later OS's/NTFS versions have made this "fixed" and "fixed to 512 bytes", I stand by my "NOT 512 bytes".

 

:duff:

Wonko



#461 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2783 posts
  • Location:Nantes - France
  •  
    France

Posted 23 May 2014 - 05:27 PM

Indeed, a file record on my hard drive is 1024 bytes (from one FILE0 to another FILE0).

Resident datas start at offset 280 so max resident datas in my case would be 1024-280 = 804 bytes in theory which is not the case in reality : I cannot explain it why but that wont affect my sleep too much :)

 

I aggree that the leap 256/512/1024 was confusing : it was a way for me to find a range (lower/upper).

Finding the exact number of bytes is actually not so important for me.

 

More important for me was to realise that actually the resident datas in the MFT of deleted files (i.e unused space in the MFT but on used clusters) is actually not that big.

I have developped a piece of code that goes thru the MFT records, spots deleted files, spots if datas is resident or not, and if resident, then fills it with zeroes. So that piece of code will stay in my draft folder.

The idea being preparing a drive for better compression (as zeros obviously compress better than junk datas).

Although it was good exercice for me (to understand the MFT better), the benefit (in size) is rather small.

 

More important for me was to zero the unused clusters.

There, the benefit is very significant when compacting a (dynamic) VHD or using EWF.

There it was worth adding a piece of code in CloneDisk :)

 

EDIT:

Using your batch (make_string_file.cmd), I managed to stuff exactly 664 bytes as resident datas.

Then (above that size), the datas became non resident BUT the 664 first bytes remained in the MFT as lost junk...

One could wish that MS was clean enough to NOT leave that junk data behind... but I guess this is a blessing for forensic guys !

 

It is sad thus to realise there is not much a user can do to keep his hard drive safe.

Take the following example : 

Create a small file containing a few userids/passwords.

Shift delete it.

Well there is a great chance that the datas actually stays forever ony our hard drive, lost in the MFT.



#462 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15030 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 23 May 2014 - 09:42 PM

And again not exactly.

The length of the data that will be stored on the $MFT is not "fixed", as it depends on the length of the filename.

So is the 664 the max with the 3 letter filename? (i.e. corresponding to the 744 I can get on XP?)

 

And yes, this is one of the effects, if you start typing (say) plain text files and save frequently, your incipits http://en.wikipedia.org/wiki/Incipit will remain in the $MFT forever.... :ph34r:

 

 

:duff:

Wonko



#463 David Lynch

David Lynch

    Member

  • Members
  • 34 posts
  •  
    United States

Posted 03 July 2014 - 05:50 PM

Maybe Speed Test could check also network drives?...

 

jg2t4.gif & f321V.gif 



#464 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15030 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 10 October 2014 - 12:14 PM

Erwan.l

 

Version 2.2.3 does NOT open a choose dialog for "Convert RAW to VMDK/VHD" at least here. :(

 

Last I have that works is 2.1.6 (but any intermediate release may well work fine).

 

Usual caveman questions :w00t: :ph34r::

  1. WHY (the heck) don't you name the file somewhat meaningfully (i.e. NOT "clonedisk.zip" fixed, no matter the version)?
  2. WHY (the heck) you don't have a repository (or something similar) where older version can be got from?

 

Quick dictionary reference (JFYI):

  • New means "new".
  • Better means "better".
  • Latest means "latest".
  • BOTH new and latest do not mean "better". 

:duff:

Wonko



#465 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2783 posts
  • Location:Nantes - France
  •  
    France

Posted 10 October 2014 - 02:20 PM

Erwan.l

 

Version 2.2.3 does NOT open a choose dialog for "Convert RAW to VMDK/VHD" at least here. :(

 

Last I have that works is 2.1.6 (but any intermediate release may well work fine).

 

Usual caveman questions :w00t: :ph34r::

  1. WHY (the heck) don't you name the file somewhat meaningfully (i.e. NOT "clonedisk.zip" fixed, no matter the version)?
  2. WHY (the heck) you don't have a repository (or something similar) where older version can be got from?

 

Quick dictionary reference (JFYI):

  • New means "new".
  • Better means "better".
  • Latest means "latest".
  • BOTH new and latest do not mean "better". 

:duff:

Wonko

 

Hi Wonko,

 

I believe you spot a bug indeed.

Will be fixed later today.

 

About why not renaming the zip file for each new version?

Because I can :)

Or more precisely because I have a script that uploads that zip everywhere needed (ftp, dropbox, etc) and that different external sources (starting with reboot.pro are linked to clonedisk.zip).

 

Why dont I store previous versions on a repository?

Because I prefer to avoid older versions in the wild.

 

Call me "Lazy bxxtard" and you would be right ;)



#466 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15030 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 10 October 2014 - 02:44 PM

It's not about you being a lazy bastard (which probably you are :whistling:), that's fine with me :) (as I am also a bastard, though an old and grumpy :w00t: but not lazy one :ph34r:).
It is about the fact that I lost half an hour today because of this little bug :ranting2: AND that this issue could be solved in no time if a "previous" version was available.
And of course you can, point was only about letting you know how foolish this is, now you know and everything is cool.
The generic idea of having a hosting area on reboot.pro is to have (hopefully) a DUPLICATE storage (and you can have all versions on here and not a stupid link to the file on your host).

I will start a new thread to see if any of the other peeps is willing to contribute to rebuilding a §@ç#ing repository.

:duff:
Wonko

#467 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2783 posts
  • Location:Nantes - France
  •  
    France

Posted 10 October 2014 - 06:27 PM

Seems i had solved the bug already as button works fine over here.
I have re uploaded the zip file.

Give it a go and let me know.

Not related but a "new" version (did not say better ;) ) should be coming soon that should include vmdk, qcow and vhd reading.

#468 Zoso

Zoso

    Silver Member

  • Advanced user
  • 640 posts
  •  
    Isle of Man

Posted 10 October 2014 - 06:53 PM

FWIW I think what Wonko has suggested is absolutely proper and makes total sense. I dont upgrade because of this except when I need to try something a newer release might help with.

at the very least though, I wish you would put the version number in the file names. youre not the only lazy one ;-)


thanks

#469 David Lynch

David Lynch

    Member

  • Members
  • 34 posts
  •  
    United States

Posted 15 November 2014 - 04:03 AM

On my tests on Windows XP SP3, offline registry > delete currentcontrolset\enum should not delete sw and root subkeys as they are needed for audio and other basic functionalities. Those keys are not completely rebuilt upon hardware detection.

 

Could someone confirm this?



#470 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15030 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 15 November 2014 - 12:02 PM

On my tests on Windows XP SP3, offline registry > delete currentcontrolset\enum should not delete sw and root subkeys as they are needed for audio and other basic functionalities. Those keys are not completely rebuilt upon hardware detection.

Could someone confirm this?

Can you explain?
WHY (the heck) would you actually want to delete all contents of the currentcontrolset\enum\ hive?
And WHY should a plain, simple, tool have "safeguards" against doing something that the user asked for?


:duff:
Wonko

#471 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2783 posts
  • Location:Nantes - France
  •  
    France

Posted 15 November 2014 - 12:43 PM

On my tests on Windows XP SP3, offline registry > delete currentcontrolset\enum should not delete sw and root subkeys as they are needed for audio and other basic functionalities. Those keys are not completely rebuilt upon hardware detection.

 

Could someone confirm this?

 

delete currentcontrolset\enum will delete all keys and subkeys.

my tests showed that xp would redetect it all at next reboot.

thus i did not test it extensively.

 

to be used with extreme care thus and only if you know what you are doing (testing, last chance action, etc).



#472 David Lynch

David Lynch

    Member

  • Members
  • 34 posts
  •  
    United States

Posted 16 November 2014 - 09:14 PM

Tested here and needed to restore SW and Root keys for audio to work.

 

I've tried to find a MS or Technet KB regarding this and I had no success. Many times a enum key rebuild is wanted; scenarios for that goes from infections to being unable to open device manager, devices erratic behaviour, etc.

 

It is a great feature but maybe it needs some changes. I'm not completely sure of that, so I'm just asking here.


Edited by David Lynch, 16 November 2014 - 09:20 PM.


#473 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2783 posts
  • Location:Nantes - France
  •  
    France

Posted 17 November 2014 - 07:12 PM

Tested here and needed to restore SW and Root keys for audio to work.

 

I've tried to find a MS or Technet KB regarding this and I had no success. Many times a enum key rebuild is wanted; scenarios for that goes from infections to being unable to open device manager, devices erratic behaviour, etc.

 

It is a great feature but maybe it needs some changes. I'm not completely sure of that, so I'm just asking here.

 

under root, i suspect that only the MEDIA key is needed for your audio.

and under sw, probably one subkey is needed for your audio.

 

still, why is not windows able to rebuild these i could not tell...

 

i usually delete the enum key when i have messed up my vm's and it did help me more than once.

never used it in real life thus.


  • David Lynch likes this

#474 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15030 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 17 November 2014 - 07:32 PM

ONLY to cheer you guys up :) a little bit, as often happens OT :ph34r:, but not much ;):

http://rwmj.wordpres...ks-technically/

 

:duff:

Wonko


  • David Lynch likes this

#475 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2783 posts
  • Location:Nantes - France
  •  
    France

Posted 18 November 2014 - 09:06 AM

Tested here and needed to restore SW and Root keys for audio to work.

 

I've tried to find a MS or Technet KB regarding this and I had no success. Many times a enum key rebuild is wanted; scenarios for that goes from infections to being unable to open device manager, devices erratic behaviour, etc.

 

It is a great feature but maybe it needs some changes. I'm not completely sure of that, so I'm just asking here.

 

If you are willing to perform tasks against an offline registry hive, I would recommend using OfflineReg .

 

There you could batch something like :

-save the root & sw key

-delete the enum key

-recreate it (empty)

-restore root & sw key

 

Regards,

Erwan


  • David Lynch likes this




2 user(s) are reading this topic

0 members, 2 guests, 0 anonymous users