Jump to content











Photo
- - - - -

WOF_Compress

windows compression ntfs wof

  • Please log in to reply
155 replies to this topic

#26 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 12 May 2019 - 12:33 PM

"File" button is using direct API's whereas "drive" and "folder" buttons are using wofcompress.exe right?

In any case, sending a IOCTL to a file handle should not create a BSOD.



#27 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 12 May 2019 - 12:58 PM

@ wimb

 

Saw in your picture on post No. 21, when selecting the target I should see original size + actual size on disk + WOF= status (1 or 0).

 

I do know by previous experience on Win7 original size & actual size of a folder on disk remain the same when checking properties, so I will assume this also afects WOF status on this OS.

But also cheked this on Win10x86 1809 and also on this OS I don't get that info, see attached picture where first compressed a folder, and latter when re-selected it for compression again, I didn't get actual size on disk or any info about it was already compressed.

 

EDIT: The info on the program apply only to file, not to folder.

 

alacran

Attached Thumbnails

  • Already compressed.png


#28 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 12 May 2019 - 01:10 PM

I will answer myself: The mentioned info on previous post only apply to FILE, not to FOLDER.

 

But It would be good to have that info for folders too.

 

I'm going to make a note on previous post.



#29 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 12 May 2019 - 01:14 PM

"File" button is using direct API's whereas "drive" and "folder" buttons are using wofcompress.exe right?

In any case, sending a IOCTL to a file handle should not create a BSOD.

 

Indeed you are right. Only in case File selected then direct API's are used.

 

In Version 2.0 Drive and Folder are treated the same as in all previous versions by using WofCompress.exe

 

@alacran

Yes, you are right. The Size on Disk and WOF info is for File only.



#30 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 12 May 2019 - 01:23 PM

@wimb, in the coming days, may be we can work together on a recursive function for autoit?

 

Once you have such a function, you will not be relying on wofcompress.exe anymore for drive and folders.



#31 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 12 May 2019 - 02:15 PM

@wimb, in the coming days, may be we can work together on a recursive function for autoit?

 

Once you have such a function, you will not be relying on wofcompress.exe anymore for drive and folders.

 

Sure it is a good idea to use a recursive function so that we don't need WofCompress.exe anymore.

 

I have already something like that available in program UEFI_MULTI (have a look at the code in makebt\au3scr\sources_au3)

 

Func _FileSearch has $ExcludeFile list and is recursing through Content Folder  to make FileList

 

Func _CopyDirWithProgress is using that FileList for Copy with Display in Progressbar and Counting Files

 

That last function can be modified for WOF Compression of Files with Display in Progressbar and Counting Files

 

Coming week I can try to make this happen ....Or do you have another idea about it ?



#32 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 12 May 2019 - 02:26 PM

Excellent : better re use a function you have so that you can easily maintain/improve it afterwards.

 

For some unknown reason yet, wofcompress.exe creates BSOD under specific circumstances : might be a good idea to report it to JFX.



#33 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 12 May 2019 - 02:41 PM

Excellent : better re use a function you have so that you can easily maintain/improve it afterwards.

 

For some unknown reason yet, wofcompress.exe creates BSOD under specific circumstances : might be a good idea to report it to JFX.

 

Under what condition did you have BSOD with WofCompress.exe ? Then I can try to repeat here and report specific issue to JFX.

 

Also a good reason to go for the recursive function ....



#34 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 12 May 2019 - 02:44 PM

About the BSOD discussion, worth using this post as reference.

@Antonino61 mentionned a transformer T100 device (tablet?) under UEFI/GPT.

 

I myself never encountered a BSOD yet (using a rather standard desktop under with BIOS/MBR).



#35 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 12 May 2019 - 03:19 PM

The ASUS Transformer T100 has Intel Atom Processor probably with 32-bits uefi



#36 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 12 May 2019 - 03:26 PM

correct



#37 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 17 May 2019 - 05:15 AM

WOF_Compress Version 3.0 is on line - Download encrypted with password = bootwimb

 

WOF Compression and Uncompression of Files Or Drives and Folders using Status and Compress Functions made by erwan.l

GUI for WOF Compression and Uncompression of Drives and Folders using WofCompress Tool of JFX

 

== == ==

 

 

:cheers:

  • erwan.l likes this

#38 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 17 May 2019 - 08:24 AM

Very cool and hard work you have done there : that will definitely make this MS WOF feature popular.

 

As a side discussion, effects on the FS of wof compressing or wof uncompressing could be interesting to document.


  • wimb likes this

#39 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 17 May 2019 - 09:37 AM

Very cool and hard work you have done there : that will definitely make this MS WOF feature popular.

 

As a side discussion, effects on the FS of wof compressing or wof uncompressing could be interesting to document.

 

Thanks  :)   -  WOF_Compress is a nice and easy to use tool now  ;)

 

I used Defraggler - Portable to study the effects of WOF Compression / UnCompression.

 

WOF Compression does not give file fragmentation, but WOF UnCompression can give quite severe file fragmentation.

 

1 - DF1-2019-05-16_103135.png == 2 - DF2-2019-05-16_112132.png == 3 - DF3-2019-05-16_112408.png == 4 - DF4-2019-05-16_112444.png == 5 - DF5-2019-05-16_112644.png

 

1 = Original Drive M: with complete Windows Folder and file W7x64NL_W4.vhd

2 = After WOF Compression of Drive M:

3 = After UnCompression of file W7x64NL_W4.vhd

4 = View File W7x64NL_W4.vhd

5 = After Compression of file W7x64NL_W4.vhd

 

WOF Compression also changes the File DateTime which is a disadvantage.

 
And it also has consequences for the extents system information as described by erwan.l  - see this post.

Indeed, as the file goes to a "backing" place, the FS no longer stores the extents details



#40 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 17 May 2019 - 10:20 AM

I think the extents details are lost when compressing : this is expected.

So the "fragmentation" concept is probably not relevant then.

 

When uncompressing, the file is probably recreated from scratch on disk and the system will then use any available clusters thus leading to possible random fragmentation.

Question would be : will it more fragmented before/after? hard to tell.

Probably if you had taken care of your FS or files to be contiguous before, while uncompressing, the system will not take any particular care and you wil then experience more fragmentation after compared to before.



#41 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 17 May 2019 - 10:35 AM

When uncompressing, the file is probably recreated from scratch on disk and the system will then use any available clusters thus leading to possible random fragmentation.

Question would be : will it more fragmented before/after? hard to tell.

Probably if you had taken care of your FS or files to be contiguous before, while uncompressing, the system will not take any particular care and you wil then experience more fragmentation after compared to before.

 

Compression will give fragmented FREE SPACE that is used in creating the Uncompressed file as a NEW file

In this way the Uncompressed file is fragmented.

 

It is important to consider what is occurring with space that is made free by Compression and made free by Uncompression.



#42 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 3041 posts
  • Location:Nantes - France
  •  
    France

Posted 17 May 2019 - 11:45 AM

Compression will give fragmented FREE SPACE that is used in creating the Uncompressed file as a NEW file

In this way the Uncompressed file is fragmented.

 

It is important to consider what is occurring with space that is made free by Compression.

 

I will try to trace the clusters before/after compression.

 

-Are the clusters freed for good or marked in some way? Probably yes.

-And if freed for good, what happens when these are reused by the FS? The the FS will use other free clusters.

-What happens if there are no more free clusters? Uncompression probably fails.

-What happens if thare are enough contiguous free clusters? Ideally, the FS would create a non fragmented file but is there any logic for this in windows? Not so sure...



#43 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 17 May 2019 - 01:57 PM

sorry from here: the compression results in a used space as great as the total disk space. something must be wrong here. I will try enlarging the vhd and let u know.

nino



#44 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 17 May 2019 - 02:26 PM

@ wimb

 

Just downloaded and tested your v3.0, it is working very fine, so basically we have two options to use now (since v2.0) for Compress  and Uncompress:

 

1 - Status and Compress Functions from erwan.l  (The default option).

 

2 - WofCompress Tool from JFX, optional but with an editable exclusion list.

 

Is there an exclusion list for Status and Compress Functions?

 

I know when dealing with OS drive, in case of dubt and just to be on the safe side I can use WofCompress with the exclusion list extensively tested and improved during our test on Wimboot VHD and VHD and Wimboot source Compression topics

 

alacran



#45 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 17 May 2019 - 02:55 PM

 

Is there an exclusion list for Status and Compress Functions?

 

 

 

Yes, when used for WOF Compression of Drive Or Folder , then Files (specified by filename or part of filename) are excluded as given in file makebt\Compress_Exclude.txt

 

At the moment only file ntoskrnl.exe is excluded in this way,

 

which corresponds to the CompressionExclusionList in makebt\WimBootReCompress.ini used by the WofCompress.exe tool of JFX.



#46 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 17 May 2019 - 03:46 PM

Yes, when used for WOF Compression of Drive Or Folder , then Files (or patterns) are excluded as given in file makebt\Compress_Exclude.txt

 

 

I saw it before and I was thinking it is some kind of under test list, but wanted to be sure.

 

Then it is good to confirm Status and Compress Functions from erwan.l  also has an exclusion list we can modify to our needs/preferences or just exclude some kind of files we want, as an example *.png or *.jpg, wich are compressed now with almost no gain as I have tested, see attached pictures.

Of course it is only a matter of time to have an exclusion list, in accordance with each one of us preferences.

 

Then, at least for me, the strategy will be use WofCompress Tool from JFX when dealing with OS drives (whit its exclusion list as it is now).
 

And use Status and Compress Functions from erwan.l for anything else (with my edited Compress_Exclude.txt exclusion list).

 

alacran

Attached Thumbnails

  • png file.png
  • jpg file.png


#47 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 17 May 2019 - 04:04 PM

sorry again from here: even trying enlarging the vhd, for the past 2 hours the compress function has been stuck on file \windows\system32\mrt.exe, which means it still is. now I will try the same operation in file compression mode and let u know.

nino



#48 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 17 May 2019 - 04:21 PM

sorry people, I think I put a foot on my mouth in both cases above. I simply mistook the use-wof*.sys-compress box for the compress-all-files belonging to the previous versions of wofcompress and ran the software with the box unticked. now that I have realised it, I tried a third time with the box ticked and it worked like a charm. now my logical perplexity (just in case the perplexity I must have caused was not enough) is the following: is there a more valid compression alternative than the wof*.sys already provided?



#49 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 17 May 2019 - 04:47 PM

Suggestions to include in Compress_Exclude.txt:

 

Compression exclusions list used by wimlib-clc & Dism when making a *.wim or *.esd file

[CompressionExclusionList]
*.mp3
*.zip
*.cab
\WINDOWS\inf\*.pnf

I would add also:

*.rar
*.jpg
*.jpeg
*.jpe
*.jfif
*.gif
*.tif
*.tiff
*.png

And video formats with high compression as:

*.avi
*.divx
*.mpeg-4

*.swf

 

alacran



#50 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 17 May 2019 - 05:32 PM

@alacran

 

Thanks for testing and for suggestions on extra entries for the Compress_Exclude.txt

 

Also *.wim  and *.7z can be excluded (and more ....)

 

In most cases WOF Compression is not really needed and in that case it might be wiser to avoid WOF Compression.

 

You can also think of only compressing files that really give a lot of free space.

 

We might rather work in that case with a Compress_FileList.txt file that is also faster and more easily to process .....

 

EDIT:

I don't see the same exclusion list in wimlib-clc  :unsure:

Compression exclusions list used by wimlib-clc & Dism when making a *.wim or *.esd file







Also tagged with one or more of these keywords: windows, compression, ntfs, wof

2 user(s) are reading this topic

0 members, 2 guests, 0 anonymous users