Jump to content











Photo
- - - - -

WOF_Compress

ntfs windows compression wof

  • Please log in to reply
134 replies to this topic

#101 antonino61

antonino61

    Silver Member

  • Advanced user
  • 720 posts
  •  
    Italy

Posted 2 weeks ago

shall I amend wimbootcompress.ini?



#102 alacran

alacran

    Gold Member

  • .script developer
  • 1446 posts
  •  
    Mexico

Posted 2 weeks ago

@ wimb

 

Thanks for running your tests, I think some good info was got from them.

 

Then for WofCompress Tool of JFX:

 

WimBootReCompress.ini content has to be same as in our modded WimBootCompress.ini

 

I always have used the previous concept and never had troubles..

 

Spoiler

 

About this from your previous post:

 

 

Exclusion of WOF Compression of drivers requires also to Exclude the Driverstore since the files refer by hardlink to the same data on disk

Thus it is needed to use in this case

\Windows\System32\drivers\*.sys
\Windows\System32\DriverStore\*
 

 

\Windows\System32\drivers\*.sys is already in our our modded WimBootCompress.ini

 

\Windows\System32\DriverStore\* is not included in our modded WimBootCompress.ini and so far the Winboot or Compact install systems boots OK.

Perhaps during boot stage only drivers on \Windows\System32\drivers\*.sys require to be uncompressed (there is located wof.sys or wofadk.sys and SvBus.sys too), because when booting on a different PC I have seen the systems installing drivers without problems just before show desktop screen.

 

I'm not trying to contradict your comment, my only intention is to save space as that folder is very big.

 

About this:

 

The WofCompress Tool of JFX works already as Trusted Installer and my program does not auto launch RunAsTI64.exe
 

 

Thanks for clarify this to me.

 

About your own program:

 

 

In case of a selected File or FileList there is no Exclusion, it is just the List that is processed in this case.

 

Risky in case the list include some critical file. Definitively only for advanced users. Maybe is not a bad idea to add some warning.

 

 

In case of a selected Path then the [CompressionExclusionList] section of makebt\Compress_Exclude.ini file is default used, but it is possible to select other Exclusion file.

 

Sounds good to me.

 

Sorry to bother you so much but you didn't commented about Compress_Exclude_Files.ini, is it only a sample list?

 

alacran



#103 wimb

wimb

    Platinum Member

  • Developer
  • 2918 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 2 weeks ago

WimBootReCompress.ini was introduced by JFX for his WofCompress Tool to have an easy to modify Custom file outside the operating system.

 

However using the [PrepopulateList] Section as [CompressionExclusionList] creates confusion

since it is for Compression Exclusion working quite differently than for WimBoot Apply.

 

I would suggest to keep WimBootCompress.ini as it is now working OK and use that in WimBoot mode Apply to Prepopulate with real files instead of WOF pointers.

 

I would suggest to modify in WimBootReCompress.ini the [PrepopulateList] so that in WOF Compression it is working correctly as [CompressionExclusionList]

That means that lines with drivers and Catroot can be removed since they refer to Multi Hard Link data.

If two or more files are referring to the same data by hardlinks, 

for proper exclusion then all entries referring to that data should be on the Compression Exclusion List.

 

Then about my own program:

Compress_Exclude_Files.ini is indeed only a sample list, a simplified version of Compress_Exclude.ini

and that can be modified to your wishes and is selectable as alternative.

 

The Compress_FileList.txt file is at the moment empty, so there is no risk.

If you want to Compress Or UnCompress say 5 files on a selected drive

then you can put these files on the list and use that FileList instead of 5 times doing a single File Selection.

In the Program I wil make clear and give a WARNING that there is no exclusion active in this case.



#104 antonino61

antonino61

    Silver Member

  • Advanced user
  • 720 posts
  •  
    Italy

Posted 2 weeks ago

so shall I change anything in my wimbootcompress.ini or any other *.ini?



#105 wimb

wimb

    Platinum Member

  • Developer
  • 2918 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted A week ago

WOF_Compress Version 3.4 Updated

 

- Compression Exclusion files Compress_Exclude.ini and WimBootReCompress.ini improved by removal of drivers and CatRoot lines 

  drivers and Catroot folder does not need to be excluded and cannot be excluded simply since the files in these folders refer to Multi Hard Link data

-  removed Compress_Exclude_Files.ini sample file that was not used anymore

 

Download encrypted with password = bootwimb

 

Info:

WOF_Compress Program will default use file makebt\Compress_Exclude.ini as exclude file for compression using [CompressionExclusionList] section

WofCompress.exe Tool of JFX will default use file makebt\WimBootReCompress.ini as exclude file for compression using [PrepopulateList] section

 
VHD_WIMBOOT wimlib Apply in Compact 4K and Compact LZX uses the WIM internal \Windows\System32\WimBootCompress.ini file [PrepopulateList] section as Compression Exclusion file
That means the files specified by that Section will be Applied Normal instead of Compressed  (property of wimlib Apply).
 

 

--compact=FORMAT

Windows-only: compress the extracted files using System Compression, when possible. This only works on either Windows 10 or later, or on an older Windows to which Microsoft’s wofadk.sys driver has been added. Several different compression formats may be used with System Compression, and one must be specified as FORMAT. The choices are: xpress4k, xpress8k, xpress16k, and lzx.

 

Exclusions are handled in the same way as with the --wimboot option. That is: if it exists, the [PrepopulateList] section of the file \Windows\System32\WimBootCompress.ini in the WIM image will be read, and files matching any of the patterns in this section will not be compressed. In addition, wimlib has a hardcoded list of files for which it knows, for compatibility with the Windows bootloader, to override the requested compression format.

 


  • alacran and antonino61 like this

#106 antonino61

antonino61

    Silver Member

  • Advanced user
  • 720 posts
  •  
    Italy

Posted A week ago

I used my happiest wimbootcompress.ini here as well, as wimbootrecompress.ini, and I can tell u that it beats my lzx.bat.



#107 wimb

wimb

    Platinum Member

  • Developer
  • 2918 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted A week ago

I used my happiest wimbootcompress.ini here as well, as wimbootrecompress.ini, and I can tell u that it beats my lzx.bat.

 

Thank you and Good to hear about it   :)


  • antonino61 likes this

#108 alacran

alacran

    Gold Member

  • .script developer
  • 1446 posts
  •  
    Mexico

Posted A week ago

@ wimb

 

Maybe this info can be useful for you: http://reboot.pro/to...ws/#entry215683

 

 

If we run Scan Now on Junction Link Magic it will show us all Pointerts in our OS, this could be useful in some cases. As to let us identify where are located the real files/folders and wich are only pointers.

 

alacran



#109 gbrao

gbrao

    Frequent Member

  • Advanced user
  • 391 posts
  •  
    India

Posted A week ago

I think this tool was developed with wimboot installs in mind.

 

But for a user who just wants to compress/uncompress files/folders, can this tool do the following :

 

I have a drive where I applied a windows wim using wimlib with --compact=lzx. So now all files are compressed.

I would like to uncompress some of the files/folders. If I make a ini file with the list of files/folders, can I use this tool ?

 

Also, in practice, what would be the difference between using the erwan.l functions vs the JFX exe ?



#110 wimb

wimb

    Platinum Member

  • Developer
  • 2918 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted A week ago

I think this tool was developed with wimboot installs in mind.

 

But for a user who just wants to compress/uncompress files/folders, can this tool do the following :

 

I have a drive where I applied a windows wim using wimlib with --compact=lzx. So now all files are compressed.

I would like to uncompress some of the files/folders. If I make a ini file with the list of files/folders, can I use this tool ?

 

Also, in practice, what would be the difference between using the erwan.l functions vs the JFX exe ?

 

WOF_Compress program is designed in general for WOF Compress / UnCompress of files / folders.

WofCompress.exe Tool of JFX needs a Path as parameter and it is not suitable for a single file or FileList.

 

For your purpose you can modify makebt\Compress_FileList.txt and use that in WOF_Compress program offline with a selected drive

 

Compress_FileList.txt might be something like

; This is an example FileList for WOF_Compress of files ocurring on selected Target Drive
; each valid entry should start with a backslash folowed by full path and filename with extension
; a leading ; can used for lines with comment
\Boot\BCD
\bootmgr
\Windows\System32\config\system
\Windows\System32\drivers\somedriver.sys

- WOF_Compress needs to be Trusted Installer in case of Path with Windows Operating System folders

- Run RunAsTI64.exe first and then launch WOF_Compress_x64.exe from Command Window - See WOF_Compr_2020-07-27_200739.jpg
  In this case WARNING - Location systemprofile\Desktop missing occurs that you simply must close and can proceed
- Select the Checkbox Use FileList for Drive
- Select Drive 
- Select UnCompress to process files of Compress_FileList.txt on selected Drive - Note that Exclusion with Compress_Exclude.ini is not taken into account
 
Instead of Drive and Compress_FileList.txt you can also Select a Single File and See the WOF Status and use Compress / UnCompress button
 
WOF_FileList_2020-08-03_071622.jpg == WOF_Status_2020-08-03_072615.jpg == WOF_Status_2020-08-03_072438.jpg

  • gbrao likes this

#111 gbrao

gbrao

    Frequent Member

  • Advanced user
  • 391 posts
  •  
    India

Posted A week ago

Thanks. Wish the program did not exit every time I compressed/uncompressed a file :-)

 

Just to confirm, this program only displays WOF=1 or WOF=0 for a file, it does not show the type of compression, right.



#112 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2877 posts
  • Location:Nantes - France
  •  
    France

Posted A week ago

*
POPULAR

Thanks. Wish the program did not exit every time I compressed/uncompressed a file :-)

 

Just to confirm, this program only displays WOF=1 or WOF=0 for a file, it does not show the type of compression, right.

 

Currently, the functions I wrote only reports true/false (compressed).

Another function could be added report the compression Algorithm.

 

Cheers,

Erwan


  • wimb, gbrao and alacran like this

#113 wimb

wimb

    Platinum Member

  • Developer
  • 2918 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted A week ago

Currently, the functions I wrote only reports true/false (compressed).

Another function could be added report the compression Algorithm.

 

 

It will be very nice if you can make AutoIt Function that can report Compression Algorithm  :)



#114 gbrao

gbrao

    Frequent Member

  • Advanced user
  • 391 posts
  •  
    India

Posted A week ago

It will be very nice if you can make AutoIt Function that can report Compression Algorithm  :)

I second that.



#115 wimb

wimb

    Platinum Member

  • Developer
  • 2918 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted A week ago

WOF_Compress Version 3.5 is on line - Download encrypted with password = bootwimb

 

Download from wimb GitHub - WOF_Compress

 

- Added AdvancedRun of NirSoft to RunAs Trusted Installer by using WOF_Compress_Trusted.cmd

- Added DirList - allows to select a folder so that Folder_List and File_List of that Path are created

- Added NTFSLinksView of NirSoft to get reports on Junctions, Hard Links and Symbolic Links

- WOF_Compress_x64.exe does not auto close anymore so that it is ready for the Next Run

- Log Files of the Compress / UnCompres process are created in folder processed

 

WOF_Compress_x64.exe needs to be Trusted Installer in case of Path with Windows Operating System folders
 
Double-click WOF_Compress_Trusted.cmd so that AdvancedRun.exe is used to Run as Trusted Installer program WOF_Compress_x64.exe
 
Thanks to Nir Sofer of NirSoft to make AdvancedRun and NTFSLinksView
 
Thanks to alacran and Wonko the Sane  to give info on NTFSLinksView and thanks to gbrao for info on AdvancedRun
 
WOF_35_2020-08-05_102621.jpg


#116 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2877 posts
  • Location:Nantes - France
  •  
    France

Posted A week ago

It will be very nice if you can make AutoIt Function that can report Compression Algorithm  :)

 

I should be able to deliver that this week end.


  • wimb and gbrao like this

#117 gbrao

gbrao

    Frequent Member

  • Advanced user
  • 391 posts
  •  
    India

Posted A week ago

I should be able to deliver that this week end.

Actually, it might be necessary. I am developing a simple GUI to compress/decompress files using WOF. I just found out that if a file is already compressed, you can compress it again - e.g. if the file was compressed with, say, XPRESS4K it is possible to (re)compress it with LZX.

 

If the user wanted to compress a file, I was testing if the file was already compressed, if so I would skip compressing it. I assumed that if a file was compressed you could not compress it again.

So what I actually have to do is check existing compression type and then recompress only if the compression type is being changed.



#118 wimb

wimb

    Platinum Member

  • Developer
  • 2918 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted A week ago

I should be able to deliver that this week end.

 

Good  :)

 

I found some useful information on wofapi.h and WOF_FILE_COMPRESSION_INFO_V1 structure here



#119 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2877 posts
  • Location:Nantes - France
  •  
    France

Posted 6 days ago

Here comes an autoit function that will send back the compression algo.

 

Will return a value from 0 to 3.

 

FILE_PROVIDER_COMPRESSION_FORMAT_XPRESS4K = 0;
FILE_PROVIDER_COMPRESSION_FORMAT_LZX = 1;
FILE_PROVIDER_COMPRESSION_FORMAT_XPRESS8K = 2;
FILE_PROVIDER_COMPRESSION_FORMAT_XPRESS16K = 3;
Func _Wof_Status2_($sFilePath)
	; we could/should create a more meaningful structure but hey, who cares :)
	Local $outbuffer=DllStructCreate("align 4;long[5]")
	; Local $outbuffer=DllStructCreate("byte[20]")
	Local $hFile, $IRes
	Local $poutbuffer=DllStructGetPtr($outbuffer)

	$hFile = _WinAPI_CreateFile($sFilePath,2, 2)
	$IReS = _WinAPI_DeviceIoControl($hFile, $FSCTL_GET_EXTERNAL_BACKING, 0, 0, $poutbuffer, 20)
	; the return code is  : 1 means compressed, 0 means not compressed
	_WinAPI_CloseHandle($hFile)
        ; 4th dword out of 5 (our struct) is CompressionFormat
	Return DllStructGetData($outbuffer, 1,4)
EndFunc ;==>_Wof_Status2_

  • wimb and gbrao like this

#120 wimb

wimb

    Platinum Member

  • Developer
  • 2918 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 6 days ago

 

Here comes an autoit function that will send back the compression algo.

 

Will return a value from 0 to 3.

 

FILE_PROVIDER_COMPRESSION_FORMAT_XPRESS4K = 0;
FILE_PROVIDER_COMPRESSION_FORMAT_LZX = 1;
FILE_PROVIDER_COMPRESSION_FORMAT_XPRESS8K = 2;
FILE_PROVIDER_COMPRESSION_FORMAT_XPRESS16K = 3;
Func _Wof_Status2_($sFilePath)
	; we could/should create a more meaningful structure but hey, who cares :)
	Local $outbuffer=DllStructCreate("align 4;long[5]")
	; Local $outbuffer=DllStructCreate("byte[20]")
	Local $hFile, $IRes
	Local $poutbuffer=DllStructGetPtr($outbuffer)

	$hFile = _WinAPI_CreateFile($sFilePath,2, 2)
	$IReS = _WinAPI_DeviceIoControl($hFile, $FSCTL_GET_EXTERNAL_BACKING, 0, 0, $poutbuffer, 20)
	; the return code is  : 1 means compressed, 0 means not compressed
	_WinAPI_CloseHandle($hFile)
        ; 4th dword out of 5 (our struct) is CompressionFormat
	Return DllStructGetData($outbuffer, 1,4)
EndFunc ;==>_Wof_Status2_

 

Thanks a lot Erwan for creating autoit3 Function _Wof_Status2_ to report WOF Status and Compression Algorithm  :)

 

It works quite well and the Function _Wof_Status2_ is applied in WOF_Compress Version 3.6

 

Update WOF_Compress Version 3.6 is on line

 

Download from reboot.pro encrypted with password = bootwimb - WOF_Compress

 

Download from wimb GitHub - WOF_Compress

 

WOF_Compress_x64.exe needs to be Trusted Installer in case of Path with Windows Operating System folders
 
Double-click WOF_Compress_Trusted.cmd so that AdvancedRun.exe is used to Run as Trusted Installer program WOF_Compress_x64.exe


#121 wimb

wimb

    Platinum Member

  • Developer
  • 2918 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 5 days ago

The AutoIt3 Function _Wof_Status2_ is working great to report WOF Status and Compression Algorithm in case of Normal and Compact Windows Installs.

 

In case of WimBoot Install it should be possible somehow to detect if a file is WIM Backed and present as WOF Pointer or present as Real File.

Where and how to detect this ? The WOF Status function reports WimBoot files always as Normal files having WOF = 0 Status

 

More Info may be here:

 

WofEnumFilesProc callback function  and  WofIsExternalFile function  and  WOF_FILE_COMPRESSION_INFO_V1 structure



#122 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2877 posts
  • Location:Nantes - France
  •  
    France

Posted 5 days ago

The AutoIt3 Function _Wof_Status2_ is working great to report WOF Status and Compression Algorithm in case of Normal and Compact Windows Installs.

 

In case of WimBoot Install it should be possible somehow to detect if a file is WIM Backed and present as WOF Pointer or present as Real File.

Where and how to detect this ? The WOF Status function reports WimBoot files always as Normal files having WOF = 0 Status

 

More Info may be here:

 

WofEnumFilesProc callback function  and  WofIsExternalFile function  and  WOF_FILE_COMPRESSION_INFO_V1 structure

 

Actually, calling deviceiocontrol + FSCTL_GET_EXTERNAL_BACKING OR using WofIsExternalFile function will actually achieve the same.

OutputBuffer [out]
A pointer to the output buffer, which must have a size large enough to receive a WOF_EXTERNAL_INFO structure followed by the provider data. 
For WIM backed files, WOF_EXTERNAL_INFO is followed by a WIM_PROVIDER_EXTERNAL_INFO structure. 
For individually compressed files, WOF_EXTERNAL_INFO is followed by a FILE_PROVIDER_EXTERNAL_INFO_V1 structure.

Currently I coded the autoit function assuming we would be looking at individually compressed files.

I could probably either code a new function to handle wim backed files or review the current function to detect what we are looking at and send back proper details.

 

For my tests, that means I need to create a wim file (using wimboot flag maybe?) and then mount it ?

 

EDIT : i checked and realised that WIM_PROVIDER_EXTERNAL_INFO and FILE_PROVIDER_EXTERNAL_INFO_V1 are the same stuctures.

 

So in theory, the current autoit function should work in both cases.

 

@Wimb : What error/behavior are you getting with wim backed files?


  • wimb likes this

#123 gbrao

gbrao

    Frequent Member

  • Advanced user
  • 391 posts
  •  
    India

Posted 5 days ago

Is regular NTFS compression the same WOF XPRESS4K compression ?

 

I just tested a file with regular NTFS compression, and _Wof_Status2_ it reported it as XPRESS4K.

 

The file sizes of a file with with regular compression and XPRESS4K are the same.



#124 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2877 posts
  • Location:Nantes - France
  •  
    France

Posted 5 days ago

Is regular NTFS compression the same WOF XPRESS4K compression ?
 
I just tested a file with regular NTFS compression, and _Wof_Status2_ it reported it as XPRESS4K.
 
The file sizes of a file with with regular compression and XPRESS4K are the same.


Before win10 it was a pure Ntfs process using a lz77 algorithm.
With win10, the os ceates an ads called WofCompressedData using xpress or lzx, un the background to compress files which are seldom used.

#125 alacran

alacran

    Gold Member

  • .script developer
  • 1446 posts
  •  
    Mexico

Posted 4 days ago

@ gbrao

 

Just to let you compare different compressions on Windows, I ran a test on a folder with a size on disk of 147 MB.

Initial folder 		Size on Disk = 147 MB 100 %

NTFS Compression	Size on Disk = 103 MB aprox 45 sec. 70,07 % 	Saved 29.93 %

Express 4K		Size on Disk = 95.9 MB 9 sec. 65,24 % 		Saved 34.76 %

Xpress 8K		Size on Disk = 91.7 MB 8 sec. 62,38 % 		Saved 37.62 %

Xpress 16 K		Size on Disk = 89.4 MB 8 sec. 60,81 %		Saved 39.19 % 

LZX			Size on Disk = 81.6 MB 26 sec. 55,51 % 		Saved 44.49 %

As we can see the old NTFS Compression (LZ77) is very slow, (it uses only 1 CPU thread) and creates an incredible fragmentation of files and free space.

 

Even with the lowest Xpress 4K compression final used size is better. And in 1/5 or 20 % compression time.

 

Even the highest LZX compression is faster almost 1/2 or 50 % compression time.

 

The optimal ratio (lower is better) for used time/% reduced is on Xpress 16 K.

 

But Xpress 8 K ratio is very close to 16 K and it requires less powerfull CPU for compressing, and also to latter decompress on the fly to load on Ram when reading each file.

 

Of course the time always highly depends on CPU, RAM and HD speeds.

 

alacran


  • gbrao likes this





Also tagged with one or more of these keywords: ntfs, windows, compression, wof

2 user(s) are reading this topic

0 members, 2 guests, 0 anonymous users