WOF_Compress

shall I amend wimbootcompress.ini?

@ wimb

Thanks for running your tests, I think some good info was got from them.

Then for WofCompress Tool of JFX:

WimBootReCompress.ini content has to be same as in our modded WimBootCompress.ini

I always have used the previous concept and never had troubles..

About this from your previous post:

 

Exclusion of WOF Compression of drivers requires also to Exclude the Driverstore since the files refer by hardlink to the same data on disk

Thus it is needed to use in this case

\Windows\System32\drivers\*.sys
\Windows\System32\DriverStore\*
 

\Windows\System32\drivers\*.sys is already in our our modded WimBootCompress.ini

\Windows\System32\DriverStore\* is not included in our modded WimBootCompress.ini and so far the Winboot or Compact install systems boots OK.

Perhaps during boot stage only drivers on \Windows\System32\drivers\*.sys require to be uncompressed (there is located wof.sys or wofadk.sys and SvBus.sys too), because when booting on a different PC I have seen the systems installing drivers without problems just before show desktop screen.

I'm not trying to contradict your comment, my only intention is to save space as that folder is very big.

About this:

 

The WofCompress Tool of JFX works already as Trusted Installer and my program does not auto launch RunAsTI64.exe
 

Thanks for clarify this to me.

About your own program:

 

In case of a selected File or FileList there is no Exclusion, it is just the List that is processed in this case.

Risky in case the list include some critical file. Definitively only for advanced users. Maybe is not a bad idea to add some warning.

 

In case of a selected Path then the [CompressionExclusionList] section of makebt\Compress_Exclude.ini file is default used, but it is possible to select other Exclusion file.

Sounds good to me.

Sorry to bother you so much but you didn't commented about Compress_Exclude_Files.ini, is it only a sample list?

alacran

WimBootReCompress.ini was introduced by JFX for his WofCompress Tool to have an easy to modify Custom file outside the operating system.

However using the [PrepopulateList] Section as [CompressionExclusionList] creates confusion

since it is for Compression Exclusion working quite differently than for WimBoot Apply.

I would suggest to keep WimBootCompress.ini as it is now working OK and use that in WimBoot mode Apply to Prepopulate with real files instead of WOF pointers.

I would suggest to modify in WimBootReCompress.ini the [PrepopulateList] so that in WOF Compression it is working correctly as [CompressionExclusionList]

That means that lines with drivers and Catroot can be removed since they refer to Multi Hard Link data.

If two or more files are referring to the same data by hardlinks, 

for proper exclusion then all entries referring to that data should be on the Compression Exclusion List.

Then about my own program:

Compress_Exclude_Files.ini is indeed only a sample list, a simplified version of Compress_Exclude.ini

and that can be modified to your wishes and is selectable as alternative.

The Compress_FileList.txt file is at the moment empty, so there is no risk.

If you want to Compress Or UnCompress say 5 files on a selected drive

then you can put these files on the list and use that FileList instead of 5 times doing a single File Selection.

In the Program I wil make clear and give a WARNING that there is no exclusion active in this case.

so shall I change anything in my wimbootcompress.ini or any other *.ini?

WOF_Compress Version 3.4 Updated

- Compression Exclusion files Compress_Exclude.ini and WimBootReCompress.ini improved by removal of drivers and CatRoot lines 

  drivers and Catroot folder does not need to be excluded and cannot be excluded simply since the files in these folders refer to Multi Hard Link data

-  removed Compress_Exclude_Files.ini sample file that was not used anymore

- Download encrypted with password = bootwimb

Info:

WofCompress.exe Tool of JFX will default use file makebt\WimBootReCompress.ini as exclude file for compression using [PrepopulateList] section

I used my happiest wimbootcompress.ini here as well, as wimbootrecompress.ini, and I can tell u that it beats my lzx.bat.

I used my happiest wimbootcompress.ini here as well, as wimbootrecompress.ini, and I can tell u that it beats my lzx.bat.

Thank you and Good to hear about it  

@ wimb

Maybe this info can be useful for you: http://reboot.pro/to...ws/#entry215683

 

If we run Scan Now on Junction Link Magic it will show us all Pointerts in our OS, this could be useful in some cases. As to let us identify where are located the real files/folders and wich are only pointers.

alacran

I think this tool was developed with wimboot installs in mind.

But for a user who just wants to compress/uncompress files/folders, can this tool do the following :

I have a drive where I applied a windows wim using wimlib with --compact=lzx. So now all files are compressed.

I would like to uncompress some of the files/folders. If I make a ini file with the list of files/folders, can I use this tool ?

Also, in practice, what would be the difference between using the erwan.l functions vs the JFX exe ?

I think this tool was developed with wimboot installs in mind.

 

But for a user who just wants to compress/uncompress files/folders, can this tool do the following :

 

I have a drive where I applied a windows wim using wimlib with --compact=lzx. So now all files are compressed.

I would like to uncompress some of the files/folders. If I make a ini file with the list of files/folders, can I use this tool ?

 

Also, in practice, what would be the difference between using the erwan.l functions vs the JFX exe ?

WOF_Compress program is designed in general for WOF Compress / UnCompress of files / folders.

WofCompress.exe Tool of JFX needs a Path as parameter and it is not suitable for a single file or FileList.

For your purpose you can modify makebt\Compress_FileList.txt and use that in WOF_Compress program offline with a selected drive

Compress_FileList.txt might be something like

; This is an example FileList for WOF_Compress of files ocurring on selected Target Drive
; each valid entry should start with a backslash folowed by full path and filename with extension
; a leading ; can used for lines with comment
\Boot\BCD
\bootmgr
\Windows\System32\config\system
\Windows\System32\drivers\somedriver.sys

- WOF_Compress needs to be Trusted Installer in case of Path with Windows Operating System folders

==  == 

Thanks. Wish the program did not exit every time I compressed/uncompressed a file :-)

Just to confirm, this program only displays WOF=1 or WOF=0 for a file, it does not show the type of compression, right.

Thanks. Wish the program did not exit every time I compressed/uncompressed a file :-)

 

Just to confirm, this program only displays WOF=1 or WOF=0 for a file, it does not show the type of compression, right.

Currently, the functions I wrote only reports true/false (compressed).

Another function could be added report the compression Algorithm.

Cheers,

Erwan

Currently, the functions I wrote only reports true/false (compressed).

Another function could be added report the compression Algorithm.

 

It will be very nice if you can make AutoIt Function that can report Compression Algorithm 

It will be very nice if you can make AutoIt Function that can report Compression Algorithm 

I second that.

WOF_Compress Version 3.5 is on line - Download encrypted with password = bootwimb

Download from wimb GitHub - WOF_Compress

- Added AdvancedRun of NirSoft to RunAs Trusted Installer by using WOF_Compress_Trusted.cmd

- Added DirList - allows to select a folder so that Folder_List and File_List of that Path are created

- Added NTFSLinksView of NirSoft to get reports on Junctions, Hard Links and Symbolic Links

- WOF_Compress_x64.exe does not auto close anymore so that it is ready for the Next Run

- Log Files of the Compress / UnCompres process are created in folder processed

It will be very nice if you can make AutoIt Function that can report Compression Algorithm 

I should be able to deliver that this week end.

I should be able to deliver that this week end.

Actually, it might be necessary. I am developing a simple GUI to compress/decompress files using WOF. I just found out that if a file is already compressed, you can compress it again - e.g. if the file was compressed with, say, XPRESS4K it is possible to (re)compress it with LZX.

If the user wanted to compress a file, I was testing if the file was already compressed, if so I would skip compressing it. I assumed that if a file was compressed you could not compress it again.

So what I actually have to do is check existing compression type and then recompress only if the compression type is being changed.

I should be able to deliver that this week end.

Good 

I found some useful information on wofapi.h and WOF_FILE_COMPRESSION_INFO_V1 structure here

Here comes an autoit function that will send back the compression algo.

Will return a value from 0 to 3.

Func _Wof_Status2_($sFilePath)
	; we could/should create a more meaningful structure but hey, who cares :)
	Local $outbuffer=DllStructCreate("align 4;long[5]")
	; Local $outbuffer=DllStructCreate("byte[20]")
	Local $hFile, $IRes
	Local $poutbuffer=DllStructGetPtr($outbuffer)

	$hFile = _WinAPI_CreateFile($sFilePath,2, 2)
	$IReS = _WinAPI_DeviceIoControl($hFile, $FSCTL_GET_EXTERNAL_BACKING, 0, 0, $poutbuffer, 20)
	; the return code is  : 1 means compressed, 0 means not compressed
	_WinAPI_CloseHandle($hFile)
        ; 4th dword out of 5 (our struct) is CompressionFormat
	Return DllStructGetData($outbuffer, 1,4)
EndFunc ;==>_Wof_Status2_

 

Here comes an autoit function that will send back the compression algo.

 

Will return a value from 0 to 3.

 

FILE_PROVIDER_COMPRESSION_FORMAT_XPRESS4K = 0;

FILE_PROVIDER_COMPRESSION_FORMAT_LZX = 1;

FILE_PROVIDER_COMPRESSION_FORMAT_XPRESS8K = 2;

FILE_PROVIDER_COMPRESSION_FORMAT_XPRESS16K = 3;

Func _Wof_Status2_($sFilePath)
	; we could/should create a more meaningful structure but hey, who cares :)
	Local $outbuffer=DllStructCreate("align 4;long[5]")
	; Local $outbuffer=DllStructCreate("byte[20]")
	Local $hFile, $IRes
	Local $poutbuffer=DllStructGetPtr($outbuffer)

	$hFile = _WinAPI_CreateFile($sFilePath,2, 2)
	$IReS = _WinAPI_DeviceIoControl($hFile, $FSCTL_GET_EXTERNAL_BACKING, 0, 0, $poutbuffer, 20)
	; the return code is  : 1 means compressed, 0 means not compressed
	_WinAPI_CloseHandle($hFile)
        ; 4th dword out of 5 (our struct) is CompressionFormat
	Return DllStructGetData($outbuffer, 1,4)
EndFunc ;==>_Wof_Status2_

Thanks a lot Erwan for creating autoit3 Function _Wof_Status2_ to report WOF Status and Compression Algorithm 

It works quite well and the Function _Wof_Status2_ is applied in WOF_Compress Version 3.6

Update WOF_Compress Version 3.6 is on line

Download from reboot.pro encrypted with password = bootwimb - WOF_Compress

Download from wimb GitHub - WOF_Compress

The AutoIt3 Function _Wof_Status2_ is working great to report WOF Status and Compression Algorithm in case of Normal and Compact Windows Installs.

In case of WimBoot Install it should be possible somehow to detect if a file is WIM Backed and present as WOF Pointer or present as Real File.

Where and how to detect this ? The WOF Status function reports WimBoot files always as Normal files having WOF = 0 Status

More Info may be here:

WofEnumFilesProc callback function  and  WofIsExternalFile function  and  WOF_FILE_COMPRESSION_INFO_V1 structure

The AutoIt3 Function _Wof_Status2_ is working great to report WOF Status and Compression Algorithm in case of Normal and Compact Windows Installs.

 

In case of WimBoot Install it should be possible somehow to detect if a file is WIM Backed and present as WOF Pointer or present as Real File.

Where and how to detect this ? The WOF Status function reports WimBoot files always as Normal files having WOF = 0 Status

 

More Info may be here:

 

WofEnumFilesProc callback function  and  WofIsExternalFile function  and  WOF_FILE_COMPRESSION_INFO_V1 structure

Actually, calling deviceiocontrol + FSCTL_GET_EXTERNAL_BACKING OR using WofIsExternalFile function will actually achieve the same.

OutputBuffer [out]
A pointer to the output buffer, which must have a size large enough to receive a WOF_EXTERNAL_INFO structure followed by the provider data. 
For WIM backed files, WOF_EXTERNAL_INFO is followed by a WIM_PROVIDER_EXTERNAL_INFO structure. 
For individually compressed files, WOF_EXTERNAL_INFO is followed by a FILE_PROVIDER_EXTERNAL_INFO_V1 structure.

Currently I coded the autoit function assuming we would be looking at individually compressed files.

I could probably either code a new function to handle wim backed files or review the current function to detect what we are looking at and send back proper details.

For my tests, that means I need to create a wim file (using wimboot flag maybe?) and then mount it ?

EDIT : i checked and realised that WIM_PROVIDER_EXTERNAL_INFO and FILE_PROVIDER_EXTERNAL_INFO_V1 are the same stuctures.

So in theory, the current autoit function should work in both cases.

@Wimb : What error/behavior are you getting with wim backed files?

Is regular NTFS compression the same WOF XPRESS4K compression ?

I just tested a file with regular NTFS compression, and _Wof_Status2_ it reported it as XPRESS4K.

The file sizes of a file with with regular compression and XPRESS4K are the same.

Is regular NTFS compression the same WOF XPRESS4K compression ?
 
I just tested a file with regular NTFS compression, and _Wof_Status2_ it reported it as XPRESS4K.
 
The file sizes of a file with with regular compression and XPRESS4K are the same.

@ gbrao

Just to let you compare different compressions on Windows, I ran a test on a folder with a size on disk of 147 MB.

Initial folder 		Size on Disk = 147 MB 100 %

NTFS Compression	Size on Disk = 103 MB aprox 45 sec. 70,07 % 	Saved 29.93 %

Express 4K		Size on Disk = 95.9 MB 9 sec. 65,24 % 		Saved 34.76 %

Xpress 8K		Size on Disk = 91.7 MB 8 sec. 62,38 % 		Saved 37.62 %

Xpress 16 K		Size on Disk = 89.4 MB 8 sec. 60,81 %		Saved 39.19 % 

LZX			Size on Disk = 81.6 MB 26 sec. 55,51 % 		Saved 44.49 %

As we can see the old NTFS Compression (LZ77) is very slow, (it uses only 1 CPU thread) and creates an incredible fragmentation of files and free space.

Even with the lowest Xpress 4K compression final used size is better. And in 1/5 or 20 % compression time.

Even the highest LZX compression is faster almost 1/2 or 50 % compression time.

The optimal ratio (lower is better) for used time/% reduced is on Xpress 16 K.

But Xpress 8 K ratio is very close to 16 K and it requires less powerfull CPU for compressing, and also to latter decompress on the fly to load on Ram when reading each file.

Of course the time always highly depends on CPU, RAM and HD speeds.

alacran