WOF_Compress

Actually, calling deviceiocontrol + FSCTL_GET_EXTERNAL_BACKING OR using WofIsExternalFile function will actually achieve the same.

OutputBuffer [out]
A pointer to the output buffer, which must have a size large enough to receive a WOF_EXTERNAL_INFO structure followed by the provider data. 
For WIM backed files, WOF_EXTERNAL_INFO is followed by a WIM_PROVIDER_EXTERNAL_INFO structure. 
For individually compressed files, WOF_EXTERNAL_INFO is followed by a FILE_PROVIDER_EXTERNAL_INFO_V1 structure.

Currently I coded the autoit function assuming we would be looking at individually compressed files.

I could probably either code a new function to handle wim backed files or review the current function to detect what we are looking at and send back proper details.

 

For my tests, that means I need to create a wim file (using wimboot flag maybe?) and then mount it ?

 

EDIT : i checked and realised that WIM_PROVIDER_EXTERNAL_INFO and FILE_PROVIDER_EXTERNAL_INFO_V1 are the same stuctures.

 

So in theory, the current autoit function should work in both cases.

 

@Wimb : What error/behavior are you getting with wim backed files?

Thanks for the interesting info on WOF structures. 

In case of WimBoot Install then for mounted VHD the REAL Files and WIM-backed Files (WOF Pointers) give WOF Status ($iReS) WOF = 0

and the 5 values in the Array of the Structure have value = 0 (as given by DllStructGetData($outbuffer, 1,4)  - with 4 being 1 till 5 to get parameter 1 till 5)

which is not yet OK since the WimBoot WIM-Backed File (WOF Pointer) is reported as WOF=0 and  this report might be improved. 

In case of Compact Install then for mounted VHD the Compressed Files give WOF = 1 and Structure Array = 1, 2, 1, 1, 0  with LZX =1 on parameter 4

In case of UnCompressed files we get WOF = 0 and Structure Array = 0, 0, 0, 0, 0

The compact Install report is already OK.

WimBoot WIM-Backed WOF Pointer and WimBoot Real File = Compact LZX and UnCompressed file

WB WOF Pointer == WB Real File == Compact LZX == UnCompressed

==  ==  == 

Your proposal:

Review the current function to detect what we are looking at and send back proper details will be the best option.   

To study WIM-Backed files you can use VHD_WIMBOOT to Capture in WimBoot mode and Apply the captured WIM file in WimBoot mode to VHD file (Unmounted)

Then use WOF_Compress to study files in Mounted VHD. Files will be Real files when Excluded by WimBootCompress.ini and otherwise they are WOF Pointers.

Done some tests in delphi and it appears that you need a bigger buffer for wim backed files.

So the updated wof_status function below should now work for both compressed individual files AND wim backed files.

Now going for 64 * char (or 64 * bytes...or 16 * dword...).

Func _Wof_Status_($sFilePath)
	; we could/should create a more meaningful structure but hey, who cares :)
	Local $outbuffer=DllStructCreate("char[64]")
	Local $hFile, $IRes
	Local $poutbuffer=DllStructGetPtr($outbuffer)

	$hFile = _WinAPI_CreateFile($sFilePath,2, 2)
	$IReS = _WinAPI_DeviceIoControl($hFile, $FSCTL_GET_EXTERNAL_BACKING, 0, 0, $poutbuffer, 64)
	; the return code is what we care about : 1 means compressed, 0 means not compressed
	_WinAPI_CloseHandle($hFile)

	; MsgBox(64, "ok", $IReS)
	Return $IReS
EndFunc ;==>_Wof_Status_

 

Done some tests in delphi and it appears that you need a bigger buffer for wim backed files.

So the updated wof_status function below should now work for both compressed individual files AND wim backed files.

Now going for 64 * char (or 64 * bytes...or 16 * dword...).

Func _Wof_Status_($sFilePath)
	; we could/should create a more meaningful structure but hey, who cares :)
	Local $outbuffer=DllStructCreate("char[64]")
	Local $hFile, $IRes
	Local $poutbuffer=DllStructGetPtr($outbuffer)

	$hFile = _WinAPI_CreateFile($sFilePath,2, 2)
	$IReS = _WinAPI_DeviceIoControl($hFile, $FSCTL_GET_EXTERNAL_BACKING, 0, 0, $poutbuffer, 64)
	; the return code is what we care about : 1 means compressed, 0 means not compressed
	_WinAPI_CloseHandle($hFile)

	; MsgBox(64, "ok", $IReS)
	Return $IReS
EndFunc ;==>_Wof_Status_

The bigger buffer is working OK so that WIM-Backed files get WOF = 1 Status and is also OK for WOF Compressed files of Compact Install. 

Now we only need to find the right Structure to determine WOF Compression Algorithm for all cases.

The bigger buffer is working OK so that WIM-Backed files get WOF = 1 Status and is also OK for WOF Compressed files of Compact Install.

Now we only need to find the right Structures to determine WOF Compression Algorithm for all cases.

Easier actually :in wof_status2, update long[5] with long[16] when defining the buffer and 20 with 64 when calling deviceiocontrol and that should do the trick.
I will post an updated code tomorrow.

Probably it must be ulong[16] instead of long[16]

Do you have documentation about the meaning of the 16 parameters in the structure ?

I found out that the second parameter of the structure is interesting and the value is:

    Compact file = 2

    WIM-Backed file = 1

For WIM-Backed file then Parameter 7 - 11 have large value and for Compact and UnCompressed the value for these parameters is zero

Parameter 5 and 6 and 12-16 are always zero

Parameter 4 indicates Algorithm of Compact File (0=XPRESS4K,  1=LZX,  2=XPRESS8K,  3=XPRESS16K)

Parameter 4 for WIM-Backed file is always 0

For UnCompressed files all parameters are zero

Parameter 1 and 3 have value 1 for Compact and for WIM_Backed file and zero for UnCompressed file

WOF WIM-Backed == Compact LZX == UnCompressed

==  == 

Based on the below MSDN statement (here) :

OutputBuffer [out]
A pointer to the output buffer, which must have a size large enough to receive a WOF_EXTERNAL_INFO structure followed by the provider data. 
For WIM backed files, WOF_EXTERNAL_INFO is followed by a WIM_PROVIDER_EXTERNAL_INFO structure. 
For individually compressed files, WOF_EXTERNAL_INFO is followed by a FILE_PROVIDER_EXTERNAL_INFO_V1 structure. 

The returned buffer will awlays start with the below 8 bytes struct (here) :

typedef struct _WOF_EXTERNAL_INFO

{

ULONG Version;

ULONG Provider;

}

About the provider field above, please note possible values below

const WOF_PROVIDER_WIM =1;

followed for individual files by the below 12 bytes struct (here)

typedef struct _FILE_PROVIDER_EXTERNAL_INFO_V1

{

ULONG Version;

ULONG Algorithm;

ULONG Flags;

}

OR followed for wim backed files by the below 36 bytes struct (here)

typedef struct _WIM_PROVIDER_EXTERNAL_INFO

{

ULONG Version;

ULONG Flags;

LARGE_INTEGER DataSourceId;

UCHAR ResourceHash[20];

}

 

Based on the below MSDN statement (here) :

OutputBuffer [out]
A pointer to the output buffer, which must have a size large enough to receive a WOF_EXTERNAL_INFO structure followed by the provider data. 
For WIM backed files, WOF_EXTERNAL_INFO is followed by a WIM_PROVIDER_EXTERNAL_INFO structure. 
For individually compressed files, WOF_EXTERNAL_INFO is followed by a FILE_PROVIDER_EXTERNAL_INFO_V1 structure. 

The returned buffer will also start with the below struct (here) :

 

typedef struct _WOF_EXTERNAL_INFO

{

ULONG Version;

ULONG Provider;

}

 

 

About the provider field above, please note possible values below

const WOF_PROVIDER_WIM =1;

const WOF_PROVIDER_FILE = 2;

 

followed for individual files by the below struct (here)

 

typedef struct _FILE_PROVIDER_EXTERNAL_INFO_V1

{

ULONG Version;

ULONG Algorithm;

ULONG Flags;

}

 

OR followed for wim backed files by the below struct (here)

 

typedef struct _WIM_PROVIDER_EXTERNAL_INFO

{

ULONG Version;

ULONG Flags;

LARGE_INTEGER DataSourceId;

UCHAR ResourceHash[20];

}

 

Looking at the above structure, I am not sure if you can retrieve the compression algo for wim backed files.

 

Thanks for explaining the Structure. It corresponds to my observation.

In case of Compact file we can detect the Algorithm used, but in case of WIM-Backed file the Algorithm cannot be specified.

But we always can use WIM Info (Button in VHD_WIMBOOT) and see what is the Compression Type and see if WIM is marked as WimBoot compatible.

In any case a Structure with ulong[16] is working OK for our purpose, so that the Status of Compact and WIM-Backed and UnCompressed is found correctly 

and also the Compression Algoithm of Compact files can be determined.

I can Update to WOF_Compress-37 unless you have other ideas that must be implemented ....

...

I can Update to WOF_Compress-37 unless you have other ideas that must be implemented ....

I would say you are safe to publish a new version using a default 64 bytes buffer so that it handles both individual files and wim backed files.

Eventually in a new version you could add a button to distinguish between individual files and wim backed files.

Although you can already do it today : if wof_status returns 1 (it is compressed) and wof_status2 returns 0 (algo unknown), you then know that you are dealing with a wim backed files.

I would say you are safe to publish a new version using a default 64 bytes buffer so that it handles both individual files and wim backed files.

 

Eventually in a new version you could add a button to distinguish between individual files and wim backed files.

 

Although you can already do it today : if wof_status returns 1 (it is compressed) and wof_status2 returns 0 (algo unknown), you then know that you are dealing with a wim backed files.

Sure, but also outbuffer parameter 2 of the Structure in _Wof_Status2_ gives this info:

Return of _WinAPI_DeviceIoControl has value for Status  1=WOF  and 0=UnCompressed

In $outbuffer the 2nd ulong has value  2=Compact  and  1=WIM-Backed  and  0=UnCompressed

In $outbuffer the 4th ulong is Algorith for Compact files   0=XPRESS4K  1=LZX  2=XPRESS8K  3=XPRESS16K

Everything can be combined in Function _Wof_Status2_   

I will launch version 3.7 ....

Update WOF_Compress Version 3.7 is on line

Download from reboot.pro  - WOF_Compress  encrypted with password = bootwimb

Download from wimb GitHub - WOF_Compress

Update WOF_Compress Version 3.8 is on line

Download from reboot.pro  - WOF_Compress  encrypted with password = bootwimb

Download from wimb GitHub - WOF_Compress

==

 

Good work my friend, the Wof Status Button is a good addition to the program, just tested on an old 10x64.vhd and it is working very fine.

I read on WinNTSetup page about the new feature to avoid all *.sys installed drivers compression on new version 4.2.3, haven't tested it myself, but I saw you already tested it, and reported it is working fine, this is also a very good addition for Wimboot and Compact installs.

alacran

JFYI

From: http://reboot.pro/to...e-2#entry215932

 

I ran Compact /compactos:always  in a elevateted command prompt from a running 10x64 (standard install, not compact) on a VHD,  In my test got a compression of 1.8 to 1, I assume they were 4K compacted (standard Compact value on 10), see attached picture.

EDIT: Using new WOF_Compress-38, I was able to verify it was 8K compressed, then 8K seems to be the new stardard for Win10 Compact on this 10 19H1.

 

But all Portables under Documents folder remained uncompressed....

Just for testing I applied NTFS compression to Documents Folder, and when checking compression with WOF_Compress-38 it reports Compression 0 wich is not compressed. See attached pictures (made from Win7) please.

alacran

Just for testing I applied NTFS compression to Documents Folder, and when checking compression with WOF_Compress-38 it reports Compression 0 wich is not compressed. See attached pictures (made from Win7) please.

 

NTFS Compression is Not WOF Compression, so indeed it is OK that the WOF Status = 0 for NTFS Compressed files.

NTFS Compression is easily recognised by the Blue mark of files and folders.

WimGAPI Mode Compact:XPRESS4K results wrongly in XPRESS8K (X) Compression, whereas wimlib is giving XPRESS4K (x)

quote from MSFN WinNTSetup  - This has been fixed in WinNTSetup 4.2.3

- fixed compact:xpress4k using 8k compression with newer wimgapi.dll

Did you use Compact /CompactOS:always on normal Install Or may be on Compact:XPRESS4K Install of older version of WinNTSetup ?

The blue colored files and folders are only seen from Win7, from where I took the pictures to make it more evident, on Win10 each folder has two small blue arows on right upper corner only, no file has any other identification of NTFS compression.

I'm aware of the recent changes on WinNTSetup 4.2.3, but that do not apply on present case, in this case it was a normal (non Compact) install, and latter it was compressed using Compact /CompactOS:always ran from the running OS itself, then that's why I'm assuming Compact 4K wich used to be standard Compact on Win10, has being replaced on new Win10 versions with Compact 8K as the standard Compact.

alacran

The blue colored files and folders are only seen from Win7, from where I took the pictures to make it more evident, on Win10 each folder has two small blue arows on right upper corner only, no file has any other identification of NTFS compression.

 

I'm aware of the recent changes on WinNTSetup 4.2.3, but that do not apply on present case, in this case it was a normal (non Compact) install, and latter it was compressed using Compact /CompactOS:always ran from the running OS itself, then that's why I'm assuming Compact 4K wich used to be standard Compact on Win10, has being replaced on new Win10 versions with Compact 8K as the standard Compact.

 

Thanks for the detailed info.

I can confirm that indeed compact /CompactOS:always as applied to online recent Win10x64 results in XPRESS8K Compression of OS files

The blue colored files and folders are only seen from Win7, from where I took the pictures to make it more evident, on Win10 each folder has two small blue arows on right upper corner only, no file has any other identification of NTFS compression.

I have a statement from the Microsoft Windows Enhancement Team :

Thank you for appreciating the visual improvements made in the Windows 10, in order to better your experience.

Wonko

One more to the list of negative improvements.

Actually any thing disabled or removed, they call it a improvement, just to give 2 samples:

• System Restore disabled by default is a improvement
• Registry backups disabled is a improvement

alacran

The blue colored files and folders are only seen from Win7, from where I took the pictures to make it more evident, on Win10 each folder has two small blue arows on right upper corner only, no file has any other identification of NTFS compression.

 

In case of NTFS Compression and Windows 10 then Files also get the two small blue arrows,

but these arrows are not visible in the icons  of some .exe files like Dir_List_x64.exe and Hard_Link_List_x64.exe

In case of NTFS Compression and Windows 10 then Files also get the two small blue arrows,

but these arrows are not visible in the icons  of some .exe files like Dir_List_x64.exe and Hard_Link_List_x64.exe

 

NTFS_C_2020-08-23_162843.jpg

Maybe it is part of the path towards transparent compression...

Wonko

@alacran

those are improvements, in that we have not needed them since we were able to copy the same vhd's across all the mass storage devices we have --> if something goes wrong we overwrite the culprit vhd with one of its twins. much faster, to say the least. 

In case of NTFS Compression and Windows 10 then Files also get the two small blue arrows,

but these arrows are not visible in the icons  of some .exe files like Dir_List_x64.exe and Hard_Link_List_x64.exe

 

I saw your picture my friend, maybe those with no arrows are not compressible, the damn two blue arrows are so small maybe I didn't see them on Win10, I like better the blue colored files/folders, they are very easy to notice them.

@ antonino61

Yes if using a Compact or Wimboot VHD those are things we don't need/use.

But my complain is they inactivate/remove/modify some things wich can be useful for some people, and at the same time they add a bunch of unuseful (CR)Apps that ocupy a lot of space, and also use a special pagefile (only to keep them running in the background) wich name I don't remember right now, I think it is swapfile (like in Linux) or something like that.

And remember there are now also 7GB reserved just for the damn monolithic mega update packages, it was never required before, with separate updates.

My very last test is a 6 GB 10x64 2004 Compact LZX VHD, used space is 4.08 GB (including Excell and Word 2007 + some portables) and has 1.91 GB free, all this in 1 GB less that the new reserved space required for updates, see attached picture, (additionally we can reduce used space on this VHD about 1 GB using cdob's base_winsxs.cmd) .

To have an almost decent OS it is required a lot of extra work to remove all unwanted on the install.wim before install it and apply certain tricks during install and latter once booting just to reduce the damn telemetry too.

alacran

Just tested your reuploaded WOF_Compress v3.8 on a running WinPE.vhd, and selecting a folder located into the VHD (X:) to compress all worked very fine.

Thanks, It was good you removed the restriction to select folders/files on drive X:

alacran

Update WOF_Compress 3.9

Download from wimb GitHub - WOF_Compress - encrypted with password = bootwimb

I'll try this one too and let u know in a bit.