Jump to content











Photo
- - - - -

WOF_Compress

windows ntfs compression wof

  • Please log in to reply
146 replies to this topic

#126 wimb

wimb

    Platinum Member

  • Developer
  • 2966 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 10 August 2020 - 06:38 AM

Actually, calling deviceiocontrol + FSCTL_GET_EXTERNAL_BACKING OR using WofIsExternalFile function will actually achieve the same.

OutputBuffer [out]
A pointer to the output buffer, which must have a size large enough to receive a WOF_EXTERNAL_INFO structure followed by the provider data. 
For WIM backed files, WOF_EXTERNAL_INFO is followed by a WIM_PROVIDER_EXTERNAL_INFO structure. 
For individually compressed files, WOF_EXTERNAL_INFO is followed by a FILE_PROVIDER_EXTERNAL_INFO_V1 structure.

Currently I coded the autoit function assuming we would be looking at individually compressed files.

I could probably either code a new function to handle wim backed files or review the current function to detect what we are looking at and send back proper details.

 

For my tests, that means I need to create a wim file (using wimboot flag maybe?) and then mount it ?

 

EDIT : i checked and realised that WIM_PROVIDER_EXTERNAL_INFO and FILE_PROVIDER_EXTERNAL_INFO_V1 are the same stuctures.

 

So in theory, the current autoit function should work in both cases.

 

@Wimb : What error/behavior are you getting with wim backed files?

 

Thanks for the interesting info on WOF structures.  :)

 

In case of WimBoot Install then for mounted VHD the REAL Files and WIM-backed Files (WOF Pointers) give WOF Status ($iReS) WOF = 0

and the 5 values in the Array of the Structure have value = 0 (as given by DllStructGetData($outbuffer, 1,4)  - with 4 being 1 till 5 to get parameter 1 till 5)

which is not yet OK since the WimBoot WIM-Backed File (WOF Pointer) is reported as WOF=0 and  this report might be improved.  ;)

 

In case of Compact Install then for mounted VHD the Compressed Files give WOF = 1 and Structure Array = 1, 2, 1, 1, 0  with LZX =1 on parameter 4

In case of UnCompressed files we get WOF = 0 and Structure Array = 0, 0, 0, 0, 0

The compact Install report is already OK.

 

WimBoot WIM-Backed WOF Pointer and WimBoot Real File = Compact LZX and UnCompressed file

WB WOF Pointer == WB Real File == Compact LZX == UnCompressed

WOF_WB_PTR_2020-08-10_074502.png == WOF_WB_REAL_2020-08-10_074334.png == WOF_COM_LZX_2020-08-10_074937.png == WOF_COM_UN_2020-08-10_080511.png

 

Your proposal:

Review the current function to detect what we are looking at and send back proper details will be the best option.  :) 

 

To study WIM-Backed files you can use VHD_WIMBOOT to Capture in WimBoot mode and Apply the captured WIM file in WimBoot mode to VHD file (Unmounted)

Then use WOF_Compress to study files in Mounted VHD. Files will be Real files when Excluded by WimBootCompress.ini and otherwise they are WOF Pointers.



#127 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2904 posts
  • Location:Nantes - France
  •  
    France

Posted 10 August 2020 - 03:13 PM

Done some tests in delphi and it appears that you need a bigger buffer for wim backed files.

So the updated wof_status function below should now work for both compressed individual files AND wim backed files.

Now going for 64 * char (or 64 * bytes...or 16 * dword...).

Func _Wof_Status_($sFilePath)
	; we could/should create a more meaningful structure but hey, who cares :)
	Local $outbuffer=DllStructCreate("char[64]")
	Local $hFile, $IRes
	Local $poutbuffer=DllStructGetPtr($outbuffer)

	$hFile = _WinAPI_CreateFile($sFilePath,2, 2)
	$IReS = _WinAPI_DeviceIoControl($hFile, $FSCTL_GET_EXTERNAL_BACKING, 0, 0, $poutbuffer, 64)
	; the return code is what we care about : 1 means compressed, 0 means not compressed
	_WinAPI_CloseHandle($hFile)

	; MsgBox(64, "ok", $IReS)
	Return $IReS
EndFunc ;==>_Wof_Status_

  • wimb and gbrao like this

#128 wimb

wimb

    Platinum Member

  • Developer
  • 2966 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 10 August 2020 - 08:55 PM

 

Done some tests in delphi and it appears that you need a bigger buffer for wim backed files.

So the updated wof_status function below should now work for both compressed individual files AND wim backed files.

Now going for 64 * char (or 64 * bytes...or 16 * dword...).

Func _Wof_Status_($sFilePath)
	; we could/should create a more meaningful structure but hey, who cares :)
	Local $outbuffer=DllStructCreate("char[64]")
	Local $hFile, $IRes
	Local $poutbuffer=DllStructGetPtr($outbuffer)

	$hFile = _WinAPI_CreateFile($sFilePath,2, 2)
	$IReS = _WinAPI_DeviceIoControl($hFile, $FSCTL_GET_EXTERNAL_BACKING, 0, 0, $poutbuffer, 64)
	; the return code is what we care about : 1 means compressed, 0 means not compressed
	_WinAPI_CloseHandle($hFile)

	; MsgBox(64, "ok", $IReS)
	Return $IReS
EndFunc ;==>_Wof_Status_

 

The bigger buffer is working OK so that WIM-Backed files get WOF = 1 Status and is also OK for WOF Compressed files of Compact Install.  :)

 

Now we only need to find the right Structure to determine WOF Compression Algorithm for all cases.



#129 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2904 posts
  • Location:Nantes - France
  •  
    France

Posted 10 August 2020 - 10:39 PM

The bigger buffer is working OK so that WIM-Backed files get WOF = 1 Status and is also OK for WOF Compressed files of Compact Install. :)

Now we only need to find the right Structures to determine WOF Compression Algorithm for all cases.


Easier actually :in wof_status2, update long[5] with long[16] when defining the buffer and 20 with 64 when calling deviceiocontrol and that should do the trick.
I will post an updated code tomorrow.

#130 wimb

wimb

    Platinum Member

  • Developer
  • 2966 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 11 August 2020 - 06:34 AM

Easier actually :in wof_status2, update long[5] with long[16] when defining the buffer and 20 with 64 when calling deviceiocontrol and that should do the trick.
I will post an updated code tomorrow.

 

Probably it must be ulong[16] instead of long[16]

 

Do you have documentation about the meaning of the 16 parameters in the structure ?

 

I found out that the second parameter of the structure is interesting and the value is:

    Compact file = 2

    WIM-Backed file = 1

    UnCompressed = 0

 

For WIM-Backed file then Parameter 7 - 11 have large value and for Compact and UnCompressed the value for these parameters is zero

 

Parameter 5 and 6 and 12-16 are always zero

 

Parameter 4 indicates Algorithm of Compact File (0=XPRESS4K,  1=LZX,  2=XPRESS8K,  3=XPRESS16K)

Parameter 4 for WIM-Backed file is always 0

 

For UnCompressed files all parameters are zero

 

Parameter 1 and 3 have value 1 for Compact and for WIM_Backed file and zero for UnCompressed file

 

WOF WIM-Backed == Compact LZX == UnCompressed

WOF_WB_2020-08-11_090616.png == WOF_COM_LZX_2020-08-11_090742.png == WOF_UN_2020-08-11_091039.png



#131 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2904 posts
  • Location:Nantes - France
  •  
    France

Posted 11 August 2020 - 11:20 AM

Based on the below MSDN statement (here) :

OutputBuffer [out]
A pointer to the output buffer, which must have a size large enough to receive a WOF_EXTERNAL_INFO structure followed by the provider data. 
For WIM backed files, WOF_EXTERNAL_INFO is followed by a WIM_PROVIDER_EXTERNAL_INFO structure. 
For individually compressed files, WOF_EXTERNAL_INFO is followed by a FILE_PROVIDER_EXTERNAL_INFO_V1 structure. 

The returned buffer will awlays start with the below 8 bytes struct (here) :

 

typedef struct _WOF_EXTERNAL_INFO

{

ULONG Version;

ULONG Provider;

}

 

 

About the provider field above, please note possible values below

const WOF_PROVIDER_WIM =1;

const WOF_PROVIDER_FILE = 2;

 

followed for individual files by the below 12 bytes struct (here)

 

typedef struct _FILE_PROVIDER_EXTERNAL_INFO_V1

{

ULONG Version;

ULONG Algorithm;

ULONG Flags;

}

 

OR followed for wim backed files by the below 36 bytes struct (here)

 

typedef struct _WIM_PROVIDER_EXTERNAL_INFO

{

ULONG Version;

ULONG Flags;

LARGE_INTEGER DataSourceId;

UCHAR ResourceHash[20];

}

 
Looking at the above structure, I am not sure if you can retrieve the compression algo for wim backed files.
 
At this stage, you would need : 
-a function to check if the file is compressed or not (wof_status)
-a function to check if the file is a wim backed file or an individual file (to be done)
-a function to report the algo used for an individual file (wof_status2)

  • wimb likes this

#132 wimb

wimb

    Platinum Member

  • Developer
  • 2966 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 11 August 2020 - 11:50 AM

 

Based on the below MSDN statement (here) :

OutputBuffer [out]
A pointer to the output buffer, which must have a size large enough to receive a WOF_EXTERNAL_INFO structure followed by the provider data. 
For WIM backed files, WOF_EXTERNAL_INFO is followed by a WIM_PROVIDER_EXTERNAL_INFO structure. 
For individually compressed files, WOF_EXTERNAL_INFO is followed by a FILE_PROVIDER_EXTERNAL_INFO_V1 structure. 

The returned buffer will also start with the below struct (here) :

 

typedef struct _WOF_EXTERNAL_INFO

{

ULONG Version;

ULONG Provider;

}

 

 

About the provider field above, please note possible values below

const WOF_PROVIDER_WIM =1;

const WOF_PROVIDER_FILE = 2;

 

followed for individual files by the below struct (here)

 

typedef struct _FILE_PROVIDER_EXTERNAL_INFO_V1

{

ULONG Version;

ULONG Algorithm;

ULONG Flags;

}

 

OR followed for wim backed files by the below struct (here)

 

typedef struct _WIM_PROVIDER_EXTERNAL_INFO

{

ULONG Version;

ULONG Flags;

LARGE_INTEGER DataSourceId;

UCHAR ResourceHash[20];

}

 
Looking at the above structure, I am not sure if you can retrieve the compression algo for wim backed files.

 

 

Thanks for explaining the Structure. It corresponds to my observation.

 

In case of Compact file we can detect the Algorithm used, but in case of WIM-Backed file the Algorithm cannot be specified.

But we always can use WIM Info (Button in VHD_WIMBOOT) and see what is the Compression Type and see if WIM is marked as WimBoot compatible.

 

In any case a Structure with ulong[16] is working OK for our purpose, so that the Status of Compact and WIM-Backed and UnCompressed is found correctly 

and also the Compression Algoithm of Compact files can be determined.

 

I can Update to WOF_Compress-37 unless you have other ideas that must be implemented ....



#133 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2904 posts
  • Location:Nantes - France
  •  
    France

Posted 11 August 2020 - 11:56 AM

...

I can Update to WOF_Compress-37 unless you have other ideas that must be implemented ....

 

I would say you are safe to publish a new version using a default 64 bytes buffer so that it handles both individual files and wim backed files.

 

Eventually in a new version you could add a button to distinguish between individual files and wim backed files.

 

Although you can already do it today : if wof_status returns 1 (it is compressed) and wof_status2 returns 0 (algo unknown), you then know that you are dealing with a wim backed files.



#134 wimb

wimb

    Platinum Member

  • Developer
  • 2966 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 11 August 2020 - 12:02 PM

I would say you are safe to publish a new version using a default 64 bytes buffer so that it handles both individual files and wim backed files.

 

Eventually in a new version you could add a button to distinguish between individual files and wim backed files.

 

Although you can already do it today : if wof_status returns 1 (it is compressed) and wof_status2 returns 0 (algo unknown), you then know that you are dealing with a wim backed files.

 

Sure, but also outbuffer parameter 2 of the Structure in _Wof_Status2_ gives this info:

 

Return of _WinAPI_DeviceIoControl has value for Status  1=WOF  and 0=UnCompressed

 

In $outbuffer the 2nd ulong has value  2=Compact  and  1=WIM-Backed  and  0=UnCompressed

 

In $outbuffer the 4th ulong is Algorith for Compact files   0=XPRESS4K  1=LZX  2=XPRESS8K  3=XPRESS16K

 

Everything can be combined in Function _Wof_Status2_    :)

 

I will launch version 3.7 ....



#135 wimb

wimb

    Platinum Member

  • Developer
  • 2966 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 11 August 2020 - 12:10 PM

Update WOF_Compress Version 3.7 is on line

 

Download from reboot.pro  - WOF_Compress  encrypted with password = bootwimb

 

Download from wimb GitHub - WOF_Compress

 

Double-click WOF_Compress_Trusted.cmd to Run as Trusted Installer program WOF_Compress_x64.exe
 

Thanks a lot Erwan for creating AutoIt3  Function _Wof_Status2_   :)

to report WOF Status of WIM-Backed and Compact files and to report the Compression Algorithm of Compact files.

 

:cheers:



#136 wimb

wimb

    Platinum Member

  • Developer
  • 2966 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 4 weeks ago

Update WOF_Compress Version 3.8 is on line

 

Download from reboot.pro  - WOF_Compress  encrypted with password = bootwimb

 

Download from wimb GitHub - WOF_Compress

 

Double-click WOF_Compress_Trusted.cmd to Run as Trusted Installer program WOF_Compress_x64.exe
 
- Added WOF Status Button for Function _List_WOF_Status to List the WOF Status of files in a given Path (Selected Drive or Folder)
 
This allows e.g. to use Notepad++ to Compare WOF_Status Lists in folder processed  made of different Installs e.g. WinNTSetup version 4.2 as compared to 4.2.3
 
In WinNTSetup 4.2.3  the Installed drivers are UnCompressed
 
new scan offlines services to exclude Driverstore *.sys files from WOF Compression

 

 
WOF_Compress_Status_2020-08-20_130321.png == WinNTSetup_WOF_2020-08-20_125044.png

 


  • alacran likes this

#137 alacran

alacran

    Gold Member

  • .script developer
  • 1484 posts
  •  
    Mexico

Posted 4 weeks ago

Good work my friend, the Wof Status Button is a good addition to the program, just tested on an old 10x64.vhd and it is working very fine.

 

I read on WinNTSetup page about the new feature to avoid all *.sys installed drivers compression on new version 4.2.3, haven't tested it myself, but I saw you already tested it, and reported it is working fine, this is also a very good addition for Wimboot and Compact installs.

 

alacran


  • wimb likes this

#138 alacran

alacran

    Gold Member

  • .script developer
  • 1484 posts
  •  
    Mexico

Posted 4 weeks ago

JFYI

 

From: http://reboot.pro/to...e-2#entry215932

 

I ran Compact /compactos:always  in a elevateted command prompt from a running 10x64 (standard install, not compact) on a VHD,  In my test got a compression of 1.8 to 1, I assume they were 4K compacted (standard Compact value on 10), see attached picture.

EDIT: Using new WOF_Compress-38, I was able to verify it was 8K compressed, then 8K seems to be the new stardard for Win10 Compact on this 10 19H1.

 

But all Portables under Documents folder remained uncompressed....

 

Just for testing I applied NTFS compression to Documents Folder, and when checking compression with WOF_Compress-38 it reports Compression 0 wich is not compressed. See attached pictures (made from Win7) please.

 

alacran

Attached Thumbnails

  • NTFS Compression.png
  • Compressed.png
  • Documents Folder.png


#139 wimb

wimb

    Platinum Member

  • Developer
  • 2966 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 3 weeks ago

Just for testing I applied NTFS compression to Documents Folder, and when checking compression with WOF_Compress-38 it reports Compression 0 wich is not compressed. See attached pictures (made from Win7) please.

 

 

NTFS Compression is Not WOF Compression, so indeed it is OK that the WOF Status = 0 for NTFS Compressed files.

NTFS Compression is easily recognised by the Blue mark of files and folders.

 

WimGAPI Mode Compact:XPRESS4K results wrongly in XPRESS8K (X) Compression, whereas wimlib is giving XPRESS4K (x)

 

quote from MSFN WinNTSetup  - This has been fixed in WinNTSetup 4.2.3

 

- fixed compact:xpress4k using 8k compression with newer wimgapi.dll

 

Did you use Compact /CompactOS:always on normal Install Or may be on Compact:XPRESS4K Install of older version of WinNTSetup ?



#140 alacran

alacran

    Gold Member

  • .script developer
  • 1484 posts
  •  
    Mexico

Posted 3 weeks ago

The blue colored files and folders are only seen from Win7, from where I took the pictures to make it more evident, on Win10 each folder has two small blue arows on right upper corner only, no file has any other identification of NTFS compression.

 

I'm aware of the recent changes on WinNTSetup 4.2.3, but that do not apply on present case, in this case it was a normal (non Compact) install, and latter it was compressed using Compact /CompactOS:always ran from the running OS itself, then that's why I'm assuming Compact 4K wich used to be standard Compact on Win10, has being replaced on new Win10 versions with Compact 8K as the standard Compact.

 

alacran



#141 wimb

wimb

    Platinum Member

  • Developer
  • 2966 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 3 weeks ago

The blue colored files and folders are only seen from Win7, from where I took the pictures to make it more evident, on Win10 each folder has two small blue arows on right upper corner only, no file has any other identification of NTFS compression.

 

I'm aware of the recent changes on WinNTSetup 4.2.3, but that do not apply on present case, in this case it was a normal (non Compact) install, and latter it was compressed using Compact /CompactOS:always ran from the running OS itself, then that's why I'm assuming Compact 4K wich used to be standard Compact on Win10, has being replaced on new Win10 versions with Compact 8K as the standard Compact.

 

 

Thanks for the detailed info.

 

I can confirm that indeed compact /CompactOS:always as applied to online recent Win10x64 results in XPRESS8K Compression of OS files



#142 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15464 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 3 weeks ago

The blue colored files and folders are only seen from Win7, from where I took the pictures to make it more evident, on Win10 each folder has two small blue arows on right upper corner only, no file has any other identification of NTFS compression.

 

I have a statement from the Microsoft Windows Enhancement Team :w00t::

 

Thank you for appreciating the visual improvements made in the Windows 10, in order to better your experience. :smiling9:

 

 

:duff:

Wonko


  • alacran likes this

#143 alacran

alacran

    Gold Member

  • .script developer
  • 1484 posts
  •  
    Mexico

Posted 3 weeks ago

One more to the list of negative improvements.

 

Actually any thing disabled or removed, they call it a improvement, just to give 2 samples:

  • System Restore disabled by default is a improvement
  • Registry backups disabled is a improvement

alacran



#144 wimb

wimb

    Platinum Member

  • Developer
  • 2966 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 3 weeks ago

The blue colored files and folders are only seen from Win7, from where I took the pictures to make it more evident, on Win10 each folder has two small blue arows on right upper corner only, no file has any other identification of NTFS compression.

 

 

In case of NTFS Compression and Windows 10 then Files also get the two small blue arrows,

but these arrows are not visible in the icons  of some .exe files like Dir_List_x64.exe and Hard_Link_List_x64.exe

 

NTFS_C_2020-08-23_162843.jpg



#145 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15464 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 3 weeks ago

In case of NTFS Compression and Windows 10 then Files also get the two small blue arrows,

but these arrows are not visible in the icons  of some .exe files like Dir_List_x64.exe and Hard_Link_List_x64.exe

 

attachicon.gifNTFS_C_2020-08-23_162843.jpg

Maybe it is part of the path towards transparent compression...

 

;)

 

:duff:

Wonko


  • antonino61 likes this

#146 antonino61

antonino61

    Silver Member

  • Advanced user
  • 726 posts
  •  
    Italy

Posted 3 weeks ago

@alacran

those are improvements, in that we have not needed them since we were able to copy the same vhd's across all the mass storage devices we have --> if something goes wrong we overwrite the culprit vhd with one of its twins. much faster, to say the least. 



#147 alacran

alacran

    Gold Member

  • .script developer
  • 1484 posts
  •  
    Mexico

Posted 3 weeks ago

In case of NTFS Compression and Windows 10 then Files also get the two small blue arrows,

but these arrows are not visible in the icons  of some .exe files like Dir_List_x64.exe and Hard_Link_List_x64.exe

 

 

I saw your picture my friend, maybe those with no arrows are not compressible, the damn two blue arrows are so small maybe I didn't see them on Win10, I like better the blue colored files/folders, they are very easy to notice them.

 

@ antonino61

 

Yes if using a Compact or Wimboot VHD those are things we don't need/use.

 

But my complain is they inactivate/remove/modify some things wich can be useful for some people, and at the same time they add a bunch of unuseful (CR)Apps that ocupy a lot of space, and also use a special pagefile (only to keep them running in the background) wich name I don't remember right now, I think it is swapfile (like in Linux) or something like that.

 

And remember there are now also 7GB reserved just for the damn monolithic mega update packages, it was never required before, with separate updates.

 

My very last test is a 6 GB 10x64 2004 Compact LZX VHD, used space is 4.08 GB (including Excell and Word 2007 + some portables) and has 1.91 GB free, all this in 1 GB less that the new reserved space required for updates, see attached picture, (additionally we can reduce used space on this VHD about 1 GB using cdob's base_winsxs.cmd) .

 

To have an almost decent OS it is required a lot of extra work to remove all unwanted on the install.wim before install it and apply certain tricks during install and latter once booting just to reduce the damn telemetry too.

 

alacran

Attached Thumbnails

  • 10x64-2004.png

  • antonino61 likes this





Also tagged with one or more of these keywords: windows, ntfs, compression, wof

1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users