it does not let me post the file proper, this is only a picture of what I promised wonko a few days ago. Just for u to see any possible technical differences between it and what alacrán has just proposed (wofcompress). Compact+LZX.png 296.76KB 2 downloads
#26
Posted 27 April 2019 - 09:47 AM
#27
Posted 27 April 2019 - 10:09 AM
FAT's are at the beginning of the volume, in the so-called reserved sectors that are NOT part of the volume space.
The last character in the above sentence is a full stop or period.
The $MFT is normally at an offset inside the volume and UNLIKE in FAT filesystem, in NTFS "everything is a file" BUT it is possible to create a NTFS volume where the metadata (including the $MFT) are moved to the beginning of the volume.
I just explicitly provided a link to a method I proposed to achieve the above (and most probably it is the method "by Steve6375" you just cited), it would be nice if you could review it before continuing this absurd mixing up of FAT with NTFS:
http://reboot.pro/to...disk-emulation/
Wonko
That old thread from Jan/2013 is what I was talking about, after all it was your idea, not steve6375 idea, to start with an small partition and latter enlarge it.
Just re-read it again and it confirms the core of my statement on post No. 17: even on a just formated NTFS all free space is not contiguous.
Following quoted from: http://reboot.pro/to...e-2#entry166258
When you format an NTFS volume, the MFT is placed at about 1/3 to 1/2 of the way inside the volume. This makes for more efficient head-seeking on a hard disk because of the frequent accessing between the MFT and the files.
alacran
#28
Posted 27 April 2019 - 10:26 AM
That old thread from Jan/2013 is what I was talking about, after all it was your idea, not steve6375 idea, to start with an small partition and latter enlarge it.
Just re-read it again and it confirms the core of my statement on post No. 17: even on a just formated NTFS all free space is not contiguous.
Following quoted from: http://reboot.pro/to...e-2#entry166258
alacran
Sure, and directly contradicts the one you posted later (post #20) where you insisted - after having been told about your mistake - that it happened on FAT32.
And a link to the thread was posted on #14 (i.e. well before your #17 AND #20).
The message is only "try to be more careful with the accuracy of your statements" .
Wonko
#29
Posted 27 April 2019 - 10:59 AM
And a link to the thread was posted on #14 (i.e. well before your #17 AND #20).
I never saw it when I read that Post. Is it possible you added that link just after I saw it?, If you see the time of my Post No. 17 it was made almost at the same time of your Post No.14, if that is not the case maybe my glaucoma is getting worse.
alacran
#30
Posted 27 April 2019 - 11:26 AM
I never saw it when I read that Post. Is it possible you added that link just after I saw it?, If you see the time of my Post No. 17 it was made almost at the same time of your Post No.14, if that is not the case maybe my glaucoma is getting worse.
alacran
No, it was in the original post AFAICR , however in my post #18 (in reply to your #17 and before your #20) I mentioned having just posted a link to the possible approach, but don't worry, it happens to everyone to miss something from time to time .
Wonko
#31
Posted 27 April 2019 - 11:55 AM
Mostly there is enough free space on my drives and no need to use compression of files and/or folders.
But if you need or want compression, then WofCompress Tool is certainly a very interesting solution.
I will have a look if I can make a GUI for WofCompress, which will enable easy use of this tool.
I'm also not having lack of space on my PCs at the moment, but this tool deserves some time to play with it and know its potential, and keep it handy for future use, you never know when need arise.
About the WofAdk reg file I attached on a previous post, It was exported with my Win7x64 regedit.exe as WinNTSetup made it, I think it is possible it could be different on other OS
I was just looking at it carefully and I noticed what I think is a Typo on this line:
"Discription"="WofAdk Mini-Filter Driver" It seems to me it should say "Description"="WofAdk Mini-Filter Driver"
alacran
#32
Posted 29 April 2019 - 06:52 PM
thank u guys for everything. now I have just "engaged in another undertaking": after maximally reducing (compressing and compacting) my latest vhd with your novelties (wofcompress etc.) to less than 1 gb + less than 3gb wim on my wife's tablet pc, I find it hard to ramboot it, since it gives no bios option. is it possible to ramboot on UEFI? if so, how can it be done? for the time being, I am booting the conventional way. thanx in advance.
#33
Posted 01 May 2019 - 07:39 PM
Very nice tool.
For dev guys over here willing to develop more tools (whether GUI or console) around this MS feature :
-createfile + deviceiocontrol(FSCTL_SET_EXTERNAL_BACKING) will compress a file
-createfile + deviceiocontrol(FSCTL_DELETE_EXTERNAL_BACKING) will uncompress a file
-createfile + deviceiocontrol(FSCTL_GET_EXTERNAL_BACKING) will give you the backing status (i.e compressed or not) of a file
#34
Posted 10 May 2019 - 07:05 AM
Very nice tool.
For dev guys over here willing to develop more tools (whether GUI or console) around this MS feature :
-createfile + deviceiocontrol(FSCTL_SET_EXTERNAL_BACKING) will compress a file
-createfile + deviceiocontrol(FSCTL_DELETE_EXTERNAL_BACKING) will uncompress a file
-createfile + deviceiocontrol(FSCTL_GET_EXTERNAL_BACKING) will give you the backing status (i.e compressed or not) of a file
Thanks for giving info on using DeviceIoControl Function
I have made WOF_Compress autoit program and Support Topic WOF_Compress
WOF_Compress is GUI for WOF Compression and Uncompression of Drives and Folders using WofCompress Tool of JFX
WOF_Compress is GUI for WOF Compression and Uncompression of Files using Func _WinAPI_WofSetCompression of BiatuAutMiahn[@Outlook.com]
Credits and Thanks to:
JFX for making WofCompress Tool - https://msfn.org/boa...comment=1162805
alacran for topic on WofCompress Tool - http://reboot.pro/to...for-win7-win10/
BiatuAutMiahn[@Outlook.com] and Danyfirex for making Func _WinAPI_WofSetCompression - https://www.autoitsc...et-compression/
- alacran and lukyp like this
#35
Posted 11 May 2019 - 04:55 PM
Very nice tool.
For dev guys over here willing to develop more tools (whether GUI or console) around this MS feature :
-createfile + deviceiocontrol(FSCTL_SET_EXTERNAL_BACKING) will compress a file
-createfile + deviceiocontrol(FSCTL_DELETE_EXTERNAL_BACKING) will uncompress a file
-createfile + deviceiocontrol(FSCTL_GET_EXTERNAL_BACKING) will give you the backing status (i.e compressed or not) of a file
How to use these ideas in autoit ?
It seems rather complicated to me and I have no experience with these functions in autoit.
I would like to use deviceiocontrol(FSCTL_GET_EXTERNAL_BACKING) to determine whether a file is already WOF Compressed.
Any help is appreciated. The code I am using now is given in Download WOF_Compress\makebt\au3
#36
Posted 11 May 2019 - 04:59 PM
I am not very agile with autoit but this post here should answer your questions.
There a few items thus which I believe are not necessary in this autoit code : NTxx API's, the FilterAttach call, etc.
Overall this code is too long IHMO to achieve something rather simple (send a IOCTL control code).
But that should help you define your structures and see how to call windows API's.
I have made a simplistic FPC version as well here.
Everything really is in the wofhelper.pas unit in a few lines of code.
#cs ---------------------------------------------------------------------------- AutoIt Version: 3.3.14.2 Author: BiatuAutMiahn[@Outlook.com] Script Function: Set filesystem compression using windows' WOF file backing. Requires Win10 in most cases. Thanks to Danyfirex for making it actually work! #ce ---------------------------------------------------------------------------- #include <WinAPI.au3> #include <WinAPIFiles.au3> Global Const $sTagWOF_EXTERNAL_INFO = "ulong WOFEI_Version;ulong WOFEI_Provider" Global Const $sTagFILE_PROVIDER_EXTERNAL_INFO_V1 = "ulong FPEI_Version;ulong FPEI_CompressionFormat;ulong Flags" Global Const $sTagIOSTATUSBLOCK = "ptr Status;ptr Information" Global Const $sTagOBJECTATTRIBUTES = "ulong Length;handle RootDirectory;ptr ObjectName;ulong Attributes;ptr SecurityDescriptor;ptr SecurityQualityOfService" Global Const $sTagUNICODESTRING = "ushort Length;ushort MaximumLength;ptr Buffer" Global Const $FILE_OPEN = 0x00000001 Global Const $STATUS_PENDING = 0x00000103 Global Const $STATUS_SUCCESS = 0x00000000 Global Const $STATUS_ACCESS_DENIED = 0xC0000022 Global Const $OBJ_CASE_INSENSITIVE = 0x00000040 Global Const $FILE_SHARE_VALID_FLAGS = 0x00000007 Global Const $FILE_OPEN_REPARSE_POINT = 0x00200000 Global Const $FSCTL_SET_EXTERNAL_BACKING = 0x9030C Global Const $FILE_OPEN_FOR_BACKUP_INTENT = 0x00004000 Global Const $STATUS_INVALID_DEVICE_REQUEST = 0xC0000010 Global Const $STATUS_COMPRESSION_NOT_BENEFICIAL = 0xC000046F Global Const $FILE_PROVIDER_COMPRESSION_FORMAT_XPRESS4K = 0 Global Const $FILE_PROVIDER_COMPRESSION_FORMAT_LZX = 1 Global Const $FILE_PROVIDER_COMPRESSION_FORMAT_XPRESS8K = 2 Global Const $FILE_PROVIDER_COMPRESSION_FORMAT_XPRESS16K = 3 Global Const $FILE_PROVIDER_CURRENT_VERSION = 1 Global Const $WOF_CURRENT_VERSION = 1 Global Const $WOF_PROVIDER_FILE = 2 Global $hDll_NTDLL = DllOpen("ntdll.dll") FileCopy("D:\wimlib-1.12.0-BETA1\wimlib-imagex.exe","D:\test.exe",1) Local $sFilePath = "\??\D:\test.exe" MsgBox(64, "", _WinAPI_GetCompressedFileSize($sFilePath) & @CRLF) ConsoleWrite(_WinAPI_WofSetCompression($sFilePath, $FILE_PROVIDER_COMPRESSION_FORMAT_LZX) MsgBox(64, "", _WinAPI_GetCompressedFileSize($sFilePath) & @CRLF) DllClose($hDll_NTDLL) Func _WinAPI_WofSetCompression($sFilePath, $iFormat) ; Local Const $FILE_FLAG_BACKUP_SEMANTICS = 0x02000000 Local $tFilePath=DllStructCreate("wchar[260]") Local $tFilePathW=DllStructCreate($sTagUNICODESTRING) Local $tObjAttr=DllStructCreate($sTagOBJECTATTRIBUTES) DllStructSetData($tFilePath,1,$sFilePath) Local $aRet=DllCall($hDll_NTDLL,"none","RtlInitUnicodeString","ptr",DllStructGetPtr($tFilePathW),"ptr",DllStructGetPtr($tFilePath)) DllStructSetData($tObjAttr,"Length",DllStructGetSize($tObjAttr)) DllStructSetData($tObjAttr,"RootDirectory",0) DllStructSetData($tObjAttr,"ObjectName",DllStructGetPtr($tFilePathW)) DllStructSetData($tObjAttr,"Attributes",$OBJ_CASE_INSENSITIVE) DllStructSetData($tObjAttr,"SecurityDescriptor",0) DllStructSetData($tObjAttr,"SecurityQualityOfService",0) Local $tIOSB=DllStructCreate($sTagIOSTATUSBLOCK) Local $thFile=DllStructCreate("handle") Local $phFile=DllStructGetPtr($thFile) $aRet=DllCall($hDll_NTDLL,"long","NtCreateFile","ptr",$phFile,"ulong",BitOR($GENERIC_READ,$GENERIC_WRITE,0x20),"ptr",DllStructGetPtr($tObjAttr),"ptr",DllStructGetPtr($tIOSB),"int64*",0,"ulong",0,"ulong",$FILE_SHARE_VALID_FLAGS,"ulong",$FILE_OPEN,"ulong",BitOR($FILE_OPEN_FOR_BACKUP_INTENT,$FILE_OPEN_REPARSE_POINT),"ptr",0,"ulong",0) If @error Or $aRet[0]<>$STATUS_SUCCESS Then Return SetError(@error, @extended, 0) Local $hFile=DllStructGetData($thFile,1) Local $tInputBuffer=DllStructCreate("STRUCT;"&$sTagWOF_EXTERNAL_INFO&";ENDSTRUCT;STRUCT;"&$sTagFILE_PROVIDER_EXTERNAL_INFO_V1&";ENDSTRUCT") DllStructSetData($tInputBuffer,"WOFEI_Version",$WOF_CURRENT_VERSION) DllStructSetData($tInputBuffer,"WOFEI_Provider",$WOF_PROVIDER_FILE) DllStructSetData($tInputBuffer,"FPEI_Version",$FILE_PROVIDER_CURRENT_VERSION) DllStructSetData($tInputBuffer,"FPEI_CompressionFormat",$iFormat) Local $pInputBuffer=DllStructGetPtr($tInputBuffer) Local $iInputBuffer=DllStructGetSize($tInputBuffer) Local $iTried=0, $iRet=0 Do $iRet=_WinAPI_NtFsControlFile($hFile,$FSCTL_SET_EXTERNAL_BACKING,$pInputBuffer,$iInputBuffer) If @error Then Return SetError(@error, @extended, 0) If $iRet=$STATUS_INVALID_DEVICE_REQUEST And Not $iTried Then __DriveAttachWOF(__PathGetDrive($sFilePath)) $iTried=1 ContinueLoop EndIf $iTried=1 Until $iTried DllCall($hDll_NTDLL,"long","NtClose","handle",$hFile);close Handle EndFunc ;==>_WinAPI_SetCompression Func _WinAPI_NtFsControlFile($hFile,$iFsControlCode,$pInputBuffer,$iInputBuffer,$pOutputBuffer=0,$iOutputAvail=0) Local $tIOSB=DllStructCreate($sTagIOSTATUSBLOCK) Local $pIOSB=DllStructGetPtr($tIOSB) Local $aRet=DllCall($hDll_NTDLL,"int","NtFsControlFile","HANDLE",$hFile,"ptr",0,"ptr",0,"ptr",0,"ptr",$pIOSB,"uint",$iFsControlCode,"ptr",$pInputBuffer,"uint",$iInputBuffer,"ptr",$pOutputBuffer,"uint",$iOutputAvail) If @error Or Not $aRet[0] Then Return SetError(@error,@extended,0) Local $iRet=0, $iError=0 If $aRet[0]=$STATUS_PENDING Then $iRet=_WinAPI_WaitForSingleObject($hFile) $iError=_WinAPI_GetLastError() If Not $iError Then SetError(1,$iError,0) Return SetError(1,$iRet,0) EndIf Return SetError(0,0,$aRet[0]) EndFunc Func __DriveAttachWOF($sDrive) ; win32_try_to_attach_wof Local $aRet,$hDll_FltLib $hDll_FltLib=DllOpen("FltLib.dll") If @error Then Return SetError(1,0,0) $aRet=DllCall($hDll_FltLib,"int","FilterAttach","WSTR","wof","wstr",$sDrive,"ptr",0,"int",0,"ptr",0) If @error Or Not $aRet[0] Then $aRet=DllCall($hDll_FltLib,"int","FilterAttach","WSTR","wofadk","wstr",$sDrive,"ptr",0,"int",0,"ptr",0) If @error Or Not $aRet[0] Then Return SetError(@error,@extended,0) EndIf DllClose($hDll_FltLib) Return SetError(0,0,$aRet[0]) EndFunc ;==>DriveAttachWOF Func __PathGetDrive($sPath) $sPath=_WinAPI_GetFullPathName($sPath) If @error Or $sPath="" Then Return SetError(1,0,0) Local $aTest=StringRegExp($sPath,"^([A-Za-z]\:).*$",1) If @error Then Return SetError(1,0,0) Return SetError(0, 0,"\\.\" & StringLower($aTest[0])) EndFunc ;==>PathGetDrive
#37
Posted 11 May 2019 - 05:08 PM
The Autoit script of Biatu is already part of my WOF_Compress program.
It works quite well for WOF Compression of a file.
But it would be better if I know the WOF Compression Status of the file so that I can skip when file is already compressed.
Similarly for Un-Compression I don't have the autoit code yet ...
Unfortunately, I cannot transfer the FPC code into autoit code ...
#38
Posted 11 May 2019 - 05:09 PM
you mean you want the code below in autoit?
i can give it a try in autoit if so.
function IsWofCompress(Filename:string):boolean; var hfile:thandle; ret:dword; Backing:ExternalBacking; begin hFile := CreateFile(pchar(Filename), {0x181} GENERIC_READ, FILE_SHARE_DELETE or FILE_SHARE_READ or FILE_SHARE_WRITE, nil, OPEN_EXISTING, $0A000000, 0); if hfile=thandle(-1) then raise exception.create('invalid handle'); fillchar(Backing,sizeof(Backing),0); Ret := 0; if DeviceIoControl(hFile, FSCTL_GET_EXTERNAL_BACKING, nil, 0, @backing, sizeof(ExternalBacking), ret, nil)<>true then raise exception.create('DeviceIoControl failed:'+inttostr(getlasterror)); end;
#39
Posted 11 May 2019 - 05:13 PM
Yes that is the code that I would like to use in Autoit.
It would be very nice if you can try to improve my Autoit program code as given in the Download
What I miss in the autoit code is both:
function IsWofCompress(Filename:string):boolean;
function wofuncompress(filename:string):boolean;
wheras
function wofcompress(filename:string):boolean;
is working already ok as given by Biatu
Autoit Info:
_WinAPI_CreateFileEx - https://www.autoitsc...reateFileEx.htm
_WinAPI_CreateFile - https://www.autoitsc..._CreateFile.htm
_WinAPI_DeviceIoControl - https://www.autoitsc...ceIoControl.htm
#41
Posted 11 May 2019 - 06:37 PM
ok working on uncompress as a prio.
#42
Posted 11 May 2019 - 07:00 PM
below code works for me for uncompress (tested it against a compressed file).
need some error checking, etc ...
now looking at getting the status of a file (compressed or not).
Func _wof_uncompress_($sFilePath) Local $hFile $hFile=_WinAPI_CreateFile ( $sFilePath,2, 4 ) ; would be wise to check the return code of deviceiocontrol... _WinAPI_DeviceIoControl ( $hFile, $FSCTL_DELETE_EXTERNAL_BACKING , 0 , 0 , 0 , 0 ) _WinAPI_CloseHandle($hFile) MsgBox(64, "ok", "ok") EndFunc
- wimb likes this
#43
Posted 11 May 2019 - 07:22 PM
the below code works for me to check the status of a file (compressed or not).
Func _Wof_Status_($sFilePath) ; we could/should create a more meaningful structure but hey, who cares :) ; i know it is 20 bytes because i counted them all on my fingers Local $outbuffer=DllStructCreate("char[20]") Local $hFile Local $IReS $hFile=_WinAPI_CreateFile ( $sFilePath,2, 2 ) ; $outbuffer should actually be a pointer but again, we are not going to use this buffer... $IReS =_WinAPI_DeviceIoControl ( $hFile, $FSCTL_GET_EXTERNAL_BACKING , 0 , 0 , $outbuffer , 20 ) ;the return code is what we care about : 1 means compressed, 0 means not compressed MsgBox(64, "ok", $IReS) _WinAPI_CloseHandle($hFile) EndFunc
- wimb likes this
#44
Posted 11 May 2019 - 07:34 PM
and the last bit, the compress part
Func _wof_compress_($sFilePath) Local $hFile Local $tInputBuffer=DllStructCreate("STRUCT;"&$sTagWOF_EXTERNAL_INFO&";ENDSTRUCT;STRUCT;"&$sTagFILE_PROVIDER_EXTERNAL_INFO_V1&";ENDSTRUCT") DllStructSetData($tInputBuffer,"WOFEI_Version",$WOF_CURRENT_VERSION) DllStructSetData($tInputBuffer,"WOFEI_Provider",$WOF_PROVIDER_FILE) DllStructSetData($tInputBuffer,"FPEI_Version",$FILE_PROVIDER_CURRENT_VERSION) DllStructSetData($tInputBuffer,"FPEI_CompressionFormat",$FILE_PROVIDER_COMPRESSION_FORMAT_LZX) Local $pInputBuffer=DllStructGetPtr($tInputBuffer) $hFile=_WinAPI_CreateFile ( $sFilePath,2, 4 ) ; would be wise to check the return code of deviceiocontrol... _WinAPI_DeviceIoControl ( $hFile, $FSCTL_SET_EXTERNAL_BACKING , $pInputBuffer , 20 , 0 , 0 ) _WinAPI_CloseHandle($hFile) MsgBox(64, "ok", "_compress_") EndFunc
- wimb likes this
#45
Posted 11 May 2019 - 07:35 PM
Very Good
Nice to hear that there is already so much progress !
#46
Posted 11 May 2019 - 07:35 PM
The last 3 posts illustrate the below with 3 functions coded with Autoit :
-createfile + deviceiocontrol(FSCTL_SET_EXTERNAL_BACKING) will compress a file
-createfile + deviceiocontrol(FSCTL_DELETE_EXTERNAL_BACKING) will uncompress a file
-createfile + deviceiocontrol(FSCTL_GET_EXTERNAL_BACKING) will give you the backing status (i.e compressed or not) of a file
#47
Posted 11 May 2019 - 07:42 PM
Amazing how fast you realised everything in Autoit, which is not your usual programming language
#48
Posted 11 May 2019 - 07:46 PM
Amazing how fast you realised everything in Autoit, which is not your usual programming language
I have been a professional developper in my early days and was "educated" to use different langages.
Once you know the basics like declare/set a variable, use conditionals and call windows api, you are pretty much set.
Although there are some langages I have a very hard time with and especially those being picky on identation and/or being case sensitive or those requiring exotic and/or fat runtimes.
There I must say that Autoit is quite intuitive and easy going (for a scripting langage...).
This has been my first true experience (apart from the classic "hello world" a long time ago) : so far so good
#49
Posted 11 May 2019 - 08:14 PM
I have done some quick testing ....
All three Functions are working apart OK, but not combined.
When you do first _Status_() and then _Uncompress() failes (there is no Uncompression)
But simply using _Uncompress() is working OK
#50
Posted 11 May 2019 - 08:20 PM
it may be that the handle (from createfile) is never "freed".
i would add a closehandle at the end of each function :
_WinAPI_CloseHandle($hFile)
that may fix this issue as one should normally always close a previous opened handle.
if not, i'll test it all in one function.
https://www.autoitsc...CloseHandle.htm
EDIT : i will edit/fix the previous posts.
- wimb likes this
Also tagged with one or more of these keywords: compress, native compression, xpress4k, xpress8k, xpress16k, lzx
Answered
Groups →
Windows Extreme →
Windows 10 →
Reducing wimboot source wim file using LZX Compression, and VHD using gzip or LZ4 Compression, to save room, and also load fas...Started by alacran , 21 Mar 2019 wimboot, ramboot, lzx, gzip, lz4 and 1 more... |
|
|
||
Groups →
Community forum →
Requests →
Super CompressStarted by Ankur.k1991 , 21 Jul 2014 compress |
|
|
||
Boot methods & tools →
WinBuilder →
Win7PE →
Shortcut compress Win7PEStarted by enciktangankidal , 24 Aug 2011 compress, win7pe |
|
|
1 user(s) are reading this topic
0 members, 1 guests, 0 anonymous users