Jump to content











Photo
* * * * * 4 votes

wimlib, with ImageX implementation

wim imagex winpe boot

  • Please log in to reply
365 replies to this topic

#276 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 04 November 2014 - 02:10 PM

Hi
maybe a little late
but the thing is, when converting (exporting index) of decrypted store's esd file to a wim file using v1.7.2, somehow it stucks at most indexes
i left it for nearly 10 minutes, no progress
i used 32-bit version, didn't check 64-bit
 

wimlib-1.7.2\wimexport ir4.esd 2 boot.wim --compress=maximum
Writing LZX-compressed data using 2 threads
327 MiB of 469 MiB (uncompressed) written (69% done)

wimlib-1.7.2\wimexport ir4.esd 3 boot.wim --compress=maximum --boot
Writing LZX-compressed data using 2 threads
301 MiB of 499 MiB (uncompressed) written (60% done)

wimlib-1.7.2\wimexport ir4.esd 4 install.wim --compress=maximum
Writing LZX-compressed data using 2 threads
301 MiB of 4860 MiB (uncompressed) written (6% done)

only first index (Setup Media Layout) succeeds

wimlib-1.7.2\wimexport ir4.esd 1 iso.wim --compress=maximum
Writing LZX-compressed data using 2 threads
235 MiB of 235 MiB (uncompressed) written (100% done)

this doesn't happen with v1.7.1, it succeeds in all indexes



#277 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 04 November 2014 - 03:52 PM

Hi,

Have you been able to reproduce the problem on any other files (in particular, a smaller one that you could send to me)?

If not, try the following commands with wimlib 1.7.2 and tell me which succeed and which fail:

- wimexport ir4.esd 2 tmp1.wim --compress=XPRESS
- wimexport ir4.esd 2 tmp2.wim --compress=LZX
- wimexport ir4.esd 2 tmp3.wim --compress=none
- wimexport ir4.esd 2 tmp4.wim
- wimverify ir4.esd

You may also try wimlib 1.7.3-BETA if you have time, but I'd be surprised if the
results are different.
 



#278 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 04 November 2014 - 07:01 PM

i used different ESD file this time, just to be sure:

wimexport ir3.esd 2 tmp1.wim --compress=XPRESS
Writing XPRESS-compressed data using 2 threads
319 MiB of 462 MiB (uncompressed) written (69% done)

wimexport ir3.esd 2 tmp2.wim --compress=LZX
Writing LZX-compressed data using 2 threads
319 MiB of 462 MiB (uncompressed) written (69% done)

wimexport ir3.esd 2 tmp3.wim --compress=none
Writing None-compressed data using 1 thread
462 MiB of 462 MiB (uncompressed) written (100% done)

wimexport ir3.esd 2 tmp4.wim
Writing LZMS-compressed data using 1 thread
462 MiB of 462 MiB (uncompressed) written (100% done)

wimverify ir3.esd
Verifying integrity of "C:\ESD\ir3.esd": 2036 MiB of 2036 MiB (100%) done
Verifying metadata for image 1 of 4
Verifying metadata for image 2 of 4
Verifying metadata for image 3 of 4
Verifying metadata for image 4 of 4
Verifying streams: 5228 MiB of 5228 MiB (100%) done

"ir3.esd" was successfully verified.

i tried smaller file, exported of the original one (you may download it here)
dism /Export-Image /SourceImageFile:ir3.esd /SourceIndex:2 /DestinationImageFile:index2.esd /compress:recovery

but i got exactly same results:

wimexport index2.esd 1 tmp1.wim --compress=XPRESS
Writing XPRESS-compressed data using 2 threads
319 MiB of 462 MiB (uncompressed) written (69% done)

wimexport index2.esd 1 tmp2.wim --compress=LZX
Writing LZX-compressed data using 2 threads
319 MiB of 462 MiB (uncompressed) written (69% done)

wimexport index2.esd 1 tmp3.wim --compress=none
Writing None-compressed data using 1 thread
462 MiB of 462 MiB (uncompressed) written (100% done)

wimexport index2.esd 1 tmp4.wim
Writing LZMS-compressed data using 1 thread
462 MiB of 462 MiB (uncompressed) written (100% done)

i did tried 1.7.3-BETA before i report, got same results as 1.7.2



#279 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 05 November 2014 - 01:02 AM

Thanks, I was able to reproduce the problem with the file you uploaded.  The library is getting "stuck" because it is repeatedly decompressing large amounts of compressed data in order to access individual files.  Normally it doesn't do this, but there is a case, which turns out to be much more common than I had anticipated, in which an individual file may be read from a solid block while exporting.  I will see if I can fix this.



#280 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 05 November 2014 - 01:53 AM

I've updated the wimlib 1.7.3-BETA files with a fix.  Let me know if you encounter any further problems.



#281 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 05 November 2014 - 03:38 AM

Thanks :)



#282 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 14 November 2014 - 12:28 AM

I've released wimlib v1.7.3.  From the NEWS file:

 

  • Fix for very slow export from solid WIM / ESD files.
  • Fix for LZX and LZMS algorithms on non-x86 architectures, such as ARM.
  • New progress message: WIMLIB_PROGRESS_MSG_HANDLE_ERROR.  Applications may use this to treat some types of errors as non-fatal.
  • The library now permits making in-memory changes to a WIMStruct backed by a read-only WIM file.
  • Fixes for "WIMBoot" extraction mode (Windows only):
  •     When not using the WOF driver, extraction no longer fails if the disk containing the WIM file has too many partitions.
  •     When matching patterns in [PrepopulateList], all hard links of each file are now considered.
  •     The system registry files are now automatically treated as being in [PrepopulateList].
  •     Added a hack to try to work around an intermittent bug in Microsoft's WOF (Windows Overlay Filesystem) driver.


#283 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 129 posts
  •  
    Netherlands

Posted 14 November 2014 - 10:25 AM

Thanks for the new version!

I was wondering if you have any plans to include decryption? I've used Qad's esddecrypt, and from what he has been saying, it probably could be possible to include on-the-fly decryption of encryped esd files?

Just wondered what your views on this are.

Thanks again for this great tool set!



#284 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 15 November 2014 - 12:14 AM

I am not sure that "ESD" decryption belongs in wimlib.  I don't currently have an example file, but my recollection is that the XML data can declare arbitrary regions of the WIM file as encrypted.  Unless there are actually some restrictions like "only whole streams can be encrypted", it seems it would be easier to decrypt the WIM file in an external program before having wimlib open it.

There are other aspects of "encrypted ESDs" that are still unclear to me, in particular the encryption algorithm being used and what the purpose of the encryption is.



#285 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 15 November 2014 - 04:14 AM

Hi :)

qad, the one programmed esddecrypt posted this respond regarding the matter (i copy it with a very slight addition to clear things)
 

I'm not sure about what he meant by "only whole streams can be encrypted" and there are some restrictions.
But, as synchronicity says, it seems any regions except for WIM header and embedded xml/integrity table can be encrypted in any length so you should examine the whole of the file beforehand to decrypt it, I think.

Also the RSA key (and even the encryption algorithm itself) can be changed every new release so it would be better to keep decryption and other functions separated for maximum flexibility.

What I thought of when Atari800X asked me was adding the decryption function to wimlib as a new command like:


wimlib-imagex decrypt foo.esd --crypto-key=<key> --crypto-algo=<aes256 or win8>
and I didn't take "on-the-fly decryption" into account.

And to the latter question, the algorithm is standard AES-256 with NULL initial vector for current Win8/Win10TP ESDs.

The AES session key is placed within embedded xml as an RSA encrypted, and then base64 encoded string.
The RSA key for decrypting the session key itself is in another xml (i.e. windlp.state.xml used for Win8.1 Store upgrade / Win10 Preview builds, or products.xml used by MS mediacreationtool).

But I don't know the purpose of the encryption, either. It actually makes using the original ESDs a bit harder but all information that is needed to reproduce the plaintext is easily accessible to everyone.

I hope this could be helpful for him.

Thanks

 



#286 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 129 posts
  •  
    Netherlands

Posted 15 November 2014 - 04:55 AM

Thank you very much for your reactions.

One of the reasons I asked about decryption with Wimlib:

I use WinNTSetup for all my OS installs, and the first encrypted esd's that appeared this year were the first files I couldn't use directly with this great tool by JFX (it uses iso, wim and normal esd). So I asked him what this thoughts were, and if he would consider supporting them. His answer was that he could only see WinNTSetup supporting encrypted esd's, if it could be implemented in Wimlib.

So I asked Qad first, if he thought it could be done, then I asked here.

Of course it would be great if Wimlib (and then, WinNTSetup) could use encrypted esd's directly, eg. for direct apply of the OS, but I have to add that I usually have to perform and extra step on the esd files anyway, eg. add NetFX3 first. So that would mean "offline" decryption first, then mounting, dism /add-feature, etc. So in this case it would also be nice if Wimlib would support it, but the esd would be deccrypted already, by the time WinNTSetup is using it.

Of course, it's all up to you to decide what to do with it.

Thanks again for your very informative replies and thanks for creating Wimlib!!

:thumbup:



#287 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2909 posts
  • Location:Nantes - France
  •  
    France

Posted 15 November 2014 - 02:58 PM

 

I've released wimlib v1.7.3.  From the NEWS file:

 

  • Fix for very slow export from solid WIM / ESD files.
  • Fix for LZX and LZMS algorithms on non-x86 architectures, such as ARM.
  • New progress message: WIMLIB_PROGRESS_MSG_HANDLE_ERROR.  Applications may use this to treat some types of errors as non-fatal.
  • The library now permits making in-memory changes to a WIMStruct backed by a read-only WIM file.
  • Fixes for "WIMBoot" extraction mode (Windows only):
  •     When not using the WOF driver, extraction no longer fails if the disk containing the WIM file has too many partitions.
  •     When matching patterns in [PrepopulateList], all hard links of each file are now considered.
  •     The system registry files are now automatically treated as being in [PrepopulateList].
  •     Added a hack to try to work around an intermittent bug in Microsoft's WOF (Windows Overlay Filesystem) driver.

 

 

Hi Synchronicity,

 

About "The system registry files are now automatically treated as being in [PrepopulateList]", this is indeed a good idea.

But then, why not also add \bootmgr and \boot\BCD ?

On a normal installation, these will be on a hidden/reserved partition but for people having one unique partition, these need to added to the PrepopulateList otherwise system wont boot (just like it wont boot if the \Windows\System32\Config\SYSTEM is not in that section).

 

Regards,

Erwan



#288 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 15 November 2014 - 03:07 PM

I'm thinking that MS is trying to do targeted software downloads, where the download can only be used by the person who downloaded it.  But of course, handling people, metaphorically, a "locked door" and the keys to unlock that door does not, in any way, actually accomplish that --- so I can only assume this scheme was dreamed up by lawyers who were excited by the word "encryption".
 
Regardless, I think any "decryption" support belongs in an external tool, given the expected difficulty of doing decryption on-the-fly rather than ahead-of-time, and also the fact that I wouldn't be able to test for compatibility with WIMGAPI/DISM because they don't seem to support this encryption scheme either.  Decryption also seems to require little knowledge of the WIM format itself.


#289 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 15 November 2014 - 03:13 PM

About "The system registry files are now automatically treated as being in [PrepopulateList]", this is indeed a good idea.

But then, why not also add \bootmgr and \boot\BCD ?

 

Maybe this is a good idea, but I don't want to defeat the purpose of [PrepopulateList] by hard-coding file paths.

 

The reason I hard-coded the SYSTEM.* files is so that an image from a stock install.wim can be applied in "WIMBoot mode" unmodified.  Before, wimlib would encounter an error when it tried to open the extracted registry, because SYSTEM.LOG would be externally backed.

 

That's slightly different from \bootmgr and \Boot\BCD, which would actually have to be added to the WIM image by the user.



#290 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2909 posts
  • Location:Nantes - France
  •  
    France

Posted 15 November 2014 - 03:49 PM

Maybe this is a good idea, but I don't want to defeat the purpose of [PrepopulateList] by hard-coding file paths.

 

The reason I hard-coded the SYSTEM.* files is so that an image from a stock install.wim can be applied in "WIMBoot mode" unmodified.  Before, wimlib would encounter an error when it tried to open the extracted registry, because SYSTEM.LOG would be externally backed.

 

That's slightly different from \bootmgr and \Boot\BCD, which would actually have to be added to the WIM image by the user.

 

Got it, Thanks for the explanation !

 

Again, brilliant tool.



#291 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 15 November 2014 - 04:41 PM

Fair enough and totally acknowledgeable

thank you for all the effort in this great utility :)



#292 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 129 posts
  •  
    Netherlands

Posted 15 November 2014 - 04:46 PM

Yep, thanks for clarifying things in relation to encryption.

At first I didn't even want to bother you with the question, but I was still curious about your plans. After all, a "normal" setup also does the decryption "on the fly", so maybe it was worth asking.

Thanks again!


Edited by Atari800XL, 15 November 2014 - 04:46 PM.


#293 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15549 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 17 November 2014 - 12:10 PM

 

... given the expected difficulty of doing decryption on-the-fly rather than ahead-of-time...

 

 

Random thoughts :w00t: :ph34r: 

 

Could the ESD "decryption" algorithm/tool output be made pipable?

Could Wimlib accept a pipe as input?

 

Something like (say) gzip....

 

:duff:

Wonko



#294 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 18 November 2014 - 12:09 AM

Could the ESD "decryption" algorithm/tool output be made pipable?

Could Wimlib accept a pipe as input?

 

This isn't possible, at least not easily.  Before wimlib can extract anything from a WIM file, it needs to read the stream lookup table and XML data, which are at the very end of the file.

 
wimlib does have support for its own "pipable WIM" format.  But it would be difficult to take a non-pipable WIM and translate it into pipable format without using wimlib itself.  Also, wimlib's pipable WIM format does not (yet) support solid compression.


#295 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 03 January 2015 - 02:45 AM

wimlib v1.7.4 has been released.  From the NEWS file:

 

  • The Windows binary distribution no longer contains third party DLLs.  These dependencies are instead compiled directly into the libwim DLL.
  • Added more fixes for wimlib on non-x86 architectures such as ARM.
  • Extracting files to a Windows PE in-memory filesystem no longer fails if the target files do not yet exist.
  • Improved the performance of XPRESS compression and LZMS decompression.
  • Enabled SSSE3 accelerated SHA-1 computation in x86_64 Windows builds.  It will automatically be faster on newer Intel and AMD processors.
  • Removed the --with-imagex-progname and --enable-more-assertions configure options.
 

  • ReTokener likes this

#296 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 26 February 2015 - 01:59 AM

I've released wimlib v1.8.0.  This release features further improvements in compression.  From the NEWS file:

 

    Improved the LZX compressor.  It is now 15-20% faster than before and
    provides a slightly better compression ratio.

    Improved the LZMS compressor.  It now provides a compression ratio
    slightly better than WIMGAPI while still being faster and using slightly
    less memory.

    The compression chunk size in solid resources, e.g. when capturing or
    exporting a WIM file using the '--solid' option, now defaults to 64 MiB
    (67108864 bytes) instead of 32 MiB (33554432 bytes).  This provides a
    better compression ratio and is the same value that WIMGAPI uses.  The
    memory usage is less than 50% higher than wimlib v1.7.4 and is slightly
    lower than WIMGAPI's memory usage, but if it is too much, it is still
    possible to choose a lower value, e.g. with the '--solid-chunk-size'
    option to wimlib-imagex.

    The '--chunk-size' and '--solid-chunk-size' options to wimlib-imagex now
    accept the 'K', 'M', and 'G' suffixes.

    Files are now sorted by name extension when creating a solid WIM file.

    Fixed various issues related to capture/apply of EFS-encrypted files on
    Windows.

    The file list printed by 'wimdir' is now sorted by the platform-specific
    case sensitivity setting, rather than always case sensitively.  This
    also affects the library function wimlib_iterate_dir_tree().

    On Windows, some error and warning messages have been improved.
 

 

 


  • martinb likes this

#297 martinb

martinb

    Newbie

  • Members
  • 15 posts
  •  
    Netherlands

Posted 26 February 2015 - 09:53 PM

I've released wimlib v1.8.0.  This release features further improvements in compression.  From the NEWS file:

 

 

 

Hello Eric/synchronicity

 

Great yob!

 

I just made a test with the "old" version vs new version

i created a full wim from a running 2008 TS server (with volume shadow)

 

WIM filesize "old" is 38.814.246.870 bytes

Wim filesize "new" is 32.934.211.373 bytes

 

So thats good, but the best part is the speed increase.

 

time creating with "old" is 75 minutes

time creating with "new" is 55 minutes

 

Thanks a lot for the optimization, especially the speed.

 

Probably a stupid question, but i guess you cant skip the "counting" part when you start the create?

When the disk has lots of small files it takes +- 10 minutes, even when i don't exclude a single file.

 

Could it be possible to do a "blind" write?

or to start already the writing then its still building the file list?

 

edit:

 

one little "bug" i have found is the writing to the console on the creation part.

You are sending a status line to the console, even when there is no change.

Its not a real problem, but when i redirect the output to a logfile i will get lots of duplicated lines like this

written (99% done)
49 GiB of 49 GiB (uncompressed)
written (99% done)
49 GiB of 49 GiB (uncompressed)
written (99% done)
49 GiB of 49 GiB (uncompressed)
written (99% done)
49 GiB of 49 GiB (uncompressed)
written (99% done)
49 GiB of 49 GiB (uncompressed)
written (99% done)
49 GiB of 49 GiB (uncompressed)
written (99% done)
49 GiB of 49 GiB (uncompressed)
written (99% done)
49 GiB of 49 GiB (uncompressed)
written (99% done)
49 GiB of 49 GiB (uncompressed)
written (99% done)
49 GiB of 49 GiB (uncompressed)
written (99% done)
49 GiB of 49 GiB (uncompressed)
written (99% done)
49 GiB of 49 GiB (uncompressed)
written (99% done)
49 GiB of 49 GiB (uncompressed)
written (99% done)
49 GiB of 49 GiB (uncompressed)

i think it should be best to make a compare on the "new" and the "old" status, and only do a update to the console when the status is different.


Edited by martinb, 26 February 2015 - 10:18 PM.


#298 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 26 February 2015 - 10:15 PM

Hi,

Your new file is quite a bit smaller.  What version of wimlib did you use for the "old" WIM, and did you use LZX compression?

You can't skip the initial scanning pass, since the library is designed to work with separate passes for metadata and data.  During the initial scan, a lot of information is saved in memory and it isn't simply counting the files.  Actually, it doesn't necessarily slow things down by much to do the separate passes, since the total amount of work is essentially the same and the operating system will cache directory entries in memory for later.  However, it could result in some amount of slowdown overall if there is an extremely large number of files.  Sometime I might see if it's possible to get a speedup on Windows by opening files by inode number.



#299 martinb

martinb

    Newbie

  • Members
  • 15 posts
  •  
    Netherlands

Posted 26 February 2015 - 10:53 PM

Thanks, i already was afraid that is more then a simple count :)

 

 

the "old" is  wimlib 1.7.4

And i did use LZX.

I use the following command:

wimlib-imagex capture %sourcedrive% %fullwim% %sourcedrive% --config=.\%COMPUTERNAME%.ini

so no additional options, and i see

Writing LZX-compressed data using 4 threads

So its using the default compression.

 

p.s. i made a little update on the first message about a "bug" in the "writing to console" part.

its somewhere on this line:

http://sourceforge.net/p/wimlib/code/ci/master/tree/programs/imagex.c#l1057

Edited by martinb, 26 February 2015 - 10:54 PM.


#300 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 27 February 2015 - 12:07 AM

I just find it strange that, based on your numbers, the WIM file you created with wimlib v1.8.0 is only 84.6% of the size of the one created with wimlib v1.7.4.  In my tests, the improvement in the LZX compression ratio with the default settings is actually only about 0.3%.  I suspect there is another variable that affected your results.

 

The messages printed by wimlib-imagex have been intended for interactive use, not redirection to a log file.  I might improve this in a future release.






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users