Jump to content











Photo
* * * * * 4 votes

wimlib, with ImageX implementation

wim imagex winpe boot

  • Please log in to reply
365 replies to this topic

#201 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 09 May 2014 - 03:46 AM

Abbodi,

 

The problem was that your WIM contains multiple solid blocks, which I had thought Microsoft's implementation didn't support.  It turns out that it does create them if you append an image when using "recovery" compression.  The way these resources appear to be listed in the WIM lookup table is somewhat illogical, but I've updated wimlib to be compatible.  You can try out the updated v1.6.3-BETA if you have time.



#202 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 09 May 2014 - 12:03 PM

Well, it's not actually my WIM or custom created, it's original install.esd file obtained from Store Upgrade

 

thanks

exporting from ESD to WIM works great :worship:



#203 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 09 May 2014 - 02:04 PM

However, i cannot help but notice that wimlib take much time than dism
 
i.e. exporting same index from same file on same system
with dism: ~19 minutes

dism /Export-Image /SourceImageFile:install.esd /SourceIndex:4 /DestinationImageFile:install.wim /compress:recovery /CheckIntegrity

with wimlib: ~35 minutes

wimlib-imagex export install.esd 4 install.wim --compress=maximum --check

Edited by Abbodi, 09 May 2014 - 02:05 PM.


#204 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15540 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 09 May 2014 - 02:09 PM

 

However, i cannot help but notice that wimlib take much time than dism
 

 

Well, you can always ask for a refund ;).

 

:duff:

Wonko



#205 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 09 May 2014 - 02:19 PM

Abbodi, can you run 'wimlib-imagex info' on both of the resulting WIMs and make sure they are in the original WIM format?  The Version should be 68864, the Compression should be LZX, and the Chunk Size should be 32768.



#206 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 09 May 2014 - 04:33 PM

Yes, both files have the same WIM Information
 
but the one created with wimlib is smaller by around 5 MB



#207 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 09 May 2014 - 04:47 PM

Yes, I just got some similar results in a test I did.  The bottleneck is the implementation of the LZX compressor.  I had thought my implementation was only about 25% slower than Microsoft's, but apparently the difference is more significant when doing export rather than capture, or maybe the data I had tested earlier happened to be biased towards my implementation.  I could tweak the compression parameters to improve the running time, but then the compression ratio wouldn't be as good as the Microsoft implementation.  The real solution is to revisit the LZX compression code and try to further improve it, which will not be easy!

 

If you aren't already you might try using the 64-bit version of wimlib and wimlib-imagex; it might be 5-10% faster than the 32-bit version.


  • Abbodi likes this

#208 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 09 May 2014 - 06:10 PM

Yep, with 64-bit i got it in ~25 minutes



#209 SIW2

SIW2

    Frequent Member

  • Advanced user
  • 123 posts

Posted 09 May 2014 - 06:37 PM

Thanks. I had the same issue as Abbodi.

 

Just downloaded new version and it is exporting esd to wim right now.

 

Haven't tried to Apply esd with it yet.

 

 

Great, finished exporting. I didn't time it but seemed fairly quick, probably about 25 mins as reported earlier using 64 bit version.

 

edit: Just tested apply.
wimlib-imagex apply C:\install.esd 1 V:\TEST

Extracting files: 7059 MiB of 7059 MiB (100%) done
Setting timestamps on all extracted files...
Done applying WIM image.

V is an empty 24 gig partition. It took 12m10s

 

Formatted V and applied using winntsetup. Can't apply to a folder with that, has to be directly to V.

It took 3m42s.

 

Wimlib is really useful, thanks, I will be using it a lot.



#210 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 10 May 2014 - 03:19 AM

@SIW2

 

Are you saying that with WinNTSetup you applied an image containing 7 gigabytes of data in under 4 minutes?  For an ESD file, which normally contains the data in LZMS-compressed blocks of 64 MiB each, it should take at least that long just for that much raw data to be decompressed in memory, let alone written to maybe 40000 different files.  I do have a few possible optimizations in mind for extraction but I don't think I can make it that fast, haha.



#211 SIW2

SIW2

    Frequent Member

  • Advanced user
  • 123 posts

Posted 10 May 2014 - 04:26 AM

Yes, it really is that fast. I have no idea how JFX does it. Winntsetup doesn't have all the extra functions like export esd to wim.



#212 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 10 May 2014 - 04:29 AM

Hmm... How long does it take if you apply the image with DISM instead?



#213 SIW2

SIW2

    Frequent Member

  • Advanced user
  • 123 posts

Posted 10 May 2014 - 04:34 AM

I have just repeated with winntsetup, 3m52 this time.

 

Will have a go with dism now.

 

EDIT :

 With  DISM v17029 about the same 3M32s



#214 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 10 May 2014 - 04:37 AM

Also is the target partition on HDD or SSD?



#215 SIW2

SIW2

    Frequent Member

  • Advanced user
  • 123 posts

Posted 10 May 2014 - 04:48 AM

WD caviar blue 1tb. Regular spinning drive. This isn't a specially fast machine, 4gb ddr2 ram, and 6 yr old E8400 processor.



#216 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 10 May 2014 - 04:54 AM

Well I'm impressed if those numbers are right, since that is a sustained write speed of 33.3 MiB/s of pure data, not even counting metadata.  When I have time I'll run some of my own tests and try to figure out what might be going on.



#217 SIW2

SIW2

    Frequent Member

  • Advanced user
  • 123 posts

Posted 10 May 2014 - 05:11 AM

Is anybody esle getting similar results? I see cdob is here, maybe he can chip in.



#218 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 10 May 2014 - 06:49 AM

I got similar results

32-bit, 2gb ram, hdd, dual core amd athlon

applying from ESD to fixed VHD on same partition:

dism: 4M25s

wimlib-imagex: 12 mins
Extracting files: 4957 MiB of 4957 MiB

Edited by Abbodi, 10 May 2014 - 06:53 AM.


#219 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 11 May 2014 - 05:53 AM

Well, I confirmed that DISM is indeed faster when extracting an image from an ESD file, although "only" twice as fast by my test.  I've just implemented two optimizations (pre-allocate file data blocks, and don't use unnecessary temporary files) that should speed things up a bit.  The first should affect extraction from all WIM files (including ESD) on Windows, whereas the second should affect extraction from ESD files on all platforms.  The results of any performance tests would be appreciated, although I think it will take more than this to get 4 minute extractions...!



#220 SIW2

SIW2

    Frequent Member

  • Advanced user
  • 123 posts

Posted 11 May 2014 - 06:03 AM

Thanks, will give a go tomorrow.

edit: Couldn't resist trying it now.

 

Applied same esd to same partition as before.

 

7m45s, that is much quicker.



#221 memoarfaa

memoarfaa

    Member

  • Members
  • 81 posts
  •  
    Egypt

Posted 14 May 2014 - 04:32 PM

Hi I want to edit RCData.wim that extracted from bootres.dll windows8 to make OEM logo
 
I try this command
wimlib-imagex update RCData.wim 1 --check --rebuild --no-acls --verbose --command="add 'C:\wimlib\RCData\winlogo1.bmp ' '\winlogo1.bmp'"
I get 
error cannot overlay winlogo1.bmp onto existing dentry: isnot dirctory-on-dirctory
exiting with error code 23 :
conflicting file in overlay win creating a WIM image
 
then I try to delete the original   winlogo1.bmp, winlogo2.bmp, winlogo3.bmp, winlogo3n.bmp, winlogo4.bmp, winlogo5.bmp
and add new winlogo1.bmp, winlogo2.bmp, winlogo3.bmp, winlogo3n.bmp, winlogo4.bmp, winlogo5.bmp
its work with this commands 
wimlib-imagex update RCData.wim 1 --check --rebuild --no-acls --verbose --command="delete '\winlogo1.bmp'"
wimlib-imagex update RCData.wim 1 --check --rebuild --no-acls --verbose --command="delete '\winlogo2.bmp'"
wimlib-imagex update RCData.wim 1 --check --rebuild --no-acls --verbose --command="delete '\winlogo3.bmp'"
wimlib-imagex update RCData.wim 1 --check --rebuild --no-acls --verbose --command="delete '\winlogo3n.bmp'"
wimlib-imagex update RCData.wim 1 --check --rebuild --no-acls --verbose --command="delete '\winlogo4.bmp'"
wimlib-imagex update RCData.wim 1 --check --rebuild --no-acls --verbose --command="delete '\winlogo5.bmp'"
wimlib-imagex update RCData.wim 1 --check --rebuild --no-acls --verbose --command="add 'C:\wimlib\RCData\winlogo1.bmp ' '\winlogo1.bmp'"
wimlib-imagex update RCData.wim 1 --check --rebuild --no-acls --verbose --command="add 'C:\wimlib\RCData\winlogo2.bmp' '\winlogo2.bmp'"
wimlib-imagex update RCData.wim 1 --check --rebuild --no-acls --verbose --command="add 'C:\wimlib\RCData\winlogo3.bmp ' '\winlogo3.bmp'"
wimlib-imagex update RCData.wim 1 --check --rebuild --no-acls --verbose --command="add 'C:\wimlib\RCData\winlogo3n.bmp ' '\winlogo3n.bmp'"
wimlib-imagex update RCData.wim 1 --check --rebuild --no-acls --verbose --command="add 'C:\wimlib\RCData\winlogo4.bmp ' '\winlogo4.bmp'"
wimlib-imagex update RCData.wim 1 --check --rebuild --no-acls --verbose --command="add 'C:\wimlib\RCData\winlogo5.bmp ' '\winlogo5.bmp'"
wimlib-imagex optimize RCData.wim --check --compress=LZX --compress-slow --chunk-size=32768

the problem  

1- that the original  RCData.wim size is  6964 bytes and the modified   RCData.wim size is  32768 bytes 
the GUID: change form 0xa77cc23b02a78e45b0bda90b5e4bfb82 in the original  RCData.wim to  0x672c00c994f00ece30567030baf03e31
 
Can anyone tell me how to make the original RCData.wim and modified RCData.wim in the same size and GUID to avoid set testsigning on


#222 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 14 May 2014 - 05:36 PM

Hi memoarfaa,

 

In the next release, 'add' update commands will replace existing nondirectory files by default.  You can test this feature in the latest v1.6.3-BETA from https://sourceforge..../files/testing/.  This should make it so your don't have to explicitly delete those files in order to replace them.

 

It is inefficient to make your changes using separate invocations of 'wimlib-imagex update', especially when --rebuild is specified.  Instead, collect your changes into a single update command file, and pass it to a single 'wimlib-imagex update' invocation.  Do not use --rebuild if you are going to run 'wimlib-imagex optimize' afterwards anyway.  Also, the --verbose option to 'wimlib-imagex update' no longer does anything.

 

Yes, rebuilding the WIM (either by passing --rebuild to a command, or by running 'wimlib-imagex optimize') will, currently, change its GUID.  This is probably a bug, although an opposing viewpoint might be that the original WIM may have contained streams that were not written to the rebuilt WIM, so a new GUID should be assigned.

 

I don't know what you mean by "the original RCData.wim size is 6964 bytes and the modified RCData.wim size is 32768 bytes".  If you are talking about the LZX compression chunk size, it may 32768 bytes, or any greater power of 2 up to 2097152 bytes.  It cannot be 6964 bytes, and it is very unlikely to be anything other than 32768 because 32768 is the default and is seemingly the only size supported by the Microsoft implementation.

 



#223 memoarfaa

memoarfaa

    Member

  • Members
  • 81 posts
  •  
    Egypt

Posted 14 May 2014 - 10:16 PM

Hi memoarfaa,
I don't know what you mean by "the original RCData.wim size is 6964 bytes and the modified RCData.wim size is 32768 bytes".  If you are talking about the LZX compression chunk size, it may 32768 bytes, or any greater power of 2 up to 2097152 bytes.  It cannot be 6964 bytes, and it is very unlikely to be anything other than 32768 because 32768 is the default and is seemingly the only size supported by the Microsoft implementation.

hi synchronicity

I download wimlib v 1.6.3-BETA

I try other command

wimlib-imagex update RCData.wim 1 < update.txt

and update.txt is

add 'C:\wimlib\RCData\winlogo1.bmp ' '\winlogo1.bmp'
add 'C:\wimlib\RCData\winlogo2.bmp' '\winlogo2.bmp'
add 'C:\wimlib\RCData\winlogo3.bmp ' '\winlogo3.bmp'
add 'C:\wimlib\RCData\winlogo3n.bmp ' '\winlogo3n.bmp'
add 'C:\wimlib\RCData\winlogo4.bmp ' '\winlogo4.bmp'
add 'C:\wimlib\RCData\winlogo5.bmp ' '\winlogo5.bmp'

 

I mean that original RCData.wim size before use wimlib-imagex update command is 7.5 KB
After usewimlib-imagex update command RCData.wim size become 94.9 KB

 

I use thise command to make maximum compression

wimlib-imagex optimize RCData.wim --compress=LZX --compress-slow --chunk-size=32768

RCData.wim size become  86.5 kb

 

the winlogo1.bmp and other files we updating in the wim is the same size of orignal winlogo1.bmp and other files

but the size of RCData.wim increase form 7.5 KB to 86.5

 

the info of the RCData.wim is tell that the compression of RCData.wim is LZX compression and chunk-size=32768

 

here is the info of the original RCData.wim

GUID:           0xa77cc23b02a78e45b0bda90b5e4bfb82
Version:        68864
Image Count:    1
Compression:    LZX
Chunk Size:     32768 bytes
Part Number:    1/1
Boot Index:     1
Size:           6964 bytes
Integrity Info: no
Relative path junction: yes
Pipable:        no

Available Images:
-----------------
Index:                  1
Name:                   Boot Resource WIM
Description:            
Directory Count:        0
File Count:             6
Total Bytes:            554140
Hard Link Bytes:        0
Creation Time:          Wed Jul 25 20:13:05 2012 UTC
Last Modification Time: Wed Jul 25 20:13:05 2012 UTC

why i can't get size 7.5 KB of RCData.wim after I update it and compress it with maximum compression


Edited by memoarfaa, 14 May 2014 - 10:40 PM.


#224 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 15 May 2014 - 05:10 AM

Maybe your new versions of those files have less redundancy than the originals, so they can't be compressed as much.  Do you get any better results with ImageX or DISM?


  • memoarfaa likes this

#225 memoarfaa

memoarfaa

    Member

  • Members
  • 81 posts
  •  
    Egypt

Posted 15 May 2014 - 10:03 PM

Maybe your new versions of those files have less redundancy than the originals, so they can't be compressed as much.  Do you get any better results with ImageX or DISM?

 

Hi synchronicity

Exactly as you say 

I extract the original winlogo1.bmp and other bitmap  form the RCData.wim and update them without change them

and the modified   RCData.wim with the same size of the original

I don't know what is redundancy of the bitmap

can you give my any guide about  make the same  redundancy of original bitmap

 

 

 

 






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users