Jump to content











Photo
* * * * * 4 votes

wimlib, with ImageX implementation

wim imagex winpe boot

  • Please log in to reply
365 replies to this topic

#226 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 15 May 2014 - 11:29 PM

Well, with image files you can use fewer colors, have lots of solid colors, or use a smaller resolution.  Other than that there's not much you can do.  These file sizes already seem really small to me!



#227 memoarfaa

memoarfaa

    Member

  • Members
  • 81 posts
  •  
    Egypt

Posted 16 May 2014 - 12:26 AM

this is the link of the original RCData.wim

http://www.mediafire...35a5/RCData.wim

the bitmap must be the same resolution of the original bitmap

now I will try the fewer colors


Edited by memoarfaa, 16 May 2014 - 12:26 AM.


#228 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 16 May 2014 - 12:53 AM

Seeing as the originals are just some solid blue, distorted squares on a black background (credits go to Microsoft's highly skilled graphics team!), I highly doubt you'll get the same filesize by replacing it with a more complex logo.  As I understand it, you're trying to get the same filesize because bootres.dll has this WIM file embedded inside it.  Perhaps you can rebuild bootres.dll with a larger resource section?


Edited by synchronicity, 16 May 2014 - 12:53 AM.


#229 ljycslg

ljycslg

    Newbie

  • Members
  • 26 posts
  •  
    China

Posted 19 May 2014 - 02:53 PM

suggest export command add --wimboot OPTIONS



#230 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 22 May 2014 - 07:54 PM

I've added a '--wimboot flag' to 'wimlib-imagex export' in the latest v1.6.3-BETA.  It will set the destination image as WIMBoot compatible, and also if creating a new WIM file it will set the compression type to XPRESS and the chunk size to 4096.

 

I've also made changes to the extraction code, and it should be faster now.  But if anyone else wants to do performance testing of the latest v1.6.3-BETA it would be appreciated.

 

(By the way, there are so many changes since v1.6.2 that I might end up going straight to v1.7.0 for the next release!)



#231 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 22 May 2014 - 08:56 PM

Applying ESD image took 8 minutes (previous was > 12 minutes)

 

I'm still getting same results in exporting ESD to LZX-WIM

 

thanks :)



#232 ljycslg

ljycslg

    Newbie

  • Members
  • 26 posts
  •  
    China

Posted 24 May 2014 - 08:37 AM

report a bug

 

export  offical win8.1 x64 install.wim with --wimboot 

 

Destination Image info show Architecture is x86



#233 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 26 May 2014 - 11:46 PM

I fixed the bug exporting the <ARCH> element of the XML data.  The code would always leave its value as 0, which happens to represent x86.



#234 misty

misty

    Gold Member

  • Developer
  • 1035 posts
  •  
    United Kingdom

Posted 29 May 2014 - 06:09 PM

I've just started playing around with file lists when extracting multiple files - a very useful (and incredibly fast) feature :thumbsup:. I was just wondering if there is any way to ignore missing files in the list and process the ones that are present in the target .wim?

At the moment, whenever I attempt to process a list file it will abort without processing any commands if any of the files or directories in the list is missing. The error is similar to the following -
 
[ERROR] No matches for wildcard path "\Windows\System32\wow32.dll"
Note: You can use `wimlib-imagex dir' to see what files and directories
      are in the WIM image.
ERROR: Exiting with error code 49:
       The path does not exist in the WIM image.
Regards,

Misty

#235 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 29 May 2014 - 06:21 PM

You need to use the --nullglob option.  (I should probably make it print that suggestion if you get that error.)


  • misty likes this

#236 misty

misty

    Gold Member

  • Developer
  • 1035 posts
  •  
    United Kingdom

Posted 29 May 2014 - 06:32 PM

@synchronicity
I love a quick and easy fix - thanks.

Regards,

Misty

Note to self - RTFM!!!!!!

#237 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 07 June 2014 - 11:00 PM

Hi,
 
Within the last two weeks I've implemented some additional optimizations for the decompression and compression code.  These include:
 
- Significantly faster LZ77 match-finding using binary trees instead of suffix arrays (affects LZX and LZMS compression).  Some compression ratio regressions, but it usually stays about the same or improves.
- Significantly faster Huffman code generation (affects XPRESS, LZX, and LZMS compression; also LZMS decompression).
- Significantly faster LZX preprocessing and postprocessing (affects LZX compression and decompression on x86_64 builds only).
- Slightly faster Huffman symbol decoding (affects XPRESS, LZX, and LZMS decompression).
 
These optimizations should speed up a number of use cases reported on here, such as extracting ESD files, exporting ESD to WIM, and also simply creating a WIM file with '--compress=maximum'.
 
I will be doing some additional benchmarks and testing, but if Abbodi or anyone else wants to do some benchmarks with the latest v1.7.0-BETA it would be useful.  Note that as before, the 64-bit build will have slightly better performance in some cases.
 


#238 ReTokener

ReTokener

    Frequent Member

  • Developer
  • 272 posts

Posted 08 June 2014 - 02:57 PM

Hi all

I updated wimlib-imagex CLC to use version 1.7.0:

http://reboot.pro/fi...-line-compiler/

Hope you like it.

 

T.



#239 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 08 June 2014 - 08:56 PM

Thanks for the update Eric :)
 
here what i got..

Applying ESD to fixed vhd on the same partition (almost same duration as dism 4:25)
Extracting files: 4957 MiB of 4957 MiB
32-bit: ~ 4:30 minutes
64-bit: ~ 4:30 minutes

Applying ESD to different partition (dism done it in ~ 5:05 minutes)
Extracting files: 4957 MiB of 4957 MiB
32-bit: ~ 6:10 minutes
64-bit: ~ 6:10 minutes

Exporting ESD
Writing LZX-compressed data using 2 threads: 4770 MiB of 4770 MiB (uncompressed) written
32-bit: ~ 28:10 minutes (dism 18:50)
64-bit: ~ 21:30 minutes (dism 16:20)



#240 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 08 June 2014 - 11:05 PM

Great --- I've done a few tests as well, but it's helpful to get numbers from other people who might have different setups and be working with different data.  The faster LZMS decompression in combination with the new Windows extraction code seems to be getting ESD extraction results much closer to DISM.  The only other idea I have that could improve the performance further is that the decompression could be done in a different thread from extraction.  Well, in theory one could also circumvent the filesystem driver and generate the resulting filesystem directly on the disk device, but there are many issues with that.

 

Yes, LZX compression is still slower than the MS implementation (except in the "fast" mode which is the default for 'wimlib-imagex capture' with no '--compress' option specified).  But I think it's more competitive now.  There's still the question of compression ratio:  If you saved the LZX-compressed WIM files you created, how did the compression ratio compare between DISM and wimlib-imagex v1.7.0-BETA?  And wimlib-imagex v1.6.2, if you tested that too?  (Note: whether you use the 32-bit or 64-bit builds should only affect performance, not the compression ratio.)


  • Abbodi likes this

#241 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 09 June 2014 - 11:22 AM

Well, i deleted the converted wim file after test, so i cannot compare the compression ratio right now

i did however took a note of the wim file size (like you said, it's exactly the same in both 32-bit and 64-bit)
Total Bytes: 8,672,836,317
Hard Link Bytes: 3,474,370,244

wimlib-imagex
2.34 GB (2,519,355,989 bytes)

dism
2.35 GB (2,533,504,285 bytes)
i'll test the three (dism, wimlib-imagex v1.7.0-BETA and wimlib-imagex v1.6.2) for capturing LZX-compressed WIM, ASAP.

Edited by Abbodi, 09 June 2014 - 11:36 AM.


#242 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 09 June 2014 - 03:07 PM

I named each file to reflect the capture tool
 

4957 MiB scanned (49896 files, 13460 directories)
Writing LZX-compressed data using 2 threads
4770 MiB of 4770 MiB (uncompressed) written (100% done)
WIM Information:
----------------
Path:           dism.wim
GUID:           0x688e9cece3031949a08e76d02ad5a872
Version:        68864
Compression:    LZX
Chunk Size:     32768 bytes
Part Number:    1/1
Size:           2534314981 bytes (2417 MiB)
Integrity Info: no
Relative path junction: yes

Available Images:
-----------------
Index:                  1
Name:                   CSL
Description:            CSL
Directory Count:        13459
File Count:             67737
Total Bytes:            8672836317
Hard Link Bytes:        3474370244

WIM Information:
----------------
Path:           wimlib-1.6.2.wim
GUID:           0xb950d8c52194d99b5130bffe2fc4bb27
Version:        68864
Compression:    LZX
Chunk Size:     32768 bytes
Part Number:    1/1
Size:           2524271954 bytes (2407 MiB)
Integrity Info: no
Relative path junction: yes

Available Images:
-----------------
Index:                  1
Name:                   CSL
Description:            CSL
Directory Count:        13459
File Count:             67709
Total Bytes:            8672836317
Hard Link Bytes:        3474370244

WIM Information:
----------------
Path:           wimlib-1.7.0-BETA.wim
GUID:           0x1bde50396cfdff4447be088b91c1955c
Version:        68864
Compression:    LZX
Chunk Size:     32768 bytes
Part Number:    1/1
Size:           2519073263 bytes (2402 MiB)
Integrity Info: no
Relative path junction: yes

Available Images:
-----------------
Index:                  1
Name:                   CSL
Description:            CSL
Directory Count:        13459
File Count:             67709
Total Bytes:            8672836317
Hard Link Bytes:        3474370244

wimlib-imagex-1.7.0-BETA is faster than wimlib-imagex-1.6.2 by ~ 2 to 3 minutes



#243 alacran

alacran

    Gold Member

  • .script developer
  • 1644 posts
  •  
    Mexico

Posted 10 June 2014 - 01:09 PM

I notice File count in both wimlib-imagex versions is 67709 and in Dism it is 67737, but total bytes are the same for the 3.

 

There is a diference of 28 files, How is it?  Are they 0 bytes files and then not copied to the wim when using wimlib-imagex?



#244 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 10 June 2014 - 01:14 PM

Good --- nice to hear that v1.7.0-BETA was both faster and compressed more than v1.6.2!  And both compressed more than the MS implementation.

 

I'm currently experimenting with some additional optimizations to LZX compression that might help close the remaining performance gap with the MS implementation.  I'll post back when I have something to test.

 

After that I'm planning to finally release v1.7.0 for real.



#245 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 10 June 2014 - 01:17 PM

Regarding the file count, I don't know; there are several possible reasons.  Abbodi, are you able to run 'wimlib-imagex dir' on each WIM image, redirect the results of each to a file, and send me the results?  (Or can you otherwise identify what the difference in the file tree is.)

 

Edit: files that are zero bytes in size should be archived by both DISM and wimlib-imagex, so I don't believe that's the cause of the issue.


Edited by synchronicity, 10 June 2014 - 01:18 PM.


#246 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 10 June 2014 - 01:46 PM

The difference in File count is due that wimlib-imagex by default doesn't capture Reparse Points (symbolic links or junction points)
or at least that what i got

i use this command for capture:
wimlib-imagex capture Z: file.wim CSL CSL --compress=maximum --flags CoreSingleLanguage
and got those warnings:
WARNING: Ignoring absolute symbolic link with out-of-tree target:
 (Use --norpfix to capture absolute symbolic links as-is)
 "Z:\\Documents and Settings" => "C:\Users"
 "Z:\\ProgramData\Application Data" => "C:\ProgramData"
 "Z:\\ProgramData\Desktop" => "C:\Users\Public\Desktop"
 "Z:\\ProgramData\Documents" => "C:\Users\Public\Documents"
 "Z:\\ProgramData\Start Menu" => "C:\ProgramData\Microsoft\Windows\Start Menu"
 "Z:\\ProgramData\Templates" => "C:\ProgramData\Microsoft\Windows\Templates"
 "Z:\\Users\All Users" => "C:\ProgramData"
 "Z:\\Users\Default\AppData\Local\Application Data" => "C:\Users\Default\AppData\Local"
 "Z:\\Users\Default\AppData\Local\History" => "C:\Users\Default\AppData\Local\Microsoft\Windows\History"
 "Z:\\Users\Default\AppData\Local\Microsoft\Windows\Temporary Internet Files" => "C:\Users\Default\AppData\Local\Microsoft\Windows\INetCache"
 "Z:\\Users\Default\AppData\Local\Temporary Internet Files" => "C:\Users\Default\AppData\Local\Microsoft\Windows\INetCache"
 "Z:\\Users\Default\Application Data" => "C:\Users\Default\AppData\Roaming"
 "Z:\\Users\Default\Cookies" => "C:\Users\Default\AppData\Local\Microsoft\Windows\INetCookies"
 "Z:\\Users\Default\Documents\My Music" => "C:\Users\Default\Music"
 "Z:\\Users\Default\Documents\My Pictures" => "C:\Users\Default\Pictures"
 "Z:\\Users\Default\Documents\My Videos" => "C:\Users\Default\Videos"
 "Z:\\Users\Default\Local Settings" => "C:\Users\Default\AppData\Local"
 "Z:\\Users\Default\My Documents" => "C:\Users\Default\Documents"
 "Z:\\Users\Default\NetHood" => "C:\Users\Default\AppData\Roaming\Microsoft\Windows\Network Shortcuts"
 "Z:\\Users\Default\PrintHood" => "C:\Users\Default\AppData\Roaming\Microsoft\Windows\Printer Shortcuts"
 "Z:\\Users\Default\Recent" => "C:\Users\Default\AppData\Roaming\Microsoft\Windows\Recent"
 "Z:\\Users\Default\SendTo" => "C:\Users\Default\AppData\Roaming\Microsoft\Windows\SendTo"
 "Z:\\Users\Default\Start Menu" => "C:\Users\Default\AppData\Roaming\Microsoft\Windows\Start Menu"
 "Z:\\Users\Default\Templates" => "C:\Users\Default\AppData\Roaming\Microsoft\Windows\Templates"
 "Z:\\Users\Default User" => "C:\Users\Default"
 "Z:\\Users\Public\Documents\My Music" => "C:\Users\Public\Music"
 "Z:\\Users\Public\Documents\My Pictures" => "C:\Users\Public\Pictures"
 "Z:\\Users\Public\Documents\My Videos" => "C:\Users\Public\Videos"


#247 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 11 June 2014 - 02:17 AM

Hi,

 

By default, capture operates in "reparse point fixup mode", where absolute symbolic links and junctions are changed to be relative to the capture directory.  Currently, this mode also enables logic where absolute symbolic links and junctions that point *outside* of the capture directory are excluded entirely.  In this case, you were capturing Z: but all the links pointed to C:, so they were excluded.

 

Microsoft's documentation claims that ImageX has this same behavior regarding exclusion of absolute links.  However, I just revisited this, and I found that this is not, in fact, true.  Their implementation will just keep such links as-is.  This actually makes more sense anyway, since then the links will still be captured if the drive is assigned a different letter by a live system.  I've therefore just made this change and updated the 1.7.0-BETA with it.

 

P.S. Microsoft really shouldn't be using absolute links with drive letters in the default installation of Windows, but oh well...


  • Abbodi likes this

#248 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 11 June 2014 - 02:39 AM

I'm also adding a "fix" for the fact that wimlib would count reparse point data in the total bytes of each image, but the MS implementation does not.  Although it's arguable what the expected behavior is in that case.



#249 ReTokener

ReTokener

    Frequent Member

  • Developer
  • 272 posts

Posted 11 June 2014 - 05:50 PM

Hi synchronicity

:thumbsup:

Thumbs up for your fantastic work.

It keeps me working on a frontend that will meet the requirements.

 

Here are questions that You might answer best:

Do options "--wimboot" and "--boot" contradict or exclude each other?

May "SIZE" in options "--chunk-size=SIZE" or "--solid-chunk-size=SIZE", be declared in quotation marks?

May other option-values that follow on a "=" be declared in quotation marks?

 

Thanks in advance. T.

 



#250 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 12 June 2014 - 12:31 AM

Hi ReTokener,
 
The --boot and --wimboot options are independent.  A WIM file can contain any number of images marked as "WIMBoot-compatible".  At the same time, one image may be marked as the "bootable" image of the WIM file.  Windows PE is seemingly able to boot from an image exported with both --wimboot and --boot, despite the fact that --wimboot sets the compression type to XPRESS with 4096 byte chunks which is not the typical compression type in WIM files.  That being said, I'm not sure if there's a realistic use case for doing a "WIMBoot" extraction of an image intended to be booted directly rather than installed.
 
Any command line argument may be enclosed in double quotes.  If a command line argument contains whitespace, it *must* be enclosed in double quotes.  (In UNIX shells, single quotes also can be used; I'm not sure about Windows.)  This is just how standard command-line processing works and is not at all specific to wimlib-imagex.
 





1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users