Jump to content











Photo
* * * * * 4 votes

wimlib, with ImageX implementation

wim imagex winpe boot

  • Please log in to reply
365 replies to this topic

#101 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 08 December 2013 - 09:54 AM

wimlib v1.5.3 has been released:

  • The new LZX compressor added in v1.5.2 has been improved and is now enabled by default, except when `wimcapture' or `wimappend' is run *without* the '--compress' option, in which case the faster LZX compressor is used (the same as before).  This behavior is reasonably consistent with ImageX which actually uses "fast" (XPRESS) compression by default.  In those cases, use '--compress=maximum' to explicitly capture a WIM image using the new (slower but better) LZX compressor.
  • The '--compress-slow' option still exists to `wimlib-imagex optimize', but its new behavior is to tweak the new LZX compressor even more to produce an even better compression ratio at the cost of more time spent compressing.
  • `wimlib-imagex optimize' now supports the '--compress=TYPE' option, which recompresses the WIM file using the specified compression TYPE.  The new library API function used for this is wimlib_set_output_compression_type().
  • Added the wimlib_get_xml_data() function to allow library clients to easily retrieve the raw XML data from a WIM file if needed.
  • Fixed a bug that could cause an error code to be incorrectly returned when writing XML data containing a <WINDOWS> element.
  • Mounted WIM images will now correctly show the default file stream even if appears in the alternate data stream entries of the corresponding WIM directory entry.

I'd be interested in hearing whether anyone finds the updated LZX compressor useful.  While the v1.5.2 LZX compressor seemed to compress slightly worse than the MS one and was several times slower, the v1.5.3 compressor seems to compress very slightly better than the MS one and is only slightly slower.  (With --compress-slow even more compression is obtained, but it's significantly slower.)



#102 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 02 January 2014 - 09:42 PM

wimlib v1.6.0 has been released.  From the NEWS file:

  • Support for extracting and updating the new version 3584 WIMs has been added.  These WIMs typically pack many streams ("files") together into a single compressed resource, thereby saving space.  This degrades the performance of random access (such as that which occurs on a mounted image), but optimizations have been implemented for extraction.  These new WIM files also typically use a new compression format (LZMS), which is similar to LZMA and can offer a better compression ratio than LZX.  These new WIM files can be created using `wimcapture' with the '--compress=lzms --pack-streams' options.  Note: this new WIM format is used by the Windows 8 web downloader, but important segments of the raw '.esd' files are encrypted, so wimlib will not be able to extract such files until they are first decrypted.
  • wimlib now supports extracting files and directories from a WIM image based on a "listfile" that itself contains the list of paths to extract.  For `wimextract', the syntax is to specify @LISTFILE instead of a PATH, and for the library itself, the new APIs are wimlib_extract_pathlist() and wimlib_extract_paths().  Path globs containing wildcard characters are supported.
  • For searching WIM files, wimlib now has configurable case sensitivity.  The default on Windows is still case-insensitive and the default on UNIX-like systems is still case-sensitive, but this can be overridden on either platform through flags to wimlib_global_init().  For `wimlib-imagex', the environmental variable WIMLIB_IMAGEX_IGNORE_CASE can be set to 1 or 0 for case-insensitive or case-sensitive behavior, respectively.
  • Support for compression chunk sizes greater than the default of 32768 bytes has been added.  A larger chunk size typically results in a better compression ratio.  However, the MS implementation is seemingly not compatible with all chunk sizes, especially for LZX compression, so the defaults remain unchanged, with the exception of the new LZMS-compressed WIMs, which use a larger chunk size by default.
  • The compression/decompression API exported by wimlib has been changed.  Now one set of functions handles all supported compression formats.
  • `wimcapture' and `wimappend' will now display the progress of scanning the directory tree to capture, in addition to the progress of writing data to the WIM.  The '--verbose' option no longer does anything.  The library API change for this is the addition of several members to `struct wimlib_progress_info_scan' available to progress callbacks.
  • `mkwinpeimg' now correctly handles the '--start-script' option when the start script is not in the working directory.
  • Sequential extraction, previously requested by using WIMLIB_EXTRACT_FLAG_SEQUENTIAL, is now the default.  WIMLIB_EXTRACT_FLAG_FILE_ORDER can be used to get the old default behavior (extract in file order).


#103 Vortex

Vortex

    Frequent Member

  • Advanced user
  • 239 posts

Posted 03 January 2014 - 09:14 AM

Hi synchronicity,

 

Thanks for the new release. I created successfully an online backup of my XP installation with wimlib. Restored it without any problem after booting to WinPE.



#104 ReTokener

ReTokener

    Frequent Member

  • Developer
  • 234 posts

Posted 07 March 2014 - 12:46 AM

Dear synchronicity

 I´m using imagex.exe for quite a while but I see your tool is much more particular in many cases.

Thanks a lot for developing and sharing.

Here is a interface that could make the usage of your tool a bit more interesting.

It´s just a demo, that is yet to be formed.

This early stage might be the right time to get some infos and avoid mistakes in the development.

Hope you like it.

Any suggestions are appreciated.

 

Sincerely T.

 

Download:

Attached File  wimlib-gui_140305_3.rar   669.56KB   609 downloads



#105 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 07 March 2014 - 01:59 AM

  Hi ReTokener,

 
I took a look at the GUI you're developing.  For a demo it's looking good and I appreciate the interest!  I do have some suggestions about the general design though.  I think a graphical interface allows for a different way of using the software, and it should not directly mirror the command-line interface.  In particular, I don't think it's very helpful to just list the command-line options in a table.  Instead, it would be more fitting to organize the options in a way that is intuitive and useful to typical users.
 
These are just a few examples of potential improvements in the interface for capturing an image as a new WIM file, assuming you will keep that use case distinct from appending an image to an existing WIM file:
 
* For choosing the compression type (like --compress), have a dropdown menu with available compression types:
    - None
    - LZX ("maximum")
    - XPRESS ("fast")
    - LZMS ("recovery") [but distinguish this somehow since it reduces compatibility]
 
* The --rebuild option only has an effect for append; don't include it in the GUI for capture.
 
* --unix-data and --dereference don't work on Windows; don't include them if your GUI is for Windows only.
 
* --norpfix can just be a check box.  (--rpfix is the default and is there for append.)  This should also be renamed to something more clear to the user, like "preserve absolute symbolic link targets".
 
* Similarly, --check can just be a check box. (--nocheck is the default and is there for append.)  Rename it to something more explanatory like "include whole-archive checksums".
 
* Similarly, --pipable can just be a check box.  Rename it to something like "create pipable WIM (incompatible with Microsoft implementation)".
 
* --solid and --pack-streams mean the same thing, as do --pack-chunk-size and  --solid-chunk-size.  Rename them to something like: "create solid archive (reduced compatibility)", "solid archive compression chunk (dictionary) size".  I expect these options to be rarely used, at least at this point in time, however.
 
Overall, I think you need to consider how to best present the functionality of the software to the user in graphical form and not overwhelm them with various options that they typically won't be familiar with.  It also might be nice to integrate documentation into the interface, although I'm not quite sure how you could do this.  Finally, you might also consider using wimlib directly rather than relying on wimlib-imagex, as this may allow you to customize the user experience better (for example, by having progress bars, or allowing operations like "opening" a WIM file without having to map onto wimlib-imagex commands).
 
Those were just a few thoughts I had --- hope they're helpful!  
 


#106 ReTokener

ReTokener

    Frequent Member

  • Developer
  • 234 posts

Posted 07 March 2014 - 01:24 PM

Hi synchronicity

 

Thanks for your hints.

Options will be sorted, according to your suggestions.

 

you might also consider using wimlib directly rather than relying on wimlib-imagex

What does this mean?  :huh:

 

 



#107 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 07 March 2014 - 03:18 PM

I meant that you could consider using the library directly (on Windows, the libwim-9.dll file), as opposed to the command-line frontend (wimlib-imagex) that uses it.  The library API is documented at http://wimlib.sourceforge.net.  I'd probably do this if I were developing a GUI frontend.  But it would be faster to develop the GUI if you only rely on wimlib-imagex, especially since you're already familiar using it.  So it's just something to think about.



#108 ReTokener

ReTokener

    Frequent Member

  • Developer
  • 234 posts

Posted 07 March 2014 - 03:43 PM

Dear synchronicity

If you don´t mind, it will become a "command line compiler".    :ermm:

T.



#109 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2794 posts
  • Location:Nantes - France
  •  
    France

Posted 11 April 2014 - 08:23 PM

Hi Synchronicity,

 

I am wondering if you had a chance to take a look at the new WIM feature (wimboot) that came out with the latest Windows 8.1.

I started a discussion here.

 

If I understand ok, it seems that when applying a "wimboot" WIM to a drive, it actually writes pointers back to the WIM file thus saving space on the target.

 

Adding this feature to wimlib could be a good idea?

 

Regards,

Erwan



#110 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 12 April 2014 - 04:11 AM

Maybe.  Unfortunately I don't have an updated Windows 8.1 to play around with.  I was, however, able to get the new DISM and WIMGAPI from the Windows ADK and create a "WIMBoot" WIM.  It had a new compression flag set, but it actually just used XPRESS compression with a different chunk size (4096 bytes --- very small, I guess so that random access is fast).  So it's trivial to support reading such files.

 

But besides that, I can't get ahold of the new driver ("wof.sys") which is apparently required for WIMGAPI (or DISM) to apply a WIM in WIMBoot mode.  Also, apparently the supported installations include a file "System\Windows32\wimbootcompress.ini", which I don't know the format of.  I don't know if you have suggestions for obtaining these files.  If not, it may be helpful if you could apply a WIMBoot WIM on your Windows 8.1 using DISM (with /WIMBoot flag), then capture a normal WIM of a small subdirectory (wimlib-imagex should work for that).  If I could examine the resulting WIM then I might be able to determine the format of the "pointer" files, which are probably reparse points, and recreate them in wimlib.



#111 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2794 posts
  • Location:Nantes - France
  •  
    France

Posted 12 April 2014 - 09:34 AM

Hi Synchronicity,

 

As an IT pro, I have access to MSDN downloads.

I'll see if I can retrieve the last version of windows 8.1 with update 1 and then apply/capture WIM files with the WIMBOOT flag.

 

As for the the ini file, if I can get a hand on it, I guess I can dump its content here without violating any copyright.

 

Regards,

Erwan



#112 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2794 posts
  • Location:Nantes - France
  •  
    France

Posted 12 April 2014 - 12:48 PM

 content of wimbootcompress.ini

; This 
is the inbox configuration file used for deploying or capture a
; WIMBoot system. Please do not remove this file because WIMCaptureImage 
; and WIMApplyImage will fail if WIM_FLAG_WIM_BOOT flag is specified.

[CompressionExclusionList]
ntoskrnl.exe

[PrepopulateList]
*winload.*
*winresume.*
wof.sys
\Windows\System32\Config\SYSTEM

[ExclusionList]
\$bootdrive$
\$dwnlvldrive$
\$lsdrive$
\$installdrive$
\$Recycle.Bin\*
\bootsect.bak
\hiberfil.sys
\pagefile.sys
\ProgramData\Microsoft\Windows\SQM
\System Volume Information
\Users\*\AppData\Local\GDIPFONTCACHEV1.DAT
\Users\*\NTUSER.DAT*.TM.blf
\Users\*\NTUSER.DAT*.regtrans-ms
\Users\*\NTUSER.DAT*.log*
\Windows\AppCompat\Programs\Amcache.hve*.TM.blf
\Windows\AppCompat\Programs\Amcache.hve*.regtrans-ms
\Windows\AppCompat\Programs\Amcache.hve*.log*
\Windows\CSC
\Windows\Debug\*
\Windows\Logs\*
\Windows\Panther\*.etl
\Windows\Panther\*.log
\Windows\Panther\FastCleanup
\Windows\Panther\img
\Windows\Panther\Licenses
\Windows\Panther\MigLog*.xml
\Windows\Panther\Resources
\Windows\Panther\Rollback
\Windows\Panther\Setup*
\Windows\Panther\UnattendGC
\Windows\Panther\upgradematrix
\Windows\Prefetch\*
\Windows\ServiceProfiles\LocalService\NTUSER.DAT*.TM.blf
\Windows\ServiceProfiles\LocalService\NTUSER.DAT*.regtrans-ms
\Windows\ServiceProfiles\LocalService\NTUSER.DAT*.log*
\Windows\ServiceProfiles\NetworkService\NTUSER.DAT*.TM.blf
\Windows\ServiceProfiles\NetworkService\NTUSER.DAT*.regtrans-ms
\Windows\ServiceProfiles\NetworkService\NTUSER.DAT*.log*
\Windows\System32\config\RegBack\*
\Windows\System32\config\*.TM.blf
\Windows\System32\config\*.regtrans-ms
\Windows\System32\config\*.log*
\Windows\System32\SMI\Store\Machine\SCHEMA.DAT*.TM.blf
\Windows\System32\SMI\Store\Machine\SCHEMA.DAT*.regtrans-ms
\Windows\System32\SMI\Store\Machine\SCHEMA.DAT*.log*
\Windows\System32\sysprep\Panther
\Windows\System32\winevt\Logs\*
\Windows\System32\winevt\TraceFormat\*
\Windows\Temp\*
\Windows\TSSysprep.log
\Windows\winsxs\poqexec.log
\Windows\winsxs\ManifestCache\*
\Windows\servicing\Sessions\*_*.xml
\Windows\servicing\Sessions\Sessions.back.xml

[CompressionFolderList]
\Windows\System32\WinEvt\Logs
\Windows\Installer



#113 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2794 posts
  • Location:Nantes - France
  •  
    France

Posted 12 April 2014 - 04:45 PM

I dir the following :

 

Capture

dism /capture-image /imagefile:c:\test.Wim /capturedir:c:\startrek /name:test /wimboot

 

deleted the content of c:\startrek

 

Apply

dism /apply-image /imagefile:c:\test.Wim /index:1 /applydir:c:\startrek /wimboot

 

and then finally capture again with wimlib wimcapture.cmd c:\startrek c:\test2.wim

 

I will not host these 2 files for you.

 

/Erwan

 

edit1: here below the WIM files.

test.wim

test2.wim

 

edit2:

See here about the trick I had to perform to run wimboot on a folder.

Wimboot seems to be meant to run on a drive, not a folder.



#114 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 12 April 2014 - 05:30 PM

Thanks.  Unfortunately I don't see any "pointer files" in test2.wim, only the original files.  I was expecting them to be reparse points.  Maybe the redirection is actually implemented at a lower level, or maybe it simply doesn't work if you don't use a full drive.  To rule out the latter case, perhaps someone has set up a fully working WIMBoot installation of Windows 8.1?  If you use wimlib-imagex to capture a directory containing system files and then run 'wimlib-imagex dir --detailed  WIMFILE', do any files show up with FILE_ATTRIBUTE_REPARSE_POINT set?



#115 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2794 posts
  • Location:Nantes - France
  •  
    France

Posted 12 April 2014 - 05:34 PM

Hi Synchronicity,

 

I was having the same doubts : not sure wimboot works at a folder level.

 

I am currently capturing a full drive as we speak.

Once done I'll format my drive and apply the wim file with the wimboot flag.

I will then run again a wimlib capture.

 

Regards,

Erwan



#116 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2794 posts
  • Location:Nantes - France
  •  
    France

Posted 12 April 2014 - 06:35 PM

See here for a working wimboot scenario.

 

Will post a wim file captured on a "wimboot applied" drive later today.

 

Edit : captured wim file and output of wimlib-imagex dir --detailed  WIMFILE

7zip.wim

7zip.txt



#117 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 12 April 2014 - 09:37 PM

I'm still not seeing any "pointer" files.  Is that 7-Zip installation something you captured into the original WIMBoot WIM, or did you add it later?  Maybe this is actually some sort of union mount...



#118 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2794 posts
  • Location:Nantes - France
  •  
    France

Posted 12 April 2014 - 09:42 PM

I'm still not seeing any "pointer" files.  Is that 7-Zip installation something you captured into the original WIMBoot WIM, or did you add it later?  Maybe this is actually some sort of union mount...


Was there before the capture.
The wim driver is hiding the pointer i believe.

#119 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 12 April 2014 - 10:25 PM

Well, I'm not sure we'll get too much farther until I'm able to get a copy of that filter driver, although I will be able to support reading WIMBoot WIMs since it's not even a new compression format.  But, what happens if you mount the WIMBoot-enabled drive from a system that does *not* have the filter driver installed?  (Maybe try a not-completely-up-to-date version of Windows PE?)



#120 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2794 posts
  • Location:Nantes - France
  •  
    France

Posted 12 April 2014 - 11:16 PM

You could get a hold of the driver from the evaluation win8.1 update one iso.

Ill mount the wim file on a earlier version of windows tomorrow.

#121 Blackcrack

Blackcrack

    Frequent Member

  • Advanced user
  • 412 posts
  •  
    Germany

Posted 13 April 2014 - 07:54 AM

may not bad to implement in Reactos, because, it's like from Microsoft and have

an standard like Microsoft, but free developed like the pedant in reactos,

the paint or notepad.. so it is also you wimlib maybe highest interested to

implement maybe into reactos .... and may better because an better compressing ..

best regards

Blacky



#122 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 13 April 2014 - 08:39 AM

All right, I got ahold of an updated Windows 8.1 for testing.  The pointer files are indeed reparse points, although as you noticed they are hidden when viewing from the system with the filter driver running.  I've figured out the format of the reparse data, with the exception of a few fields that seem to be invariant.  Also, in the reparse data the WIM file is identified by an integer ID, and there's a separate undocumented ioctl (probably handled by WoF.sys) to translate the full path of a WIM file into a unique integer ID.  I will keep working on this when I have time and try to get something working.

 

Blacky, I don't know if there would be demand for having this specific project in ReactOS, since wimlib is not binary compatible with WIMGAPI, nor is wimlib-imagex an exact clone of imagex.  Although I suppose if they are interested in the WIM format, or XPRESS or LZX compression, then it could be useful anyway.  (Also note that WIM is not, in fact, a standard, although of course hardly anything that MS does is a standard.)



#123 Blackcrack

Blackcrack

    Frequent Member

  • Advanced user
  • 412 posts
  •  
    Germany

Posted 13 April 2014 - 08:52 AM

*g* MS make his own Standarts with stolen ideas , already for years (Xerox/lisa *g*) *g* is it not so? * giggle *.

Reactos hold it self on MS to have an standard and be up to 100% compatible ..

Your program mus only 100% compatible, and it is more better and if more enhanced as the

Microsoft's program, it's only good for the Reactos users , you see :) imo ;)

So, it is may be worth considering to add it to reactos.

 

best regards

Blacky



#124 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2794 posts
  • Location:Nantes - France
  •  
    France

Posted 13 April 2014 - 09:02 AM

All right, I got ahold of an updated Windows 8.1 for testing.  The pointer files are indeed reparse points, although as you noticed they are hidden when viewing from the system with the filter driver running.  I've figured out the format of the reparse data, with the exception of a few fields that seem to be invariant.  Also, in the reparse data the WIM file is identified by an integer ID, and there's a separate undocumented ioctl (probably handled by WoF.sys) to translate the full path of a WIM file into a unique integer ID.  I will keep working on this when I have time and try to get something working.

 

 

Hi Synchronicity,

 

This is good news :) A wimboot compatible wimlib will be quiet appreciated !

 

At the system level, I must say this is quite frustrating to not being able to see which files are pointers and which one are not.

Anyway you know of to identify/enumerate these pointers?

 

At the WIM level, what does the wimboot do?

 

Last, do you think wimboot works only at the drive level or could it work too at a folder level?

For now dism seems to be hardcoded in a way it always look for "capturedir"\windows\system32\wimbootcompress.ini.

 

Regards,

Erwan



#125 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 13 April 2014 - 04:29 PM

When running the WoF driver, there might be an ioctl to detect whether a file is a WIMBoot pointer file or not.  I don't yet know what it is though.

 

WIMBoot seems to do very little at the WIM level.  The only differences I've noticed so far are (1) a new compression flag set, but it's actually just XPRESS compression with a smaller chunk size, and (2) <WIMBOOT>1</WIMBOOT> shows up in the XML data for the WIMBoot-enabled image.  However, I don't yet know what the [PrepopulateList] and [CompressionFolderList] sections of the wimbootcompress.ini file do.

 

DISM/WIMGAPI is indeed hard-coded to look for the file Windows\System32\wimbootcompress.ini when capturing a WIM in "WIMBoot mode".  But besides that, WIMBoot seems to work at a folder level.  In my implementation it probably will be possible to allow capturing without wimbootcompress.ini present.

 

Note that the registration of WIM files with the WoF driver may be global to the entire system.






1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users