Jump to content











Photo
* * * * - 3 votes

wimlib, with ImageX implementation

wim imagex winpe boot

  • Please log in to reply
293 replies to this topic

#1 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 130 posts
  •  
    United States

Posted 21 March 2013 - 06:19 PM

Hi,

 

I have been developing wimlib, which is WIM library similar to Microsoft's WIMGAPI.  It comes with an ImageX tool similar to Microsoft's imagex.exe.  Previously, wimlib has only been available on UNIX (mainly Linux).  However, I have just released version 1.3.0, which comes with support for Windows.

 

wimlib is free and open source software (GPLv3+) and does not depend on Microsoft's wimgapi.dll, unlike some other tools that have been released.  You can simply download my ZIP file, which is under 2 MB, and have an ImageX implementation ready to use on Windows Vista, 7, or 8.

 

wimlib for Windows comes with an ImageX implementation that supports the following subcommands, which are similar to but not always exactly the same as the subcommands to Microsoft's ImageX:

 

imagex append - Append a directory tree to a WIM file as a new image

imagex apply - Apply a WIM image to a directory or drive

imagex capture - Create a new WIM file from a directory

imagex delete - Delete an image from a WIM image

imagex dir - List files contained in a WIM image

imagex export - Export an image from one WIM file to another

imagex info - Display information about a WIM image, or change image names/descriptions

imagex join - Join the parts of a split WIM into the original standalone WIM

imagex optimize - Remove wasted space in a WIM

imagex split - Split a WIM into parts

 

Additional notes:

 

- XPRESS and LZX compression are fully supported.  The XPRESS compressor is better than Microsoft's, while the LZX compressor is almost as good as Microsoft's.  Compression is done with multiple threads by default, so this should make it much faster to create WIM files.

 

- Capturing and restoring alternate data streams, security descriptors, reparse points, and file attributes is supported and on Windows is done by default.  In principle you should be able to back up and restore an entire Windows operating system, although I have not yet tried this with the Windows version.  Creating Windows PE images should work even if this doesn't, though.

 

- Mounting WIM images is supported on UNIX but not on Windows.  This is also the only way to modify the files contained in a WIM image, so this means that you cannot add or remove files from a WIM image with the Windows version.  (I realize this may be a significant limitation.)

 

- I think you have to run wimlib's imagex.exe as an Administrator because it requests permission to read files' SACLs (System Access Control List), which requires the SE_SECURITY_NAME privilege, which only the administrator account has.  I may be able to remove this restriction if there is interest.

 

- The ZIP file containing the Windows binaries comes with the UNIX man pages for imagex in the 'doc' subdirectory, converted to plain text.

 

- wimlib's 'imagex.exe' currently cannot be used as a "drop-in" replacement for Microsoft's 'imagex.exe' because the command-line syntax is different and in some cases the supported options and subcommands differ.  It should be apparent that they are quite similar, though.

 

WARNING:  The Windows version of wimlib and imagex (and even the UNIX version, to some extent) is experimental.  You use it at your own risk.  If you find bugs or have any suggestions, please let me know, but I am NOT responsible if something goes wrong.

 

The download link for the Windows binaries, which should work on both 32 and 64-bit versions of Windows Vista, 7, and 8, is:

 

http://downloads.sou...ws-i686-bin.zip

 

Note: The ZIP file also contains precompiled DLLs for libiconv, libxml2, and winpthreads.  These are also free and open source libraries, and you can easily obtain their source code if desired.

 

Note:  I have not released 64-bit binaries yet, although I can do this if there is interest.

 

The main SourceForge page for wimlib is http://sourceforge.net/projects/wimlib (go there if you want the source code).

 

In the (perhaps unlikely) event that you would like to write your own program that uses my API, the documentation for wimlib's API is at http://wimlib.sourceforge.net.


Edited by synchronicity, 21 March 2013 - 06:21 PM.

  • erwan.l and misty like this

#2 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 10,958 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 21 March 2013 - 07:19 PM

It seems like a nice piece of work :thumbsup: and very good news :).

 

Of course, as soon as you let us know about it, you will get a few questions and comments :whistling:

 

Questions:

Any reason why XP (and 2K) are seemingly not supported?

Even if the compression is differently implemented are the .wim's made with your tool still accessible thorugh the MS original tools and viceversa, right?

 

Comments:

IMHO it is VERY, VERY wrong :ph34r: to have this tool executable named EXACTLY as the similar IMAGEX.EXE by MS, if the syntax (and output) are EVEN SLIGHTLY different.

Noone (in his right mind) has EVER used the original IMAGEX.EXE on command line but exclusively through a batch or other form of script, so there is the GREATEST RISK that having two tools identically named on a system may create - by mistake - pure havoc.

Possibly renaming the exexutable to (say) UMAGEX.EXE or WMAGEX.EXE would not be a bad idea.  :dubbio:

 

:cheers:

Wonko



#3 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 130 posts
  •  
    United States

Posted 21 March 2013 - 07:31 PM

It seems like a nice piece of work :thumbsup: and very good news :).

 

Of course, as soon as you let us know about it, you will get a few questions and comments :whistling:

 

Questions:

Any reason why XP (and 2K) are seemingly not supported?

Even if the compression is differently implemented are the .wim's made with your tool still accessible thorugh the MS original tools and viceversa, right?

 

Comments:

IMHO it is VERY, VERY wrong :ph34r: to have this tool executable named EXACTLY as the similar IMAGEX.EXE by MS, if the syntax (and output) are EVEN SLIGHTLY different.

Noone (in his right mind) has EVER used the original IMAGEX.EXE on command line but exclusively through a batch or other form of script, so there is the GREATEST RISK that having two tools identically named on a system may create - by mistake - pure havoc.

Possibly renaming the exexutable to (say) UMAGEX.EXE or WMAGEX.EXE would not be a bad idea.  :dubbio:

 

:cheers:

Wonko

 

Thanks for the comments.  It should be possible to support Windows XP and 2000 if there was demand for it.  However, I said Windows Vista and later because there is at least one Win32 function that wimlib calls that is documented as first being available in Windows Vista (namely, FindFirstStreamW, which is used to read the alternate data streams of files).

 

To the extent that I've tested it, the WIMs made with wimlib are fully compatible with those made with Microsoft's tools, and vice versa.  I would, of course, be interested to hear about any incompatibilities.

 

I agree that imagex.exe should be renamed to avoid any confusion.  I guess I was just used to developing this on UNIX, where you can't use Microsoft's version anyway.  I am considering using "wimlib-imagex.exe" so that the program's origin is clear.



#4 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 10,958 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 21 March 2013 - 07:55 PM

Thanks for the comments.  It should be possible to support Windows XP and 2000 if there was demand for it.  However, I said Windows Vista and later because there is at least one Win32 function that wimlib calls that is documented as first being available in Windows Vista (namely, FindFirstStreamW, which is used to read the alternate data streams of files).

I don't know, 2K may be a "queer" request, but XP should be IMHO *somehow* done.

 

The MS original tools like IMAGEX.EXE and WIMGAPI.DLL all work on XP (tested), see also:

http://reboot.pro/to...ndling-wo-waik/

http://reboot.pro/to...wimtoolbatches/

 

To the extent that I've tested it, the WIMs made with wimlib are fully compatible with those made with Microsoft's tools, and vice versa.  I would, of course, be interested to hear about any incompatibilities.

Good :).

JFYI, the original IMAGEX.EXE, see here:

http://www.911cd.net...showtopic=19355

was never the "right" way to image a System, and expecially a Vista :ph34r: or later one because it had more than one issue with hardlinks and the structure of a "standard" Vista ph34r: install.

And, additionally, it was NOT licensed for this use.

If your "version" manages to be reliable in such use it would be a great thing.

If I were you I wouldn't care much about the "mounting" feature, which all in all is available to people running a MS OS through the wimftr.sys, and 7-zip is OK to access wim's read-only, I find a higher priority to have compatibility with XP as a number of possible users oft this thingy may want it on a PE 1.x.

 

I agree that imagex.exe should be renamed to avoid any confusion.  I guess I was just used to developing this on UNIX, where you can't use Microsoft's version anyway.  I am considering using "wimlib-imagex.exe" so that the program's origin is clear.

Perfect! :fine:

If you could additionally "compress" that into a 8.3 name, you would do a favour to a lot of old dinosaurs.  WLimageX.exe doesn't sound that bad :unsure:.

 

The idea of having a "different" tool named the same on two different OS's is not however a good idea, so you might want to rename it on Linux too, the pains of Unix vs. Microsoft compress.exe are re-known (among the dinosaurs ;)).

 

:cheers:

Wonko



#5 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 130 posts
  •  
    United States

Posted 21 March 2013 - 08:12 PM

Hi,

 

I do not even have a copy of Windows XP anymore, but you're welcome to try to run my program on Windows XP, and if it doesn't work, I can try to identify the incompatibilities.

 

I have written wimlib to support hard links and reparse points, and I have used to capture and restore an entire Windows 7 installation using the libntfs-3g support on Linux.  The Windows version relies on different code to capture and restore files, however, so I can't guarantee it will work at this point--- please remember that the Windows build is still highly experimental.  Another thing to note is that I can't guarantee that special file attributes such as encryption and compression are restored properly.  I mean, I specify the exact file attributes from the WIM when extracting files, but having become somewhat familiar with the eccentricities of the Win32 API, there probably are special functions that need to be called to restore certain attributes.

 

I will consider renaming the UNIX version to "wimlib-imagex" to keep the naming consistent.  I do not really like the idea of naming it something cryptic like WMAGEX.EXE, although perhaps people on this forum are used to stuff like that.  Keep in mind that it's possible to simply rename the executable file if you wish.



#6 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 10,958 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 21 March 2013 - 08:30 PM

Hi,

 

I do not even have a copy of Windows XP anymore, but you're welcome to try to run my program on Windows XP, and if it doesn't work, I can try to identify the incompatibilities.

Will do. :)

 

I will consider renaming the UNIX version to "wimlib-imagex" to keep the naming consistent.  I do not really like the idea of naming it something cryptic like WMAGEX.EXE, although perhaps people on this forum are used to stuff like that.  Keep in mind that it's possible to simply rename the executable file if you wish.

Wimlib-imagex is fine on MS OS also :thumbsup:, but please don't tell me about cryptic things coming from *nix :frusty:, an explicit name is nothing compared to the complexity of names AND parameters when typing in a command prompt.  :whistling:

 

:cheers:

Wonko



#7 Vortex

Vortex

    Frequent Member

  • Advanced user
  • 229 posts

Posted 21 March 2013 - 08:36 PM

Hi synchronicity,

 

Running wimlib-imagex.exe without any parameter, I got the following message displayed in a message box :

 

wimlib-imagex.exe - cannot not find entry point

 

Exception Processing Message c0000139 Parameters 16c6b8 172178 75b4bf7c 75b4b7c

 

OS : Win XP SP3

 

Later, any chance to support XP?



#8 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 130 posts
  •  
    United States

Posted 22 March 2013 - 06:41 AM

Well I am unable to edit my original post for some reason, but I have made a few changes and have updated the library (including the ZIP file with Windows binaries) to version 1.3.1:

 

* The program previously known as "imagex" is now installed as "wimlib-imagex" by default.

* Fixed a couple bugs with capturing and restoring alternate data streams.

* There also is now an option to skip applying or capturing security descriptors, which may be helpful when running as a non-administrator.

* FindFirstStreamW() and FindNextStreamW() are now loaded dynamically, so if those were the only things preventing the library from working on Windows XP (possible, but I wouldn't count on it), it should work now.

 

I would appreciate any further testing by anyone interested.



#9 Vortex

Vortex

    Frequent Member

  • Advanced user
  • 229 posts

Posted 22 March 2013 - 10:07 AM

Hi synchronicity,

 

Thanks for the update. Testing the new release on XP Sp3, I receive the same error message with different parameter numbers.

 

Exception Processing Message c0000139 Parameters 1732d8...



#10 panreyes

panreyes

    Member

  • Members
  • 52 posts
  •  
    Spain

Posted 22 March 2013 - 10:43 AM

This tool is really impressive :|

 

I've used the optimize option and it made my XP wim 150MB smaller, from 1900MB to 1750MB!

 

Little question. Why does this not work?

 

> wimlib-imagex.exe optimize xp.wim --recompress --compress=maximum
wimlib-imagex.exe: unknown option -- compress=maximum


#11 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 130 posts
  •  
    United States

Posted 22 March 2013 - 02:09 PM

Thanks for the update. Testing the new release on XP Sp3, I receive the same error message with different parameter numbers.

 

Exception Processing Message c0000139 Parameters 1732d8...

 

Thanks for testing, but the error message is of no help and I don't know what went wrong.  Does someone else want to try it on Windows XP and see if they get a different error message?  Maybe something about a certain function being missing from a DLL?

 

 

 

 

 

Little question. Why does this not work?

 

> wimlib-imagex.exe optimize xp.wim --recompress --compress=maximum

 

Good question!  Apparently I accidentally put in the help for the 'optimize' subcommand that the --compress option is supported, but I never implemented it.  As a workaround, you should be able to get the desired behavior by running:

 

 

wimlib-imagex.exe export xp.wim all xp-optimized.wim --compress=maximum

where xp-optimized.wim can be any non-existent file.  Then just rename xp-optimized.wim to the original if it's smaller and you notice no problems with it.

 

EDIT:  You can also pass --recompress to 'wimlib-imagex export' if you would like to force your "maximum"-compressed WIM to be recompressed, although specifically in the case of the "maximum" compression do not expect wimlib to do a better job than Microsoft's compressor.


Edited by synchronicity, 22 March 2013 - 02:15 PM.


#12 panreyes

panreyes

    Member

  • Members
  • 52 posts
  •  
    Spain

Posted 22 March 2013 - 02:18 PM

wimlib-imagex.exe export xp.wim all xp-optimized.wim --compress=maximum

where xp-optimized.wim can be any non-existent file.  Then just rename xp-optimized.wim to the original if it's smaller and you notice no problems with it.

 

EDIT:  You can also pass --recompress to 'wimlib-imagex export' if you would like to force your "maximum"-compressed WIM to be recompressed, although specifically in the case of the "maximum" compression do not expect wimlib to do a better job than Microsoft's compressor.

 

Thanks :)
 

Optimization made my WIM 150MB smaller, but recompressing made it grow 40MB XD



#13 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 130 posts
  •  
    United States

Posted 22 March 2013 - 02:25 PM

Optimization made my WIM 150MB smaller, but recompressing made it grow 40MB XD

 

Yeah, the LZX compression is something I've wanted to improve on for quite a while now.  Currently it's only a couple percent worse than Microsoft's, though.  It's funny that "optimizing" made such a big difference compared to the difference in compression ratios--- this is basically due to the fact that some space is wasted every time you append an image or mount a WIM file read-write.  And Microsoft does not even fix this in some (or maybe all...) of the WIMs that they distribute.



#14 panreyes

panreyes

    Member

  • Members
  • 52 posts
  •  
    Spain

Posted 22 March 2013 - 03:15 PM

Hi again!

 

This happened right now optimizing a big file which contains all editions of XP, Vista, 7 and 8:

 

 
wimlib-imagex optimize d:\all.wim
`d:\all.wim' original size: 20319698 KiB
[ERROR] Failed to convert UTF-16LE string to multibyte string!
[ERROR] This may be because the UTF-16LE data could not be represented in your locale's character encoding.
 
-------- [More info]
 
C:\temp\wim>c:\wimlib\wimlib-imagex optimize d:\all.wim
`d:\all.wim' original size: 20319698 KiB
[ERROR] Failed to convert UTF-16LE string to multibyte string!
[ERROR] This may be because the UTF-16LE data could not be represented in your locale's character encoding.
Writing LZX-compressed data using 8 threads
33793 MiB of 33793 MiB (uncompressed) written (100% done)
[ERROR] Failed to convert UTF-16LE string to multibyte string!
[ERROR] This may be because the UTF-16LE data could not be represented in your locale's character encoding.
[ERROR] Failed to write the WIM file `d:\all.wimP3D1G26cg'
ERROR: Exiting with error code 56:
       A Unicode string could not be represented in the current locale's encoding
 

----- Even more info:

 

I'm learning even more today. I'm trying with PowerShell ISE, which comes with Unicode support...

 

----- Last edit, I swear!

 

No luck with PowerShell ISE either :\ Same error.



#15 Nuno Brito

Nuno Brito

    Platinum Member

  • Team Reboot
  • 10,207 posts
  • Location:boot.wim
  • Interests:I'm just a quiet simple person with a very quiet simple life living one day at a time..
  •  
    European Union

Posted 22 March 2013 - 03:48 PM

Hi synchronicity, welcome to our community.

 

This topic works good as a development point but for the future I would recommend that you use the download portal at http://reboot.pro/files/ to ensure that the majority of our visitors can find your binaries when browsing for tools there.

 

I mention this because within some time the activity on this topic will decrease and the topic falls away from the front page, however the download portal is specifically intended for hosting tools from the community developers.

 

:cheers:



#16 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 130 posts
  •  
    United States

Posted 22 March 2013 - 04:17 PM

panreyes, the Unicode support is something I have been struggling with on the Windows build.  On UNIX, usually you use UTF-8 and everything just magically works (from the programmer's point of view).  But on Windows, Microsoft forces developers to use UTF16-LE, which requires significant changes to application code.  Currently I have only partially done this (as I am not going to drop UNIX support just because of design flaws in Windows) and still rely on converting some UTF16-LE strings to the locale-dependent "multibyte" encoding.  On Windows this typically will be one of the "Windows code pages" that supports fewer than 256 different characters.  If you can find some way to set your locale to UTF-8 (I tried doing this by running 'chcp 65001', but it had no effect...) then you should have no problems; otherwise you'll have to wait until I'm able to find the best way to change the code to support UTF16-LE.

 

Nuno Brito, thanks for the suggestion.  I may post my files there when the Windows support has been further improved.



#17 panreyes

panreyes

    Member

  • Members
  • 52 posts
  •  
    Spain

Posted 22 March 2013 - 06:18 PM

Another error when applying :\
 
wimlib-imagex.exe apply xp.wim 3 c:\
Applying image 3 (XPPROVLK) from `y:\wim\xp.wim' to directory `c:\'
[ERROR] Failed to create directory "c:\"
[ERROR] Win32 error: Acceso denegado.
 
[ERROR] Error extracting c:\
ERROR: Exiting with error code 31:
       Failed to create a directory.


#18 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 130 posts
  •  
    United States

Posted 22 March 2013 - 06:31 PM

All right, looks like CreateDirectoryW() will give ERROR_ACCESS_DENIED rather than ERROR_ALREADY_EXISTS if run on the root directory of a volume, even if running as an administrator.  This is not documented by Microsoft.  I will fix this in the next version.

 

I am considering solving the Unicode issue by conditionally representing strings in UTF-8 or UTF-16LE depending on the platform.



#19 panreyes

panreyes

    Member

  • Members
  • 52 posts
  •  
    Spain

Posted 22 March 2013 - 07:46 PM

Good luck with it and congratulations for this great piece of software! :)



#20 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 130 posts
  •  
    United States

Posted 27 March 2013 - 05:50 AM

All right, I have released version 1.3.2.  Strings are now represented in UTF-16LE on Windows or UTF-8 on UNIX, so there shouldn't be any problems with character encodings (but probably there's something I messed up still).  I also added options to specify whether you want to capture security descriptors not at all, exactly as is, or (the default) only when access is granted by the system.  This should make it possible to run the program as a non-administrator.  I also fixed a problem with the ordering of images within WIM files.  And the access denied problem on the root directory should be fixed as well.

 

I still have no guarantee that wimlib will work on Windows XP, as I do not have a copy of Windows XP to test, and I do not know what I need to change to make wimlib compatible with Windows XP.

 

I still am unable to edit my original post for some reason; perhaps that's an intentional feature of these forums.

 

Any testing would be appreciated.


Edited by synchronicity, 27 March 2013 - 06:08 AM.


#21 panreyes

panreyes

    Member

  • Members
  • 52 posts
  •  
    Spain

Posted 27 March 2013 - 10:02 AM

Hi again,

 

I've just tried again to optimize my almighty all.wim and this is what happened:

 

 
>wimlib-imagex.exe optimize all.wim
"all.wim" original size: 20319698 KiB
Writing LZX-compressed data using 8 threads
48831 MiB of 48831 MiB (uncompressed) written (100% done)
"all.wim" optimized size: 19642106 KiB
Space saved: 677591 KiB
 
Seems to be ok. I hope it didn't break anything :)
 
------ UPDATE
I also tried to apply a Windows 7 image and it worked flawlessly :)
One detail I've seen: Some files that are supposed to be hidden, are not: like "$Recycle.bin", "Documents and settings" or "Recovery"


#22 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 10,958 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 27 March 2013 - 11:57 AM

I still have no guarantee that wimlib will work on Windows XP, as I do not have a copy of Windows XP to test, and I do not know what I need to change to make wimlib compatible with Windows XP.

 

You may use (or peruse) the Windows_XP_IE6.exe here ;):

http://www.microsoft...s.aspx?id=11575

(of course only if you have the time/will)

 

:cheers:

Wonko



#23 Vortex

Vortex

    Frequent Member

  • Advanced user
  • 229 posts

Posted 27 March 2013 - 02:23 PM

Trying to build the tool from the source code :

 

./configure

 

configure: error: Cannot find libntfs-3g version 2011-4-12
        or later. Without libntfs-3g, wimlib cannot include support for
        capturing and applying WIMs on NTFS filesystems while preserving
        NTFS-specific data such as security descriptors and alternate data
        streams.  You should either install libntfs-3g, or configure with
        --without-ntfs-3g to disable support for these NTFS-specific
        features.
[root@mars wimlib-1.3.2]# rpm -qa | grep ntfs
ntfs-3g-2011.4.12-5.el6.i686
ntfsprogs-2011.4.12-5.el6.i686
 
I already installed ntfs-3g and ntfsprogs but configuration step complains that libntfs-3g version 2011-4-12 cannot be found.
 
OS  : CentOS 6.3


#24 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 130 posts
  •  
    United States

Posted 27 March 2013 - 02:35 PM

panreyes-  Great!  (I would, however, recommend that you make a back up copy of your data before overwriting it with *any* experimental software...).  About the hidden files issue (when applying an image with wimlib-imagex, I assume)--- there may be some special API calls I need to make to restore certain file attributes, so I will look into that.

 

Wonko the Sane-  I may look into that when I have a chance.

 

Vortex-  Perhaps the ntfs-3g headers are in a separate package?  Maybe ntfs-3g-devel?  You also can of course configure --without-ntfs-3g, although I considered that to be an important feature (on UNIX), so I made it assume you want it by default.



#25 Vortex

Vortex

    Frequent Member

  • Advanced user
  • 229 posts

Posted 27 March 2013 - 02:49 PM

Hi synchronicity,

 

Installing ntfs-3g-devel solved the problem, thanks. I wish to try the NTFS feature on Linux.