Jump to content











Photo
* * * * * 4 votes

wimlib, with ImageX implementation

wim imagex winpe boot

  • Please log in to reply
365 replies to this topic

#326 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 14 August 2015 - 04:51 PM

@misty

 

External backing from WIM files (i.e. "WIMBoot") is indeed deprecated in Windows 10.

 

It is deprecated in favor of "compact mode" or "System Compression".  It uses the same compression formats as WIMBoot, but the compressed data of each file is transparently stored in an alternate data stream of the file rather than in an external WIM file.

 

With Windows 10 DISM, as well as wimlib v1.8.2-BETA, you can apply any WIM image in "compact" mode.  This just compresses each extracted file using "System Compression".  It will be slower than a "WIMBoot" extraction because all data will need to be recompressed. Note that so far I've been stuck using Microsoft's API to compress the files, which means that wimlib's superior compressors don't get used.



#327 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 22 August 2015 - 06:08 PM

wimlib v1.8.2 has been released.

 

From the NEWS file:

 

 

   

         This release primarily contains various minor bug fixes and
        improvements, including:
 
                Improved handling of deep directory structures.
 
                Fixed a bug where on 32-bit systems, the library could enter an
                infinite loop if a WIM file was malformed in a specific way.
 
                Added a workaround for a case where libntfs-3g may report
                duplicate streams in an NTFS file.
 
                Windows symbolic links and junctions in mounted WIM images are
                now automatically rewritten to be valid in the mounted location.
 
                Reparse point fixes: correctly handle the "ReparseReserved"
                field, and correctly handle "empty" (data-less) reparse points.
 
                On Windows, wimlib now acquires SeManageVolumePrivilege, which
                is needed to create externally backed files using the
                "wofadk.sys" driver.
 
                Improved validation of filenames.
 
                Improved LZMS decompression speed.
 
                The configure script now honors alternate pkg-config settings.
 
                Links have been updated to point to the new website.
 
        In addition, experimental support has been added for compressing
        extracted files using System Compression on Windows 10.  This
        functionality is available through the new '--compact' option to
        'wimapply' and 'wimextract' as well as new library flags.
 

 

All releases of wimlib are available at http://wimlib.net/downloads.


  • alacran likes this

#328 alacran

alacran

    Gold Member

  • .script developer
  • 1309 posts
  •  
    Mexico

Posted 24 August 2015 - 02:39 PM

Script to transform *.wim to *.esd and *.esd to *.wim

 

If there is some one interested there is an script made by abbody1406 to make an *.esd from a *.wim and viceversa using synchronicity's wimlib v1.82 (script attached)

 

Source: http://forums.mydigi...ll=1#post903619

 

Best Regards

 

alacran

Attached Files


  • devdevadev likes this

#329 misty

misty

    Gold Member

  • Developer
  • 1035 posts
  •  
    United Kingdom

Posted 27 August 2015 - 07:33 AM

:offtopic:

@chenall


There has bug in wimboot function.(windows 7 with wof driver).Without wof driver it works

I think should not use wof drive when the operating system is not higher than windows 8.1(>=6.3)...


When wof driver not loaded it displays a warning, but it works fine



[WARNING] WOF driver is not available; updating WimOverlay.dat directly.
This is easy to reproduce, install wof driver on win7 will get this error.


Whilst the issue you reported has been resolved in the 1.8.2 release of wimlib, I'd be very interested to know how you integrated the wof.sys driver in Windows 7.

The wofadk.sys driver appears to work fine, however not everyone will want to download the ADK for Windows 10 to get it. And yes I know that it's possible to use JFX's GetWaikTools to avoid downloading the full ADK - just would be convenient to have the option of using the wof.sys driver.

Regards,

Misty

P.s. @synchronicity - apologies for the offtopic post!

#330 alacran

alacran

    Gold Member

  • .script developer
  • 1309 posts
  •  
    Mexico

Posted 27 August 2015 - 09:27 PM

If there is someone interested in adding wofadk.sys driver to 7 or 8.x in order to make it able to run wofadk service on it.

 

Donload GetWaikTools from this page:  http://www.msfn.org/...hl=getwaiktools

 

Run it and select Win10 Dism, go to ADK_6\amd64\DISM or ADK_6\x86\DISM according to your current OS, copy wofadk.sys to:  C:\Windows\System32\drivers then run the attached WofAdk.reg to register it and reboot.

After reboot open a command prompt as administrator and type net start wofadk and enter to make sure the service is running. 

 

Best Regards

 

alacran

Attached Files



#331 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 14 November 2015 - 08:00 PM

wimlib v1.8.3 has been released.

 

From the NEWS file:

 

 

Fixed a bug with libntfs-3g extraction present since v1.8.1.  Sometimes,

some Microsoft software would not correctly recognize data in the
resulting filesystem.
 
Made some small improvements to the compression algorithms:
LZX compression ratio was slightly improved.
XPRESS compression ratio and speed was slightly improved.
LZMS compression speed was slightly improved.
 
Improved handling of WIM XML data.  wimlib no longer drops unrecognized
elements when exporting images.  In addition, two API functions were
added for better access to elements in the XML document:
wimlib_get_image_property() and wimlib_set_image_property().
 
Added support for (unsafe) in-place compaction of WIM files.
 
Improved performance of image export by reusing metadata resources
instead of always rebuilding and recompressing them.
 
Improved performance of wimlib_update_image() by delaying the update to
the WIM's XML document until a write is requested.
 
On Windows, the target of an extraction may now be a reparse point
(which will be dereferenced).
 
On Windows, wimlib now correctly restores non-Microsoft reparse points.
However, this remains broken in NTFS-3g mode due to a libntfs-3g bug.
 
On Windows, wimlib now has improved performance when archiving files
from a filesystem backed by a WIM (a "WIMBoot" setup).
 
Several improvements to System Compression (compact mode) support:
 
wof.sys (or wofadk.sys) is now automatically attached to the
target volume if needed.
 
Compact-mode extractions now work correctly with wofadk.sys on
older versions of Windows.
 
For compatibility with the Windows bootloader, the requested
compression format now is overridden on certain files.
 
Other minor bugfixes.
 
 

 

All releases of wimlib are available from http://wimlib.net/downloads.



#332 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 21 December 2015 - 05:57 AM

If anyone is interested:

 

wimlib-v1.8.4-BETA2 includes experimental support for VSS snapshots.

 

Just pass the --snapshot argument to 'wimcapture' or 'wimappend' to create a WIM image from a temporary VSS snapshot of the source directory.  This works on a "live" Windows system.


  • martinb likes this

#333 martinb

martinb

    Newbie

  • Members
  • 15 posts
  •  
    Netherlands

Posted 21 December 2015 - 06:24 PM

If anyone is interested:

 

wimlib-v1.8.4-BETA2 includes experimental support for VSS snapshots.

 

Just pass the --snapshot argument to 'wimcapture' or 'wimappend' to create a WIM image from a temporary VSS snapshot of the source directory.  This works on a "live" Windows system.

 

Great!

I was using external commands to create the snapshot, and the script was getting a bit complicates on the part of deleting the old snapshots. (in case of a crash/reboot/interrupted backup etc)

I will test this in a few weeks, and will let you know if i find any problems.



#334 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 21 December 2015 - 06:45 PM

 

Great!

I was using external commands to create the snapshot, and the script was getting a bit complicates on the part of deleting the old snapshots. (in case of a crash/reboot/interrupted backup etc)

I will test this in a few weeks, and will let you know if i find any problems.

 

wimlib uses "nonpersistent" snapshots (VSS_CTX_BACKUP), which will hopefully eliminate the need to delete them manually.



#335 martinb

martinb

    Newbie

  • Members
  • 15 posts
  •  
    Netherlands

Posted 21 December 2015 - 07:38 PM

wimlib uses "nonpersistent" snapshots (VSS_CTX_BACKUP), which will hopefully eliminate the need to delete them manually.

 

Perfect !

Can't wait to test. (will be first week of January)



#336 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 01 January 2016 - 06:35 PM

wimlib-1.9.0-BETA3 (v1.8.4 is renamed to v1.9.0) adds functionality that has been requested a number of times: setting of Windows-specific XML information, such as architecture, system root, and version details.  This should make WIM images captured with wimlib compatible with WDS.

 

The behavior in the current beta is that the new information is set automatically when a new WIM image is captured.  No special option is needed.



#337 steom

steom

    Newbie

  • Members
  • 28 posts
  •  
    Italy

Posted 02 January 2016 - 11:24 AM

wimlib's 'imagex.exe' currently cannot be used as a "drop-in" replacement for Microsoft's 'imagex.exe'


when it will be? in the next beta? or it is abandoned?

#338 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15275 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 02 January 2016 - 12:14 PM

when it will be? in the next beta? or it is abandoned?

Are you serious? :unsure:

What was written was:
 
 

- wimlib's 'imagex.exe' currently cannot be used as a "drop-in" replacement for Microsoft's 'imagex.exe' because the command-line syntax is different and in some cases the supported options and subcommands differ.  It should be apparent that they are quite similar, though.

 

What makes you think that making it a "drop-in" replacement is actually "desired" or set as a "goal" for the project? :dubbio:

 

:duff:

Wonko

 

 



#339 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 02 January 2016 - 02:51 PM

when it will be? in the next beta? or it is abandoned?

 

There are no plans to do this.  It would reduce flexibility, require different syntax for the Windows and UNIX versions, and could never be made 100% compatible with all strange options and quirks of Microsoft's ImageX.

 

It is, of course, still fairly easy to transition from imagex to wimlib-imagex in most cases.

 

By the way: if you're so inclined, you could use wimlib's public API to implement a different command-line interface, if you don't like the provided one.



#340 steom

steom

    Newbie

  • Members
  • 28 posts
  •  
    Italy

Posted 02 January 2016 - 07:33 PM

What makes you think that making it a "drop-in" replacement is actually "desired" or set as a "goal" for the project? :dubbio:
 
:duff:
Wonko

Are you serious? :unsure:

The currently adverb used in 2013!
Only this!

What was written was:

wimlib's 'imagex.exe' currently cannot be used as a "drop-in" replacement for Microsoft's 'imagex.exe' because the command-line syntax is different and in some cases the supported options and subcommands differ.



#341 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15275 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 02 January 2016 - 07:55 PM

Well, after that 2013 INITIAL post, there are 14 (fourteen) pages of posts on this thread and I believe that - had you read them -  you would have gathered how the tool not only evolved, but even changed name exactly because there was a risk of confusion with the original tool since some of the options/syntax are different.

 

So the "currently" in 2013 should now be read as "initially" and later the tool clearly took a path further differencing itself from the MS implementation, as you can read on the home page:

https://wimlib.net/

 

What is wimlib?

wimlib is an open source, cross-platform library for creating, extracting, and modifying Windows Imaging (WIM) archives. WIM is a file archiving format, somewhat comparable to ZIP (and many other file archiving formats); but unlike ZIP, it allows storing various Windows-specific metadata, allows storing multiple "images" in a single archive, automatically deduplicates all file contents, and (recently) supports optional solid compression to get a better compression ratio. wimlib and its command-line frontend wimlib-imagex provide a free and cross-platform alternative to Microsoft's WIMGAPI, ImageX, and DISM.

 

 

It is an alternative, not a drop-in replacement.

 

 

:duff:

Wonko



#342 martinb

martinb

    Newbie

  • Members
  • 15 posts
  •  
    Netherlands

Posted 15 January 2016 - 11:08 PM

Perfect !

Can't wait to test. (will be first week of January)

 

I just made some tests on the --snapshot part and I'm very pleased with this option.

I was using a complex script to create the shadow image (server 2003/2008/2012/32bit/64 bit/xp/vista/win7/win10) and the most complex part was deleting the shadows.

With the --snapshot option i reduces the backupscript size with 70% , but i only made test on win7 and server 2008

 

When i compare a backup of a win7 c:\windows copy with the "old" way (based on 1.8.2 and shadowbackup with powershell) versus the 1.9.0 beta6 i get:

 

old: 75 minutes -> 15 GB compressed

new: 63 minutest -> 15 GB compressed

 

I still have to make some more tests on more OS versions and i need to test the restore, but it looks promising.

 

The only thing I'm still wondering is the long time it takes on the "scanning" part.

If the directory structure is huge (like the c:\windows on a win7 os) , it takes 20 minutes before 1 bit is written to disk.

 

Anyhow, I'm very pleased with the --snapshot option, and will post the results off restoring a full backup as soon i have some time



#343 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 16 January 2016 - 12:29 AM

@martinb

 

Great!

 

The scan to read all file metadata before file data is unavoidable based on how wimlib is currently designed.  Doing it combined would usually be a bit faster, especially on Windows, but it would be less flexible from an implementation standpoint.  I am going to see if there is any way to speed up the current method though.



#344 AnonVendetta

AnonVendetta

    Silver Member

  • Advanced user
  • 775 posts
  • Location:A new beginning.....
  • Interests:Self-development, computing

Posted 16 January 2016 - 01:09 AM

A few questions:

1. Is there any reason I should use this instead of MS's dism/imagex implementation?

2. This tool can be used to install Windows while booted into my Linux distro of choice? What about using the bcdboot command to install either legacy MBR/UEFI GPT boot files?

3. This tool is compatible with Windows 10 install.wim's?

4. Can I mount and install from already-decrypted install.esd files?

Thanks!

#345 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 16 January 2016 - 06:14 AM

1.) This has been discussed and answered extensively here and in the documentation files, and there are many obvious reasons.  See for yourself.

2.) Yes, you can apply WIM images to NTFS volumes from Linux, as explained in the documentation.  This could be used in conjunction with bcdboot.exe to set up Windows' boot files, although bcdboot.exe is, of course, a Windows program, not a Linux program.

3.) Yes, since Microsoft didn't change the WIM format for Windows 10.

4.) wimlib fully supports reading and writing solid LZMS-compressed WIM files, often called "ESD files".  Images from such files can also be mounted on Linux; however, mount performance on such files is very poor due to the solid compression.



#346 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 17 January 2016 - 06:33 PM

I've posted wimlib-1.9.0-BETA7.

This contains an optimization for directory tree scans on Windows which avoids having to open every file, which was the main bottleneck.

However, it only works on Windows 8 and later when capturing an entire NTFS volume (e.g. C:\).  It is automatically used when supported.


  • martinb likes this

#347 martinb

martinb

    Newbie

  • Members
  • 15 posts
  •  
    Netherlands

Posted 17 January 2016 - 10:00 PM

I've posted wimlib-1.9.0-BETA7.

This contains an optimization for directory tree scans on Windows which avoids having to open every file, which was the main bottleneck.

However, it only works on Windows 8 and later when capturing an entire NTFS volume (e.g. C:\).  It is automatically used when supported.

 

Sounds good, but what about server 2008 ?

So when i would test this for a backup c:\windows on win7 the "speedup" will not be supported?

 

I always wanted a good reason to upgrade from win7 to win10, so I'm afraid this will be it :)

I never knew that where was a different between win7 and win8 on the NTFS part



#348 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 17 January 2016 - 10:27 PM

The new code uses the FSCTL_QUERY_FILE_LAYOUT ioctl which was added in Windows 8 (and probably Windows Server 2012).  It allows quickly listing all files on an NTFS filesystem.

 

On older versions of Windows, or when the source directory is not the root directory of an NTFS filesystem, wimlib falls back to the regular recursive directory tree scan.


  • martinb likes this

#349 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15275 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 18 January 2016 - 09:02 AM

As a side note, maybe - just maybe - it would be possible to add to Wimlib direct NTFS $MFT listing, not unlike:

http://reboot.pro/to...y-that-is-fast/

 

:duff:

Wonko



#350 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 18 January 2016 - 04:05 PM

As a side note, maybe - just maybe - it would be possible to add to Wimlib direct NTFS $MFT listing, not unlike:

http://reboot.pro/to...y-that-is-fast/

 

 

Well that's basically what it does.  But wimlib uses a supported API rather than reading the MFT directly.






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users