365 replies to this topic

[synchronicity]

I found out why wimlib wasn't working on Windows XP: it calls _wcserror_s(), which Microsoft incorrectly documents as being available on Windows 95 and later, but it is actually not available in MSVCRT.DLL on Windows XP.  (Perhaps Visual Studio statically links in its own implementation.)  However, I will also look into extracting files with special attributes, such as hidden files, before making a new release.  I am also considering using a new method to make capturing WIM images even faster.

[Wonko the Sane]

I found out why wimlib wasn't working on Windows XP: it calls _wcserror_s(), which Microsoft incorrectly documents as being available on Windows 95 and later, but it is actually not available in MSVCRT.DLL on Windows XP.  (Perhaps Visual Studio statically links in its own implementation.) 

Good, then it's only a matter of finding the correct version of redistributable files, MSVCRT.DLL is one of the redistributable files in vcredist.

Examples:

http://www.microsoft...ls.aspx?id=5555

http://www.microsoft...s.aspx?id=30679

 

Wonko

[synchronicity]

Good, then it's only a matter of finding the correct version of redistributable files, MSVCRT.DLL is one of the redistributable files in vcredist.

 

Yes, but I don't want to distribute a copy of the C library just for 1 function that I can replace in a few lines of code.

[synchronicity]

If anyone is still following this:  I have released wimlib 1.3.3, which fixes most of the remaining issues brought up here and has various other improvements.  From the NEWS:

• Capturing a WIM image should now be significantly faster in most cases due to improved use of the operating system's cache and avoiding reading files twice whenever possible.
• The Windows build now works on Windows XP.
• The Windows build now supports capturing and restoring hidden, compressed, sparse, and encrypted files.
• The Windows build now supports capturing and applying WIM images from filesystems other than NTFS (with some reduced functionality).
• The Windows build now extracts short names correctly.
• Added support for "reparse-point" fixups (i.e. fixing up of symbolic links).  See docs for --rpfix and --norpfix flags of `wimlib-imagex capture' and `wimlib-imagex apply'.
• The performance of splitting and joining WIMs should be slightly improved.
• The LZX and XPRESS compression and decompression functions are now exported from the library.

​I have done quite a bit of testing, including capturing and restoring Windows 7 and Windows 8 images, so I hope that the software is very usable now.  Of course, any suggestions or bug reports are appreciated.

 

As suggested earlier I may post the source code and Windows binaries on http://reboot.pro/files/, but I'm hoping to hear from a few people who have tried this new version first.

 

The Windows binaries and source code for wimlib 1.3.3, including the wimlib-imagex program, can be downloaded from https://sourceforge....s/wimlib/files/.

[Wonko the Sane]

VERY good news

 

Wonko

[ilko]

Thanks for wimlib, quite useful!

A few test on XP SP3 and wimlib 1.3.3- using boot.wim from Windows Defender Offline since it has some issues in the wim structure as reported by 7-Zip 9.30- "Some files have incorrect reference count".
Applying the image to "test" folder- OK.
Capturing an image from "test" folder- OK. Resulting wim file of greater size compared to the original. Did I miss an option? Using

 

wimlib-imagex capture test test.wim --compress=maximum --boot

Original file boot.wim 171,909 KB, resulting test.wim file 182,077KB.

 

An issue when used optimize command with --recompress option:

 

E:\___tmp\wimlib>wimlib-imagex optimize boot.wim --recompress
[WARNING] Could not find function FindFirstStreamW() in Kernel32.dll!
[WARNING] Capturing alternate data streams will not be supported.
"boot.wim" original size: 171908 KiB
Writing LZX-compressed data using 2 threads
429 MiB of 429 MiB (uncompressed) written (100% done)
[ERROR] MoveFileEx(): Can't rename "E:\___tmp\wimlib\boot.wimudc7Spzwq" to "E:\___tmp\wimlib\boot.wim"
[ERROR] Win32 error: Access is denied.

[ERROR] Failed to rename `E:\___tmp\wimlib\boot.wimudc7Spzwq' to `E:\___tmp\wimlib\boot.wim': Unknown error
[WARNING] Failed to remove `E:\__': No such file or directory
ERROR: Exiting with error code 46:
       Could not rename a file.

boot.wimudc7Spzwq exists- 172,643KB, boot.wim has not special attributes- hidden or system and is not in use, antivirus is turned off.

 

 

Omitting --recompress option works fine and does decrease file size:

 

 

E:\___tmp\wimlib>wimlib-imagex optimize boot.wim
[WARNING] Could not find function FindFirstStreamW() in Kernel32.dll!
[WARNING] Capturing alternate data streams will not be supported.
"boot.wim" original size: 171908 KiB
Writing LZX-compressed data using 1 thread
429 MiB of 429 MiB (uncompressed) written (100% done)
"boot.wim" optimized size: 167412 KiB
Space saved: 4496 KiB

 

Again, thanks for your work.

[synchronicity]

Capturing an image from "test" folder- OK. Resulting wim file of greater size compared to the original.

 

Sorry, wimlib's LZX compression is still several percent worse than Microsoft's compressor.  If you know anything about compression, here are the details:  A small fraction of the difference is accounted for by wimlib not implementing block splitting within WIM chunks, where each block gets its own Huffman codes.  Actually, I did implement this in an experimental branch, but it only helped a tiny bit and made compression a lot slower.  The rest of the difference is apparently from suboptimal choices of LZ77 matches.  In particular, I may need to design an algorithm that chooses when to use the very short length-2 matches allowed by LZX.  Note: this is different from zlib compression used in ZIP and GZIP files, which only allows matches of 3 bytes or longer.  That was probably more detail than you wanted; just understand that implementing these compression algorithms is difficult.  I really would like to improve the LZX compression further, though.

 

 

 

 

boot.wimudc7Spzwq exists- 172,643KB, boot.wim has not special attributes- hidden or system and is not in use, antivirus is turned off.

 

 

Omitting --recompress option works fine and does decrease file size:

 

I don't know why this is.  All I can tell is that Windows did not let my code rename the WIM file for some reason.  Also the path name in  the link "[WARNING] Failed to remove `E:\__':" was wrong for some reason, even though based on the code this is not possible.  I don't know if this would even help, but what filesystem were you running this on?  Also, is this problem reproducible (for example, with a smaller WIM file)?

 

 

 

A few test on XP SP3- using boot.wim from Windows Defender Offline since it has some issues in the wim structure as reported by 7-Zip 9.30- "Some files have incorrect reference count".

 

I have not looked at that particular WIM (which I assume is a variant of Windows PE) but I do know that Microsoft had released a Windows 8 WIM with incorrect reference counts for some streams in the WIM.  This is a bug in their software, and I have made wimlib work around this by recalculating all the reference counts when it deletes an image or file.  (If this is not done, it is not possible to safely perform those operations.)  `wimlib-imagex optimize' will also fix all the reference counts as a side effect as that command simply sends the WIM unmodified through wimlib's WIM writing code, which will write the reference counts correctly.

 

Thanks for testing!

[synchronicity]

Oh, actually I have an idea about the rename problem:  I don't check that the WIM file is actually writable by the current user at any point before doing the rename, so such a permission problem will not be reported until the rename.  I will add in an earlier check as well, since it's obviously best to report this problem immediately, although this still doesn't explain why the problem went away when you removed the --recompress option.

[ilko]

I don't know why this is.  All I can tell is that Windows did not let my code rename the WIM file for some reason.  Also the path name in  the link "[WARNING] Failed to remove `E:\__':" was wrong for some reason, even though based on the code this is not possible.  I don't know if this would even help, but what filesystem were you running this on?

 

Moved to another folder, same wim file, NTFS formatted partition:

 

 

E:\tmp>wimlib-imagex optimize boot.wim --recompress
[WARNING] Could not find function FindFirstStreamW() in Kernel32.dll!
[WARNING] Capturing alternate data streams will not be supported.
"boot.wim" original size: 171908 KiB
Writing LZX-compressed data using 2 threads
429 MiB of 429 MiB (uncompressed) written (100% done)
[ERROR] MoveFileEx(): Can't rename "E:\tmp\boot.wimFFJu4PmEK" to "E:\tmp\boot.wim"
[ERROR] Win32 error: Access is denied.

[ERROR] Failed to rename `E:\tmp\boot.wimFFJu4PmEK' to `E:\tmp\boot.wim': Unknown error
[WARNING] Failed to remove `E:\tm': No such file or directory
ERROR: Exiting with error code 46:
       Could not rename a file.

 

[WARNING] Failed to remove `E:\tm': No such file or directory

`E:\tm' ?

 

Without --recompress it works fine just as above.

 

Using smallish boot.wim from Vista, --recompress fails with the same error:

 

E:\tmp>wimlib-imagex optimize boot1.wim --recompress
[WARNING] Could not find function FindFirstStreamW() in Kernel32.dll!
[WARNING] Capturing alternate data streams will not be supported.
"boot1.wim" original size: 132045 KiB
Writing LZX-compressed data using 2 threads
358 MiB of 358 MiB (uncompressed) written (100% done)
[ERROR] MoveFileEx(): Can't rename "E:\tmp\boot1.wimg0dW5qCfc" to "E:\tmp\boot1.wim"
[ERROR] Win32 error: Access is denied.

[ERROR] Failed to rename `E:\tmp\boot1.wimg0dW5qCfc' to `E:\tmp\boot1.wim': Unknown error
[WARNING] Failed to remove `E:\tm': No such file or directory
ERROR: Exiting with error code 46:
       Could not rename a file.

This time, without --recompress, failed too, the same way, 3 attempts in a row:

 

E:\tmp>wimlib-imagex optimize boot1.wim
[WARNING] Could not find function FindFirstStreamW() in Kernel32.dll!
[WARNING] Capturing alternate data streams will not be supported.
"boot1.wim" original size: 132045 KiB
Writing LZX-compressed data using 2 threads
358 MiB of 358 MiB (uncompressed) written (100% done)
[ERROR] MoveFileEx(): Can't rename "E:\tmp\boot1.wimzPdoleVeL" to "E:\tmp\boot1.wim"
[ERROR] Win32 error: Access is denied.

[ERROR] Failed to rename `E:\tmp\boot1.wimzPdoleVeL' to `E:\tmp\boot1.wim': Unknown error
[WARNING] Failed to remove `E:\tm': No such file or directory
ERROR: Exiting with error code 46:
       Could not rename a file.

 

 

Immediately tried the first boot.wim from Windows Defender Offline without --recompress option and it worked fine, then again the new boot1.wim from Vista and it again failed.

Hope you can figure out what's happening, just let me know if there is anything I could help with.

 

 

I have not looked at that particular WIM (which I assume is a variant of Windows PE) but I do know that Microsoft had released a Windows 8 WIM with incorrect reference counts for some streams in the WIM.  This is a bug in their software, and I have made wimlib work around this by recalculating all the reference counts when it deletes an image or file.  (If this is not done, it is not possible to safely perform those operations.)  `wimlib-imagex optimize' will also fix all the reference counts as a side effect as that command simply sends the WIM unmodified through wimlib's WIM writing code, which will write the reference counts correctly.

Very helpful, thanks.

 

 

Sorry, wimlib's LZX compression is still several percent worse than Microsoft's compressor.

No worries, I just got the impression from the first posts to expect slight decrease in wim size, not a big deal at all.

By the way, why optimize reduced original wim file, but apply->capture the same file decreased its size?

 

Do you have any plans for mounting with RW access option in Windows?

 

Added:

Did the same tests under Windows 7 x64, SP1- same results with both files.

[vigipirate]

Hello Thanks for your work
but I really do not understand anything I have it downloaded but I do not know what to do with possibly a short tutorial or capture your video is used with the other software I do not know how to thank you for your help?

[synchronicity]

Immediately tried the first boot.wim from Windows Defender Offline without --recompress option and it worked fine, then again the new boot1.wim from Vista and it again failed.

Hope you can figure out what's happening, just let me know if there is anything I could help with.

 

I don't think --recompress or the version of Windows makes a difference here.  As I mentioned this may be caused by your user account simply not having permission to delete the WIM file.  So I expect that if you were to try to do this with 'del', then it would not work.  I still don't know why the path got truncated.  I will read my code very carefully when I have a chance.

 

 

 

By the way, why optimize reduced original wim file, but apply->capture the same file decreased its size?

 

If you use --optimize without --recompress it will not recompress the data.  It will only remove "holes" in the WIM left by appending.  So, it has a good chance of reducing the size if the WIM has holes.  But as I mentioned before, if you change MS-compressed LZX data to wimlib-compressed LZX data (either by optimize with --recompress, or simply capture and apply) you can expect the size to increase by a few percent.

 

 

 

Do you have any plans for mounting with RW access option in Windows?

 

No.

 

 

from vigipirate:

Hello Thanks for your work
but I really do not understand anything I have it downloaded but I do not know what to do with possibly a short tutorial or capture your video is used with the other software I do not know how to thank you for your help?

 

wimlib-imagex is similar to ImageX, written by Microsoft, so any documentation they have will be fairly applicable.  Also I have provided a lot of documentation, including the README, README.WINDOWS, and man pages (in the Windows distribution: provided as text files in the 'doc' directory).  They are not translated into other languages, sorry.  So you need to figure out which `wimlib-imagex' subcommand (such as "apply") that accomplishes what you want, then read the docs to figure out how to run it.  Also if you just run the program incorrectly then it prints out the usage info.

[ilko]

I don't think --recompress or the version of Windows makes a difference here.  As I mentioned this may be caused by your user account simply not having permission to delete the WIM file.  So I expect that if you were to try to do this with 'del', then it would not work.  I still don't know why the path got truncated.  I will read my code very carefully when I have a chance.

That shouldn't be the case- admin accounts in both cases, UAC turned off on Win7. Can delete, rename, whatever the wim file from explorer and the same command prompt.

 

If you use --optimize without --recompress it will not recompress the data.  It will only remove "holes" in the WIM left by appending.  So, it has a good chance of reducing the size if the WIM has holes.  But as I mentioned before, if you change MS-compressed LZX data to wimlib-compressed LZX data (either by optimize with --recompress, or simply capture and apply) you can expect the size to increase by a few percent.

Got it, thanks.

[Wonko the Sane]

That shouldn't be the case- admin accounts in both cases, UAC turned off on Win7. Can delete, rename, whatever the wim file from explorer and the same command prompt.

Just for the sake of the experiment, can you re-try on a FAT32 partition (provided that the file is smaller than 4 Gb of course)?

 

Wonko

[synchronicity]

Thanks, but I don't think there's need for more testing on this specific bug.  I think the real problem is that in certain cases (at least when --recompress is specified), an additional handle will be opened to the WIM, which I then failed to close before doing the rename.  I will fix this in the next release.  This bug is only present in the Windows version, due to the different semantics of open files compared with UNIX.  This bug may also affect appending an image to a WIM.

[synchronicity]

By the way, as a substitute for mounting on Windows, I have planned possible support for adding, renaming, and deleting files in a WIM image, as well as extracting specific files or directories from a WIM.  These capabilities would be exposed in wimlib-imagex through new subcommands `wimlib-imagex update' and `wimlib-imagex extract'.  But I have not written the code yet, so let me know if there is interest in having these features.

[Wonko the Sane]

By the way, as a substitute for mounting on Windows, I have planned possible support for adding, renaming, and deleting files in a WIM image, as well as extracting specific files or directories from a WIM.  These capabilities would be exposed in wimlib-imagex through new subcommands `wimlib-imagex update' and `wimlib-imagex extract'.  But I have not written the code yet, so let me know if there is interest in having these features.

Sure people are interested in these.

At least myself.

The whole point is that .wim format/approach is basically available only to Vista and later users, whilst it is in itself an "independent format" with a number of "attractive" features.

 

People playing/having fun with PE 1.x's have the issue of needing - one way or the other - MS files/library for which they may be not holding a valid licence and/or create some tool/script whatever needing those non-redistributable files.

 

7-zip already solved a number of problems (i.e. accessing the files inside a .wim) as said earler:

http://reboot.pro/to...ation/?p=169800

but if we could not need those files at all it would be better IMHO.

 

Wonko

[vigipirate]

Hello Thanks for your work
but I really do not understand anything I have it downloaded but I do not know what to do with possibly a short tutorial or capture your video is used with the other software I do not know how to thank you for your help?

[ilko]

By the way, as a substitute for mounting on Windows, I have planned possible support for adding, renaming, and deleting files in a WIM image, as well as extracting specific files or directories from a WIM.  These capabilities would be exposed in wimlib-imagex through new subcommands `wimlib-imagex update' and `wimlib-imagex extract'.  But I have not written the code yet, so let me know if there is interest in having these features.

 

 

Sure people are interested in these.

At least myself.

 

Me too, and most likely a lot more people once they find out about wimlib.

[cdob]

I have planned possible support for adding, renaming, and deleting files in a WIM image, as well as extracting specific files or directories from a WIM.

I like to add files to boot.wim.
Mounting and unmounting the file take some time so far. A direct adding is highly welcome.

[synchronicity]

Mounting and unmounting the file take some time so far.

 

Only with Microsoft's imagex on Windows.  If you use mine on Linux it is much faster, and it uses the FUSE (Filesystem in Userspace) API which works very well.  This is partly why I'm afraid to even attempt implementing mounting on Windows, as MS probably does all kinds of hacks to get the thing to work properly as a "Filesystem Filter" driver.  Nor can you mount anything on filesystems other than NTFS, which makes no sense.

 

But yes, since there's interest in update functionality, I will try to get the basic update and extract functionality I mentioned implemented when I have time, which should be easy compared to mount support.

[Wonko the Sane]

Only with Microsoft's imagex on Windows.  If you use mine on Linux it is much faster, and it uses the FUSE (Filesystem in Userspace) API which works very well.  This is partly why I'm afraid to even attempt implementing mounting on Windows, as MS probably does all kinds of hacks to get the thing to work properly as a "Filesystem Filter" driver.  Nor can you mount anything on filesystems other than NTFS, which makes no sense.

 

But yes, since there's interest in update functionality, I will try to get the basic update and extract functionality I mentioned implemented when I have time, which should be easy compared to mount support.

I will throw these on the table (and swiftly hide my hand behind my back ):

http://bazislib.sysprogs.org/

http://dokan-dev.net/en/

 

 

Wonko

[vigipirate]

Hello Thanks for your work
but I really do not understand anything I have it downloaded but I do not know what to do with possibly a short tutorial or capture your video is used with the other software I do not know how to thank you for your help?

[synchronicity]

Hi,

 

wimlib 1.4.0 has been released.  From the NEWS:

 

• Added new "extract" and "update" subcommands to wimlib-imagex, along with associated APIs in the library.  These commands are intended mainly for Windows use but can be used on UNIX as well.
• Many documentation improvements.
• Fixed a bug in the Windows build where relative symbolic links were not captured in reparse-point fixup mode.
• Fixed a bug in the Windows build where file handles were left open to the WIM file, causing `wimlib_imagex optimize' to fail in some cases.
• Fixed a bug in the Windows build of wimlib-imagex where globbing split-WIM parts could cause the program to crash.
• Fixed a bug where the creation time of WIM images would be shown instead  of the last modification time.
• With the Windows build it is now possible to restore a WIM containing symbolic links as a non-Administrator; however you will receive warnings about not being able to extract the symbolic links.

So, I implemented the "update" and "extract" functionality that people thought would be useful.  I also looked into supporting mounting WIM files on Windows some more, but it just does not seem realistic at this point.  Furthermore, I was prototyping some improvements to LZX compression but was only able to get about a 1% improvement with making compression several times slower, so it was not good enough to release.

[erwan.l]

Hi Synchonicity,

 

Beautiful work.

 

Tested it with success.

 

 

wimlib-imagex.exe extract u:install.wim 3 /Windows/Boot/PCAT/bootmgr

[WARNING] Re-assigning inode numbers due to inode inconsistencies

Extracting "Windows/Boot/PCAT/bootmgr" from image 3 ("Windows 7 PROFESSIONAL") in

"U:\dvd\Windows 7 SP1 X86\install.wim" to "./bootmgr"

Extracting files: 0 MiB of 0 MiB (100%) done

Done extracting files.

 

Side notes :

-filename is case sensitive : not something users are used to on a windows platform.

-not sure i read it right : 0 MiB of 0 MiB (100%) done . Looks strange to me. may be switch to kb if file < 1mb?

 

Regards,

Erwan

 

[synchronicity]

-filename is case sensitive : not something users are used to on a windows platform.

 

Good catch!  Yes, I agree that on Windows, case-insensitive paths would be expected.  This will be easy to change, although there are a couple points to consider (for example, what happens if a WIM image contains files or directories with names differing only in case--- maybe print a warning message and arbitrarily choose one version...?)

 

There also is the question of whether the paths should be case-insensitive on UNIX as well.

 

 

-not sure i read it right : 0 MiB of 0 MiB (100%) done . Looks strange to me. may be switch to kb if file < 1mb?

 

Yes, the code is simply rounding down to the nearest megabyte, but you extracted less than a megabyte of data.  I agree that the units should be changed to kilobytes for small extractions.

 

Thanks for testing!