WimTool

Posted 29 July 2008 - 06:38 PM

Hi community,

as some of you already know I am working on replacement for ImageX that can be freely distributed (in difference to ImageX which can be legally only obtained as part of the WAIK).

First I made good progress in understanding the WIM file format but now I am confronted with some problems regarding the used compression algorithms:

1. The compression algorithm used for fast compression is currently unknown. I documented all the details I know about it in WimTool Wiki.

Now I want to find out which algorithm is really used. My algorithm comparison method is rather simple but usually effective:

- compress LGPL.txt using a compression tool containing the algorithm to be tested
- open the output file on a hex editor
- search if there is a large binary block that is identical to the compressed file of the WIM file (the compressed LGPL.txt in test_fast.wim starts at offset 0xD0)

A large number of compression tools provided by Microsoft has to be tested and that is the part I am hoping to get some support by all of you here on boot-land. All you need can be found in this attachment:  wimtest.zip   39.36KB   1150 downloads.
Please perform the test outlined above and post the results. That would really help me knowing what algorithm it is respectively not is.

In the meanwhile I want to focus myself on finding a library that implements compression/decompression for the MS-LZX compression algorithm (used for compression level maximum).

I am counting on your help,
booty#1

• Back to top

Posted 29 July 2008 - 07:12 PM

Maybe a "stupid" remark, but it seems like 7-zip:
1) fails to open test_none.wim
2) opens allright test_maximum.wim (LZX)
3) fails to open test_fast.wim

Something that may be unrelated, but knowing how MS guys like to re-use code, it may be not:
http://www.msfn.org/...ind-t81068.html
(Search for "Diamond" on the thread)

I will try the "usual suspects", and let you know.

Things to checK:
http://www.speakeasy.../~russotto/chm/

http://www.cabextrac...g.uk/libmspack/

(this latter one should support more formats)

jaclaz

• Back to top

Posted 29 July 2008 - 07:22 PM

Maybe a "stupid" remark, but it seems like 7-zip:
1) fails to open test_none.wim
2) opens allright test_maximum.wim (LZX)
3) fails to open test_fast.wim

That is not stupid, that is simply a sad fact. 7zip's WIM implementation bases on reverse engineering and it only covers the maximum compression level (LZX algorithm), nothing else.

I will try the "usual suspects", and let you know.

Thanks, I really appreciate your help

Things to checK:
http://www.speakeasy.../~russotto/chm/
http://www.cabextrac...g.uk/libmspack/
(this latter one should support more formats)

The problem is that both are libraries not read-to-use tools. The first one is designed to be compiled under Linux (makes it not easy to port to Win32), the other one I already compiled on Win32/Visual Studio, but using it is really complicated. No easy to use interface at least not for the compression algorithm, only for the base file formats (CHM, CAB, ...). And documentation of the compression back end isn't very good.

booty#1

• Back to top

Posted 29 July 2008 - 07:44 PM

Found a reference here and on some other places

All WIM images feature one of two compression technologies - LZX or XPress. LZX gives greater compression when space restrictions are an issue, and is the option of choice when installing from fast media like a DVD.

Looking for "Xpress" compression, now.

If it is "another name" for lznt1 or lznt1+, dd for windows /FAU version) should support it:
http://gmgsystemsinc.com/fau/

I'll check and report.

jaclaz

• Back to top

Posted 30 July 2008 - 08:23 AM

This:
http://www.planet-so...p...1&lngWId=-1

could be useful (I cannot get most of it, but you should be able to)

jaclaz

• Back to top

Posted 30 July 2008 - 09:54 AM

The dd for windows (FAU edition) lznt1 and lznt1+ -comp option appear to produce files almost, but not quite, completely unlike the original test_fast.wim

I have found some reference to the effect that lznt1 compression is the algorithm used in NTFS filesystem compression (which makes sense, since thhe .wim is, more or less, a filesystem).

This article explains how to use that kind of compression for Inno Setup scripts:
http://www.vincenzo....p;printable=yes

there may be something of worth in it.

jaclaz

PS: just found a somewhat related article (in Russian):
http://www.nf-team.o...zf5/zf5_025.htm
I am attaching a google translation of it.

The output format of the dd for windows appears like the one described in the article, maybe there are DIFFERENT lznt1 algorithms.

Attached Files

•  RussianPage.txt   9.35KB   959 downloads

• Back to top

Posted 30 July 2008 - 11:01 AM

• Back to top

Posted 30 July 2008 - 11:52 AM

likeness: 0%

On further tests, the dd for windows lznt1 appears (say 70%) similar to NTFS compression on disk.

So, it seems like the lznt1 algorithm is used (if at all) in a deeply modified way in XPRESS (Wim image "fast") compression.

jaclaz

P.S.: bincompress is available also in the 6.0 version:
http://msdn.microsof...y/aa924411.aspx
See if this one installs:
http://www.microsoft...;displaylang=en

• Back to top

Posted 30 July 2008 - 02:19 PM

• Back to top

Posted 30 July 2008 - 02:48 PM

@booty

Something more that may be of use:
http://gpwiki.org/fo...opic.php?t=2623
http://gpwiki.org/fo...opic.php?t=3037

there is a link to this compression Library:
http://www.pscode.co.......78&lngWId=3

though it seems to me it's related to the same topic of the Alex Ionescu's article I previously linked to:
http://www.planet-so...p...1&lngWId=-1

More info from Alex:
http://www.alex-ionescu.com/Native.Pdf

I'll go on "stand-by", waiting for your checks on the above, will simply try some more archivers/analyzers/decompressors to see if anything works with test_fast.wim.

Would it be possible that "native" compression has changed in Vista?

Thus:
2K/XP "Native" compression = lznt1
Vista "Native" conpression=Xpress



jaclaz

• Back to top

Posted 06 August 2008 - 07:56 AM

That is not stupid, that is simply a sad fact. 7zip's WIM implementation bases on reverse engineering and it only covers the maximum compression level (LZX algorithm), nothing else.

Hi booty#1, how do you know that 7zip used reverse engineering to implement the LZX algorithm? This algorithm is fully documented in the Microsoft Cabinet Software Development Kit which is freely available.

Iwish that you finally success to create your tool! Good luck!

• Back to top

Posted 06 August 2008 - 08:05 AM

• Back to top

Posted 06 August 2008 - 08:17 AM

• Back to top

Posted 07 August 2008 - 12:40 PM

• Back to top

Posted 07 August 2008 - 03:22 PM

• Back to top

Posted 07 August 2008 - 03:50 PM

The only problem is that the mentioned document has more than 400 pages and I just cand find the right reference.

I'll check them and see if I can find anything.

jaclaz

• Back to top

Posted 07 August 2008 - 05:04 PM

Of course, you remember is MS, don't you, what is elsewhere referred to as "XPRESS" or "Fast compression" has an altogether different name in these docs, guess what?

Win2k3


The file is [MS-DRSR].pdf within Windows_Server_Protocols.zip

Section:
4.1.10.6.15 DecompressWin2k3

Page 219/474

Throughout all the document, the "right" keyword to search for is "Win2k3".

The contents are of course "Chinese" to me, but if this is the XPRESS compression, you shold be able to get how it works.



jaclaz

• Back to top

Posted 07 August 2008 - 08:38 PM

• Back to top

Posted 29 August 2008 - 03:23 PM

I had posted a request for help on the encode.ru forum (home of compressor PIMv2 and mostly frequented by people expert or interested in compression):
http://www.encode.ru/
http://encode.ru/for...read.php?p=2791

Apart from the news that Beta 4.59 of 7-zip does support XPRESS, there is a VERY interesting link to someone who checked the pseudo-code and even fixed it:
http://www.msuiche.n...ity-initiative/

There is even the code in C (interesting for you, I presume, I cannot get head or tail of C Language):
http://www.msuiche.n...es/xpress.c.txt

It seems like the XPRESS compression is used as well in hybernation file:
http://sandman.msuiche.net/
http://msuiche.net/c..._profit-0.6.pdf

And also in some other things, like LDAP, Outlook and Exchange.

Good work, let me know if you need anything else I may help you with.

jaclaz

• Back to top

Posted 05 December 2009 - 07:31 AM

• Back to top

Posted 05 December 2009 - 08:40 AM

• Back to top