Jump to content











Photo
- - - - -

WimTool


  • Please log in to reply
20 replies to this topic

#1 booty#1

booty#1

    Frequent Member

  • .script developer
  • 285 posts
  • Location:Near Frankfurt
  •  
    Germany

Posted 29 July 2008 - 06:38 PM

Hi community,

as some of you already know I am working on replacement for ImageX that can be freely distributed (in difference to ImageX which can be legally only obtained as part of the WAIK).

First I made good progress in understanding the WIM file format but now I am confronted with some problems regarding the used compression algorithms:

1. The compression algorithm used for fast compression is currently unknown. I documented all the details I know about it in WimTool Wiki.

Now I want to find out which algorithm is really used. My algorithm comparison method is rather simple but usually effective:

- compress LGPL.txt using a compression tool containing the algorithm to be tested
- open the output file on a hex editor
- search if there is a large binary block that is identical to the compressed file of the WIM file (the compressed LGPL.txt in test_fast.wim starts at offset 0xD0)

A large number of compression tools provided by Microsoft has to be tested and that is the part I am hoping to get some support by all of you here on boot-land. All you need can be found in this attachment: Attached File  wimtest.zip   39.36KB   965 downloads.
Please perform the test outlined above and post the results. That would really help me knowing what algorithm it is respectively not is.

In the meanwhile I want to focus myself on finding a library that implements compression/decompression for the MS-LZX compression algorithm (used for compression level maximum).

I am counting on your help,
booty#1

#2 was_jaclaz

was_jaclaz

    Finder

  • Advanced user
  • 7100 posts
  • Location:Gone in the mist
  •  
    Italy

Posted 29 July 2008 - 07:12 PM

Maybe a "stupid" remark, but it seems like 7-zip:
1) fails to open test_none.wim
2) opens allright test_maximum.wim (LZX)
3) fails to open test_fast.wim

Something that may be unrelated, but knowing how MS guys like to re-use code, it may be not:
http://www.msfn.org/...ind-t81068.html
(Search for "Diamond" on the thread)

I will try the "usual suspects", and let you know.

Things to checK:
http://www.speakeasy.../~russotto/chm/

http://www.cabextrac...g.uk/libmspack/

(this latter one should support more formats) :cheers:

jaclaz

#3 booty#1

booty#1

    Frequent Member

  • .script developer
  • 285 posts
  • Location:Near Frankfurt
  •  
    Germany

Posted 29 July 2008 - 07:22 PM

Maybe a "stupid" remark, but it seems like 7-zip:
1) fails to open test_none.wim
2) opens allright test_maximum.wim (LZX)
3) fails to open test_fast.wim


That is not stupid, that is simply a sad fact. 7zip's WIM implementation bases on reverse engineering and it only covers the maximum compression level (LZX algorithm), nothing else.

I will try the "usual suspects", and let you know.


Thanks, I really appreciate your help :cheers:

Things to checK:
http://www.speakeasy.../~russotto/chm/
http://www.cabextrac...g.uk/libmspack/
(this latter one should support more formats) :cheers:


The problem is that both are libraries not read-to-use tools. The first one is designed to be compiled under Linux (makes it not easy to port to Win32), the other one I already compiled on Win32/Visual Studio, but using it is really complicated. No easy to use interface at least not for the compression algorithm, only for the base file formats (CHM, CAB, ...). And documentation of the compression back end isn't very good.

booty#1

#4 was_jaclaz

was_jaclaz

    Finder

  • Advanced user
  • 7100 posts
  • Location:Gone in the mist
  •  
    Italy

Posted 29 July 2008 - 07:44 PM

Found a reference here and on some other places

All WIM images feature one of two compression technologies - LZX or XPress. LZX gives greater compression when space restrictions are an issue, and is the option of choice when installing from fast media like a DVD.


Looking for "Xpress" compression, now. :cheers:

If it is "another name" for lznt1 or lznt1+, dd for windows /FAU version) should support it:
http://gmgsystemsinc.com/fau/

I'll check and report.

jaclaz

#5 was_jaclaz

was_jaclaz

    Finder

  • Advanced user
  • 7100 posts
  • Location:Gone in the mist
  •  
    Italy

Posted 30 July 2008 - 08:23 AM

This:
http://www.planet-so...p...1&lngWId=-1

could be useful (I cannot get most of it, but you should be able to) :cheers:

jaclaz

#6 was_jaclaz

was_jaclaz

    Finder

  • Advanced user
  • 7100 posts
  • Location:Gone in the mist
  •  
    Italy

Posted 30 July 2008 - 09:54 AM

The dd for windows (FAU edition) lznt1 and lznt1+ -comp option appear to produce files almost, but not quite, completely unlike the original test_fast.wim :cheers:

I have found some reference to the effect that lznt1 compression is the algorithm used in NTFS filesystem compression (which makes sense, since thhe .wim is, more or less, a filesystem).

This article explains how to use that kind of compression for Inno Setup scripts:
http://www.vincenzo....p;printable=yes

there may be something of worth in it.

jaclaz

PS: just found a somewhat related article (in Russian):
http://www.nf-team.o...zf5/zf5_025.htm
I am attaching a google translation of it.

The output format of the dd for windows appears like the one described in the article, maybe there are DIFFERENT lznt1 algorithms.

Attached Files



#7 booty#1

booty#1

    Frequent Member

  • .script developer
  • 285 posts
  • Location:Near Frankfurt
  •  
    Germany

Posted 30 July 2008 - 11:01 AM

The dd for windows (FAU edition) lznt1 and lznt1+ -comp option appear to produce files almost, but not quite, completely unlike the original test_fast.wim :cheers:

Can you express the alikeness in percent?

PS: just found a somewhat related article (in Russian):
http://www.nf-team.o...zf5/zf5_025.htm
I am attaching a google translation of it.
The output format of the dd for windows appears like the one described in the article, maybe there are DIFFERENT lznt1 algorithms.

Thanks, the article was quite informative.

I found a reference to a small compression util that uses (according to the documentation) "XPRESS" as well:
BinCompress from the "Platform Builder for Microsoft Windows CE 5.0". From what I have read some C-header files and source code (partial) should be included as well. This platform builder package may be very helpful but it is not freely available. I only found an evaluation edition that fails on installation.

booty#1

#8 was_jaclaz

was_jaclaz

    Finder

  • Advanced user
  • 7100 posts
  • Location:Gone in the mist
  •  
    Italy

Posted 30 July 2008 - 11:52 AM

likeness: 0% :cheers:

On further tests, the dd for windows lznt1 appears (say 70%) similar to NTFS compression on disk.

So, it seems like the lznt1 algorithm is used (if at all) in a deeply modified way in XPRESS (Wim image "fast") compression.

jaclaz

P.S.: bincompress is available also in the 6.0 version:
http://msdn.microsof...y/aa924411.aspx
See if this one installs:
http://www.microsoft...;displaylang=en

#9 was_jaclaz

was_jaclaz

    Finder

  • Advanced user
  • 7100 posts
  • Location:Gone in the mist
  •  
    Italy

Posted 30 July 2008 - 02:19 PM

It seems that there was some activity on the matter when Longhorn came out, and before Ximage was made available, some header info that may be (or nay not) be useful:
http://board.iexbeta...showtopic=51516

jaclaz

#10 was_jaclaz

was_jaclaz

    Finder

  • Advanced user
  • 7100 posts
  • Location:Gone in the mist
  •  
    Italy

Posted 30 July 2008 - 02:48 PM

@booty

Something more that may be of use:
http://gpwiki.org/fo...opic.php?t=2623
http://gpwiki.org/fo...opic.php?t=3037

there is a link to this compression Library:
http://www.pscode.co.......78&lngWId=3

though it seems to me it's related to the same topic of the Alex Ionescu's :cheers: article I previously linked to:
http://www.planet-so...p...1&lngWId=-1

More info from Alex:
http://www.alex-ionescu.com/Native.Pdf

I'll go on "stand-by", waiting for your checks on the above, will simply try some more archivers/analyzers/decompressors to see if anything works with test_fast.wim.

Would it be possible that "native" compression has changed in Vista?

Thus:
2K/XP "Native" compression = lznt1
Vista "Native" conpression=Xpress

:cheers:

jaclaz

#11 smiley

smiley

    Silver Member

  • .script developer
  • 905 posts
  •  
    Greece

Posted 06 August 2008 - 07:56 AM

That is not stupid, that is simply a sad fact. 7zip's WIM implementation bases on reverse engineering and it only covers the maximum compression level (LZX algorithm), nothing else.



Hi booty#1, how do you know that 7zip used reverse engineering to implement the LZX algorithm? This algorithm is fully documented in the Microsoft Cabinet Software Development Kit which is freely available.

Iwish that you finally success to create your tool! Good luck! :cheers:

#12 booty#1

booty#1

    Frequent Member

  • .script developer
  • 285 posts
  • Location:Near Frankfurt
  •  
    Germany

Posted 06 August 2008 - 08:05 AM

Hi booty#1, how do you know that 7zip used reverse engineering to implement the LZX algorithm?

I did never say that. I assumed that they used reverse engineering for the wim format, because their implementation is so incomplete. If they would have had the documentation I assume that 7zip would be able to read at least uncompressed WIM files.

This algorithm is fully documented in the Microsoft Cabinet Software Development Kit which is freely available.

I know, I documented that already on the WimTool wiki page linked to in the first post of this thread.

booty#1

#13 smiley

smiley

    Silver Member

  • .script developer
  • 905 posts
  •  
    Greece

Posted 06 August 2008 - 08:17 AM

Hi booty#1, htanks for the quick answer.

I have found in microsoft's site some information about the RtlCompressBuffer here: http://msdn.microsof...y/bb981783.aspx

#14 pecd.net

pecd.net

    Silver Member

  • .script developer
  • 947 posts
  •  
    Germany

Posted 07 August 2008 - 12:40 PM

maybe you can just ask this guy...its worth a try:-)

http://technet.micro...e/cc160878.aspx

#15 booty#1

booty#1

    Frequent Member

  • .script developer
  • 285 posts
  • Location:Near Frankfurt
  •  
    Germany

Posted 07 August 2008 - 03:22 PM

I found another interesting information in the Samba wiki:

There's also another compression algorithm "XPRESS" used in windows 2003. The algorithm is described in the MS-DRSR document of the Microsoft WSPP documentation (See http://msdn2.microso.../cc203213.aspx).

The only problem is that the mentioned document has more than 400 pages and I just cand find the right reference.

booty#1

#16 was_jaclaz

was_jaclaz

    Finder

  • Advanced user
  • 7100 posts
  • Location:Gone in the mist
  •  
    Italy

Posted 07 August 2008 - 03:50 PM

The only problem is that the mentioned document has more than 400 pages and I just cand find the right reference.


I'll check them and see if I can find anything. :cheers:

jaclaz

#17 was_jaclaz

was_jaclaz

    Finder

  • Advanced user
  • 7100 posts
  • Location:Gone in the mist
  •  
    Italy

Posted 07 August 2008 - 05:04 PM

Of course, you remember is MS, don't you, what is elsewhere referred to as "XPRESS" or "Fast compression" has an altogether different name in these docs, guess what?

Win2k3 :cheers:


The file is [MS-DRSR].pdf within Windows_Server_Protocols.zip

Section:
4.1.10.6.15 DecompressWin2k3

Page 219/474

Throughout all the document, the "right" keyword to search for is "Win2k3". :cheers:

The contents are of course "Chinese" to me, but if this is the XPRESS compression, you shold be able to get how it works. :cheers:

:cheers:

jaclaz

#18 booty#1

booty#1

    Frequent Member

  • .script developer
  • 285 posts
  • Location:Near Frankfurt
  •  
    Germany

Posted 07 August 2008 - 08:38 PM

The file is [MS-DRSR].pdf within Windows_Server_Protocols.zip
Section:
4.1.10.6.15 DecompressWin2k3
Page 219/474

The contents are of course "Chinese" to me, but if this is the XPRESS compression, you shold be able to get how it works. :cheers:

Thanks, jaclaz. At least we should now have the XPRESS decompression. I will try to translate the pseudo code from the document into real programming code the next days.
Funny that the pseudo code is very similar to Pascal/Delphi code...

booty#1

#19 was_jaclaz

was_jaclaz

    Finder

  • Advanced user
  • 7100 posts
  • Location:Gone in the mist
  •  
    Italy

Posted 29 August 2008 - 03:23 PM

I had posted a request for help on the encode.ru forum (home of compressor PIMv2 and mostly frequented by people expert or interested in compression):
http://www.encode.ru/
http://encode.ru/for...read.php?p=2791

Apart from the news that Beta 4.59 of 7-zip does support XPRESS, there is a VERY interesting link to someone who checked the pseudo-code and even fixed it:
http://www.msuiche.n...ity-initiative/

There is even the code in C (interesting for you, I presume, I cannot get head or tail of C Language):
http://www.msuiche.n...es/xpress.c.txt

It seems like the XPRESS compression is used as well in hybernation file:
http://sandman.msuiche.net/
http://msuiche.net/c..._profit-0.6.pdf

And also in some other things, like LDAP, Outlook and Exchange.

Good work, let me know if you need anything else I may help you with. :whistling:

jaclaz

#20 agni

agni

    Frequent Member

  • Tutorial Writer
  • 270 posts
  • Location:Bengaluru (Bangalore)
  •  
    India

Posted 05 December 2009 - 07:31 AM

Maybe somebody can contact Nuhi.
He used only wimfltr.sys to manipulate install.wim and boot.wim
WAIK did not actually have to be installed on the computer.

#21 was_jaclaz

was_jaclaz

    Finder

  • Advanced user
  • 7100 posts
  • Location:Gone in the mist
  •  
    Italy

Posted 05 December 2009 - 08:40 AM

Maybe somebody can contact Nuhi.
He used only wimfltr.sys to manipulate install.wim and boot.wim
WAIK did not actually have to be installed on the computer.


Agni,
this thread has been staling since 1 1/2 year.

Right now we have a way out, thanks to psc:
http://www.boot-land...?showtopic=9765

jaclaz




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users