Jump to content











Photo
* * * * * 4 votes

wimlib, with ImageX implementation

wim imagex winpe boot

  • Please log in to reply
365 replies to this topic

#176 misty

misty

    Gold Member

  • Developer
  • 1035 posts
  •  
    United Kingdom

Posted 27 April 2014 - 01:25 PM

@erwan/l

...I know what the problem is.  The [PrepopulateList] is only used during extraction, and for that the configuration file used is \Windows\System32\WimBootCompress.ini *within the WIM image*.  I guess I will need to change it so that when a capture configuration file is explicitly specified for a WIMBoot capture, it will be placed in the WIM image at \Windows\System32\WimBootCompress.ini, overriding the copy from the filesystem.

I did a quick test yesterday with DISM and this failed to process additions I'd made to the [PrepopulateList] - despite the edited file being in the captured wim. I need to retest this at some point as I was tired when I did the test and may have made an error myself.

Wimlib is now copying my edited --config file to \Windows\System32\wimbootcompress.ini when this flag is used during the capture process. I'll do a test later and see if manually adding my configuration file afterwards with the update command also works.

Will report back later.

Regards,

Misty

#177 misty

misty

    Gold Member

  • Developer
  • 1035 posts
  •  
    United Kingdom

Posted 27 April 2014 - 01:56 PM

  • Use the wimlib-imagex update command and delete /Windows/System32/wimbootcompress.ini, then add your amended file
I suspect that, based on a comment synchronicity made elsewhere in this thread, that the [PrepopulateList] is only used when an image is applied and not during the capture phase....


I love it when I'm right! Just tested this and it worked :thumbsup:

:cheers:

#178 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2911 posts
  • Location:Nantes - France
  •  
    France

Posted 27 April 2014 - 01:57 PM

oki, updated my wimbootcompress.ini in my wim file with the below batch.

currently applying.

lets see :)

# wimlib-imagex.exe update boot.wim --verbose < update.txt
delete '\windows\system32\wimbootcompress.ini' --force
add 'c:\wimbootcompress.ini' '\windows\system32\wimbootcompress.ini'


#179 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2911 posts
  • Location:Nantes - France
  •  
    France

Posted 27 April 2014 - 02:24 PM

 

oki, updated my wimbootcompress.ini in my wim file with the below batch.

currently applying.

lets see :)

# wimlib-imagex.exe update boot.wim --verbose < update.txt
delete '\windows\system32\wimbootcompress.ini' --force
add 'c:\wimbootcompress.ini' '\windows\system32\wimbootcompress.ini'

 

Works :)

I did not have to perform a bcdboot c:\windows /s c:\ this time !

And indeed, checking bootmgr and boot\bcd, they are not reparse points but plain files.

 

So indeed, when capturing, this is the wimbootcompress.ini on disk which is used whereas when applying this is the wimbootcompress.ini within the wim file which is used.

Ideally the prepoluated section in that file should contain bootmgr and \boot\bcd in the case where you have one unique partition (boot+system).

MS could/should have thought about that...

Luckily, it is "easy" to update the wim file afterwards with a correct wimbootcompress.ini.

 

Not only can we now apply a wimboot / win8.1u1 image from a non win8.1u1 system, but we also dont have to re apply the bootmgr afterwards :)

 

Question @Synchronicity, capturing from a WinPE 3/4 (i.e not win8.1u1) is possible as well now right?

Or do you still have a dependency there with WoF ?



#180 misty

misty

    Gold Member

  • Developer
  • 1035 posts
  •  
    United Kingdom

Posted 27 April 2014 - 02:53 PM

@synchronicity
Just tested the new build. I used WinPE 3.1 to capture my Windows 8.1 (Update 1) installation, then formatted the OS partition and then reapplied the .wim file - it works :worship:

@erwan.l
I hope this answers your question.

Regards,

Misty

#181 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 27 April 2014 - 04:17 PM

I think at this point we need to review the use of wimbootcompress.ini.

 

To me this file is needed only when capturing but I read different understandings on this point.

As a whole if I understand correctly, apart from a different compression scheme (XPRESS 4k chunks) and a wimboot (XML) tag in the wim file, the capture wim file is not much different from an "ordinary" wim file.

Thus, it seems that files inside the wim get some sort of a flag that will make a difference when restoring/applying? (i.e bootmgr, bcd, etc).

 

The only differences between a WIM captured in WIMBoot mode and one captured in the default mode are that the WIMBoot mode adds <WIMBOOT>1</WIMBOOT> to the XML data and also uses XPRESS compression with 4096 byte chunks.  I was unable to detect any per-file flags.  DISM/WIMGAPI appears to use the [PrepopulateList] section of the file \Windows\System32\WimBootCompress.ini inside the WIM when applying to determine which files will be extracted normally, not as WIMBoot pointer files.  So I made wimlib do the same thing, and yes you can simply replace this file in the WIM image instead of completely recapturing.  I will double check that DISM/WIMGAPI really has the same behavior.

 

Note that the <WIMBOOT> section and XPRESS compression are not, in fact, required for creating pointer files to the WIM.  With wimlib (but not WIMGAPI/DISM), you can take a WIM captured in the default mode (without --wimboot) and apply it with --wimboot, and it will still work.  The file \Windows\System32\WimBootCompress.ini will retain the same importance, however.  In addition, XPRESS compression with 4096 byte chunks may be better suited for backing files than LZX compression with 32768 byte chunks.

 

It would be possible to add a --config option to apply, to override \Windows\System32\WimBootCompress.ini in the WIM image.  I'm not sure this would encourage good practices, however, since if you have to use it then I think that would mean you captured a custom WIM image with the wrong configuration, and it would be better to fix the WIM image itself by updating \Windows\System32\WimBootCompress.ini.

 

Last, reparse points files cannot be dealt with without the MS WOF driver unless you deal with them at the MFT level.

 

 

Well, kind of.  At the programmatic level, you can open a reparse point with CreateFile() with FILE_FLAG_OPEN_REPARSE_POINT, then you can get, set, or delete the reparse data using DeviceIoControl().  But yes, from a practical standpoint if you have a reparse point that is not handled by any running driver, it will appear as an inaccessible file to all programs that aren't designed to deal with reparse points specifically.



#182 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 27 April 2014 - 04:23 PM

 

Question @Synchronicity, capturing from a WinPE 3/4 (i.e not win8.1u1) is possible as well now right?

Or do you still have a dependency there with WoF ?

 

There is no dependency on WoF for capturing in WIMBoot mode.  As mentioned in my previous post, there are very few differences between a "normal" WIM and one captured in WIMBoot mode.  (Actually, you can even capture a WIM in WIMBoot mode on Linux as well!  But you can only apply in WIMBoot mode on Windows.)



#183 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2911 posts
  • Location:Nantes - France
  •  
    France

Posted 27 April 2014 - 04:29 PM

Thanks for this detailed explanations.

 

I really enjoy this last Wimlib-ImageX version :)

It is fast and flexible.

 

This is really a fantastic job you have done in a few days around this new wimboot feature.

 

Regards,

Erwan



#184 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 27 April 2014 - 04:37 PM

Also as I brought up earlier, I'm looking into having the add command of wimlib-imagex update replace existing files by default, which would make updating \Windows\System32\WimBootCompress.ini easier.  I'm also considering implementing a special option to make updating that file specifically easier:

 

wimlib-imagex update install.wim 1 --wimboot-config=CustomWimBootCompress.ini

which would be short for

wimlib-imagex update install.wim 1 --command="add CustomWimBootCompress.ini \Windows\System32\WimBootCompress.ini"

(assuming that I do indeed make add replace existing files by default).

 
 


#185 misty

misty

    Gold Member

  • Developer
  • 1035 posts
  •  
    United Kingdom

Posted 27 April 2014 - 07:57 PM

It looks like it's also possible to use wimlib-imagex to create a WinPE 5.1 WimBoot - see here

@synchronicity :worship:

#186 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 28 April 2014 - 06:05 PM

I've implemented the two features mentioned in my previous post:
  • The 'add' command of 'wimlib-imagex update' will now overwrite existing nondirectory files by default.  Use '--no-replace' to get the old behavior.
  • Added '--wimboot-config=FILE' option to 'wimlib-imagex update'.  When this option is specified, the FILE is placed at /Windows/System32/WimBootCompress.ini, and no update commands are read from standard input.
Uploaded the new v1.6.3-BETA files.

  • ilko and misty like this

#187 misty

misty

    Gold Member

  • Developer
  • 1035 posts
  •  
    United Kingdom

Posted 28 April 2014 - 07:11 PM

@synchronicity
Fantastic news. I'll test the new add feature as soon as I've finished this post. Wimlib really is developing into The tool for handling .wim files. In my (not so) humble opinion I'd say it's better than the MS tools. One of the features that made me first fall in love with the .wim format was the ability to mount it so easily using imagex - sadly this process of mounting and capturing any changes appears to have gotten slower over time and the add feature in Wimlib-Imagex is far superior if only a few files need replacing.

Amazing work. And the new features were added very quickly :worship:

Thanks :thumbsup:

#188 misty

misty

    Gold Member

  • Developer
  • 1035 posts
  •  
    United Kingdom

Posted 29 April 2014 - 07:01 AM

I didn't finish my tests last night as I went to the pub instead! The new update add command appears to function very well.

My current projects (MistyPE and Mini-WinFE) have been deleting and adding individual files in the Windows directory of a WinPE build. For each file I was using something similar to the following -
wimlib-imagex.exe#$q update D:\boot.wim 1 --command="delete '\Windows\System32\Explorerframe.dll'"
wimlib-imagex.exe#$q update D:\boot.wim 1 --command="add 'D:\ExplorerFrame.dll' '\Windows\System32\Explorerframe.dll"
In a recent test, this method was taking approximately 1 minute 7 seconds to complete.

This was very slow - but I didn't understand Wimlib-ImageX well enough at the time I drafted my projects to use a file list instead. So at the weekend I played around with using a file list to delete files, and a seperate file list to inject them. This was considerably faster - taking approximately 24 seconds to complete.

This morning I've used the new update add command instead. This took approximately 18 seconds to complete. :thumbsup:

And the other benefit is that I can now just copy a root directory to the wim root - allowing me to also inject folders containing spaces in the path. This was something that I hadn't been able to easily implement before (not due to wimlib I hasten to add).

I haven't tested the new '--wimboot-config=FILE' option to 'wimlib-imagex update' yet. Will report back when I have - although I'm busy with family commitments so this will take a few days.

:worship:

Kind regards,

Misty

#189 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15551 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 29 April 2014 - 07:44 AM

I didn't finish my tests last night as I went to the pub instead! 
 

Working on the Wimbeer project ? ;)

:lol:

 

:duff:

Wonko



#190 ljycslg

ljycslg

    Newbie

  • Members
  • 26 posts
  •  
    China

Posted 29 April 2014 - 11:55 AM

about wimbootcompress.ini 

how to write folder path in [PrepopulateList]

i use

\Windows\Boot\*.*

after use wimlib apply

some file still is pointer files,some files is real file 

is this a bug?



#191 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 29 April 2014 - 07:55 PM

@ljycslg

 

I think you might be trying to match files in subdirectories of \Windows\Boot, not files in \Windows\Boot itself.  Also note that the period is being treated as significant, so your pattern doesn't match anything which doesn't have a period in its name.

 

Also, I've just made an update to the pattern matching, and now the patterns in [PrepopulateList] will match recursively.  And wimlib-imagex now prints an informational message when it extracts a file as "normal" rather than WIMBoot.  So maybe you just want to specify \Windows\Boot (after downloading the updated-yet-again v1.6.3-BETA).


Edited by synchronicity, 29 April 2014 - 07:55 PM.

  • misty likes this

#192 umeboshi

umeboshi
  • Members
  • 2 posts
  •  
    United States

Posted 29 April 2014 - 09:00 PM

I just found this project this weekend when I started to add the ability to auto install win 7 machines in a pxe netboot environment.  I used the mkwinpeimg tool to create a an iso with a startup script that basically does this

@echo Hello world.

wpeinit

net use z: \\paella\win7
z:\setup

@pause

The iso is delivered via http in the gPXE environment, so it boots faster than tftp.  I include the autounattend.xml 

file in the root of the iso image.  Things seem to work pretty well so far.  The mkwinpeimg doesn't work when using - as the output file so the iso can be written to stdout.  I will need that functionality to create the boot iso images on demand on the web server.

 

I am currently working on making the reference system, then booting back to the debian live system and capturing the image with wimcapture to an nfs export on the server.  It will probably be tomorrow before I work on installing that system to a new machine with wimlib.  My project can be found here http://umeboshi2.github.io/paella/.

Thanks for making this tool.  I have been able to achieve a fully automated, yet minimal, windows 7 install without having to execute any microsoft code!  :1st:



#193 misty

misty

    Gold Member

  • Developer
  • 1035 posts
  •  
    United Kingdom

Posted 29 April 2014 - 09:12 PM

...And wimlib-imagex now prints an informational message when it extracts a file as "normal" rather than WIMBoot....

Another :thumbsup: from me - this is a very useful feature. I hadn't realised how many files are fully extracted until testing this new feature - I may have to redirect output to a text file next time I use it as there's a possibility that any files I add to the [PrepopulateList] section may become lost in the other information.

:cheers:

#194 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 29 April 2014 - 09:38 PM

@umeboshi

Sounds good!  Actually, making a Windows PE with a start script like that (without relying on MS tools) was the original reason I started working on wimlib in the first place.  But yes, now you can make use of the NTFS-3g capture and apply modes from Linux, so you don't even need Windows at all.  (Except for WIMBoot extractions, which has been discussed for a couple pages now...!)

 

Having mkwinpeimg write the ISO image to standard output almost works; however I see that other stuff gets printed to standard output, not just the ISO image.  I'll try to change this.

 

@misty

Maybe the [PrepopulateList] matches should only be printed if --verbose is also specified?  (Actually there used to be a --verbose option, but it doesn't do anything anymore; maybe it's time to give it a new use.)



#195 ljycslg

ljycslg

    Newbie

  • Members
  • 26 posts
  •  
    China

Posted 30 April 2014 - 01:57 AM

123.png

 

add

\Windows\Boot\Fonts

to [PrepopulateList]

 

 

as you see seven .ttf extract suceess

 

msyh_boot.ttf

.......

......

wgl4_boot.ttf

 

this six ttf have not extract

 

 



#196 umeboshi

umeboshi
  • Members
  • 2 posts
  •  
    United States

Posted 01 May 2014 - 03:04 PM

@synchronicity

 

Thanks for making the stdout work!  I haven't had time use the feature yet.  I'm used wimcapture on  the debian live system to 

make a reference.wim and copy it to the nfs/samba share.  I had to rebuild the system to include parted and nfts-3g to install the wim, but I haven't had a chance to actually test this yet.  I am still not sure if I can use wimlib to make the system bootable, but what I can do is 

after applying the reference.wim, I can send a POST request to the server telling it to create a specific pxe config file based on the system

uuid that will boot a custom WinPE iso set to execute bcdboot properly (this is where the stdout becomes handy).  I'll be honest, I don't have

much experience with windows systems, although I have managed many samba servers.  The last time I used windows regularly was with windows 98.



#197 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15551 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 01 May 2014 - 04:33 PM

....I may have to redirect output to a text file next time I use it as ....

Yep :).

@synchronicity
If I may, something like an optional internal "tee" to output BOTH to the screen AND to a log file would be a nice improvement, of course with a low-low priority level.

:duff:
Wonko

#198 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 08 May 2014 - 12:02 AM

Hi
 
just want to point a matter regarding the ESD files
 
new dism version (adk's 17029 or update1's 17031) is capable of exporting an image "index" of esd file to a regular wim file
this is technically like un-compressing the esd file to the standard levels
 
but the limitation is that the wim file must be already exists, and we have to use the parameter "/compress:recovery" regardless of the target wim compression level
 
i.e.

dism /Export-Image /SourceImageFile:test.esd /SourceIndex:1 /DestinationImageFile:temp.wim /compress:recovery

i tried to do the same using wimlib-imagex, but no luck

wimlib-imagex export test.esd 1 temp.wim --compress=recovery

[ERROR] Invalid resource entry (offset overflow)
ERROR: Exiting with error code 20:
An entry in the WIM's lookup table is invalid.



#199 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 08 May 2014 - 01:20 AM

Hi Abbodi,

 

Can you temporarily replace your libwim-15.dll with libwim-15_debugonly.dll, downloaded from http://sourceforge.n...files/testing/?  Then run any command, such as 'wimlib-imagex info', on the problematic "ESD" file.  Then will dump the lookup table of the WIM file to a file "lookup_table.bin", which you should then post here or send to me.  This is metadata only --- no actual file data.  That should allow me to identify the problem.



#200 Abbodi

Abbodi

    Newbie

  • Members
  • 20 posts
  •  
    Saudi Arabia

Posted 08 May 2014 - 05:27 AM

Here you go:

http://fex.net/rest-...061042/download






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users