Jump to content











Photo
* * * * * 4 votes

wimlib, with ImageX implementation

wim imagex winpe boot

  • Please log in to reply
365 replies to this topic

#126 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1920 posts
  • Location:Nantes - France
  •  
    France

Posted 13 April 2014 - 04:54 PM

About the WoF driver at the entire system, it may be so :

Dism /Get-WIMBootEntry /Path:C:\

DISM.exe /Update-WIMBootEntry /Path:C:\ /DataSourceID:0 /ImageFile:R:\Install.wim

These 2 commands are a bit obscure to me for now but they seem to indicate a strong link between the system and a WIM file.

The DISM web page has been updated less that 10 days ago to reflect latest wimboot changes.

 

Now this is developper talking : i have been interfacing wimgapi in one of my software (CloneDisk).

If during your investigatings, you happen to stumble upon this flag (not documented yet) : WIM_FLAG_WIM_BOOT, I would gladly use its value :)



#127 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 13 April 2014 - 05:03 PM

Yes, the Get-WIMBootEntry and Update-WIMBootEntry DISM commands have to do with that registry of WIM files for WIMBoot.

 

WIM_FLAG_WIM_BOOT has value 0x2000.  It's in the latest wimgapi.h.  You should be able to use wimlib instead once I add support though, haha.

 

There also are new functions in WIMGAPI that aren't declared in the header:

 

- WIMGetWIMBootEntries()

- WIMGetWIMBootWIMPath()

- WIMUpdateWIMBootEntry()

 

Those are probably used to implement the DISM commands I just mentioned.  They themselves are probably implemented with ioctls handled by the WoF driver; I'll probably need to replicate this in wimlib.



#128 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 17 April 2014 - 05:41 AM

I had time to look into this a bit more tonight.  Turns out, "WoF" actually stands for "Windows Overlay File System Filter", and it was just added in Windows 8.1.  It's apparently designed to allow data sources other than WIM files, although only WIM seems to have been implemented now.  And it looks like the ioctls I was trying to figure out are, in fact, documented, although like any MS documentation it is of questionable quality (already found one mistake).  Most of the docs are linked to from http://msdn.microsof...7(v=vs.85).aspx.  It looks like the WIMBoot "pointer files" should be created with the help of the FSCTL_ADD_OVERLAY and FSCTL_SET_EXTERNAL_BACKING ioctls.  And you asked about determining whether a file is backed by a WIM:  FSCTL_GET_EXTERNAL_BACKING should work for that.  Or FSCTL_ENUM_EXTERNAL_BACKING to enumerate all files on a volume that are backed externally (at least according to the documentation --- whether it actually works properly is another question, haha).



#129 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 18 April 2014 - 07:00 AM

All right, I've released an experimental version of wimlib that supports creating WIM files as "WIMBoot compatible", and also creating "pointer" files when extracting (Windows only).  In wimlib-imagex, the option is --wimboot to the capture, append, apply, or extract commands.

 

I decided not to be as strict as the MS implementation.  wimlib-imagex doesn't require that the file Windows\System32\WimBootCompress.ini exists, but it does use the ExclusionList section from it if available.  It also is, in fact, possible to take a completely ordinary WIM file (captured without --wimboot) and apply it with --wimboot, although the compression format and chunk size will typically be less optimized for fast access (usually, LZX chunks of size 32768 rather than XPRESS chunks of size 4096).

 

You can download the experimental v1.6.3-BETA version from https://sourceforge..../files/testing/.

 

It's not tested very much yet, of course.  Also, I might look into writing a standalone program to analyze a directory for files backed by a WIM image and report statistics about disk usage.

 



#130 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 20 April 2014 - 05:51 AM

I've written the standalone program I mentioned in the previous post and uploaded it to

 https://sourceforge....ze.zip/download (wimboot-analyze.zip).  As usual it takes a lot of code to actually write a reliable Windows program that interacts with the filesystem!  I learned a few more things about namespaces in Windows NT...



#131 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1920 posts
  • Location:Nantes - France
  •  
    France

Posted 21 April 2014 - 09:03 AM

Hi Synchronicity,

 

Thats excellent news :)

 

I will play with the --wimboot parameter and wimboot-analyze today and will report back.

 

My plan is to capture my win 8.1 installation with wimcapture.cmd c:\ install.wim --wimboot and to apply it back with wimapply.cmd install.wim c:\ --wimboot.

 

If that works, then no need anymore for a windows 8.1 update 1 dism.

 

Regards,

Erwan



#132 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1920 posts
  • Location:Nantes - France
  •  
    France

Posted 21 April 2014 - 09:06 AM

Side note, that may be me not running wimboot-analyse properly but for now the only result I got was a long list of :

 

WARNING: "C:\a_file_in_use.xxx": Can't open file (Le processus ne peut pas accÚder au fichier car ce fichier est utilisÚ par un au
tre processus).
 
Translated : the process cannot access this file since it is already in use by another process.
 
I run the process from a command line with "run as admin".
 
/Erwan


#133 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1920 posts
  • Location:Nantes - France
  •  
    France

Posted 21 April 2014 - 10:04 AM

while capturing I got :

 

WARNING:ignoring  [compressionexclusionlist] section of capture config file

WARNING:unknown capture config file section [prepopulatelist]

WARNING:unknown capture config file section [compressionfolderlist]

 

/Erwan



#134 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1920 posts
  • Location:Nantes - France
  •  
    France

Posted 21 April 2014 - 10:23 AM

Hi Synchronicity,

 

Thats excellent news :)

 

I will play with the --wimboot parameter and wimboot-analyze today and will report back.

 

My plan is to capture my win 8.1 installation with wimcapture.cmd c:\ install.wim --wimboot and to apply it back with wimapply.cmd install.wim c:\ --wimboot.

 

If that works, then no need anymore for a windows 8.1 update 1 dism.

 

Regards,

Erwan

 

wimcapture went fine (thus some warnings, see here).

 

But when triggering wimapply I get the below error.

I am running wimlib from a WinPE 4.0 (built from MS ADK).

 

Edit1 : tryied from a winpe built upon a windows 8.1 update 1 and got the same error.

Edit2 : using the wim file generated by wimblib, the following command works : 

dism /apply-image /imagefile:e:\install.Wim /index:1 /applydir:c:\ /wimboot

X:\extra\wimlib163>wimapply.cmd e:\install.wim c:\ --wimboot
[ERROR] Failed to add overlay source "e:\install.wim" to volume "\\.\c:": Unknow
n error
ERROR: Exiting with error code 75:
       Failed to set WIMBoot pointer data.


#135 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 23 April 2014 - 05:56 PM

Hi, thanks for testing!

 

The problems with running wimboot-analyze on an in-use drive are largely unavoidable because it is seemingly impossible to determine external backing information from a file without opening it.  And on Windows you can't open a file for which no access is permitted due to locks taken by other programs. But, I will change the program so that:  (1) some information is acquired from directory listings rather than from the files themselves, and (2) when a file can't be opened due to a sharing violation, the program doesn't print a warning for that individual file but rather waits until the end to print a warning that summarizes what the program was unable to do.

 

The warnings about the capture config file are simply telling you about parts of the file that were ignored.  I don't believe any of those sections are particularly significant, but the warnings should remain in place in case any of the sections is important to the user.

 

The failure of wimapply to add the overlay source was caused by the WOF filter not running on the target volume.  I hadn't caught this problem as I had not tested on a brand new volume.  I've written a fix for this.

 

I've updated the 1.6.3-BETA wimlib files with the fix for the last issue.  You can test applying again (but capturing won't be any different).  I will post back when I have an updated version of wimboot-analyze.



#136 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1920 posts
  • Location:Nantes - France
  •  
    France

Posted 23 April 2014 - 08:58 PM

Tested : works :)

 

Install.wim is 7.9 gig.

After applying with wimboot, 234 mb are occupied on my C drive.

 

 

EDIT1 : spoke too fast... after reboot, winload.exe is missing ... 0xc000007b error.

 

Note that winload is specified in a section which is ignored by wimlib.

 

[PrepopulateList]
*winload.*
*winresume.*
wof.sys
\Windows\System32\Config\SYSTEM

 

EDIT2:

-restored with wimlib (boot not ok) : winload.exe size is 1.44mb and 858kb size on disk (right click file properties)

-restored with wimgapi (boot ok) : winload.exe size is 1.44mb and 1.44mb size on disk (right click file properties)

 

Could it be that wimload needs to be fully restored to disk and not just "pointed" at the wimlib? or is about (ntfs?) compression on disk ?

 

/Erwan

Attached Thumbnails

  • screenshot.4.png


#137 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 23 April 2014 - 09:41 PM

Yes, I bet some files cannot actually be restored as WIMBoot files.  Makes sense, since those are involved in booting, and the filter driver presumably won't be running yet!  I will look into what the PrepopulateList section means exactly, and if there is any special information that should be set in the WIM file that I might have missed.

 

I've updated wimboot-analyze to be quiet about sharing violations until the end and make a couple other tweaks.



#138 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 24 April 2014 - 06:39 PM

As far as I can tell, the PrepopulateList simply indicates which paths cannot be extracted as WIMBoot pointer files, and the file \Windows\System32\WimBootCompress.ini is expected to be read directly from the WIM image and parsed.  I've implemented this and updated the v1.6.3-BETA files, so feel free to test this again if you have time.



#139 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1920 posts
  • Location:Nantes - France
  •  
    France

Posted 24 April 2014 - 07:28 PM

Now works :)

 

Bravo !

 

No need anymore for a windows 8.1 update 1 source.

 

EDIT : discard the last statement above, you do need windows 8.1 update 1.



#140 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 24 April 2014 - 07:31 PM

Hmm are you sure about that?  I don't think the code will work when the WOF driver isn't running, and that was seemingly added in Windows 8.1 Update 1...



#141 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1920 posts
  • Location:Nantes - France
  •  
    France

Posted 24 April 2014 - 07:34 PM

oki you are right : let me re do the test from a windows 7 / winpe.

 

post #139 updated !

 

I thought I had read somewhere that the WoF was available since Windows 8.1 but that wimboot DISM command line actually came later with Update1



#142 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1920 posts
  • Location:Nantes - France
  •  
    France

Posted 24 April 2014 - 07:57 PM

Next question then : anyway to kill that dependency to the WoF driver to generate these overlay files? :)

 

At the file system level, I guess this is a specific flag in the MFT?

 

/Erwan



#143 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 24 April 2014 - 09:43 PM

The pointer files themselves are just reparse points.  Theoretically they could be created without the use of WOF (and as I mentioned earlier I know their format), although there's a chance that Windows doesn't actually allow applications to create arbitrary reparse points.

 

I just dumped the contents of a volume after a simple WIMBoot application, and it looks like the list of WIM archives registered for the volume is stored in a binary file "\System Volume Information\WimOverlay.dat".  More binary data to reverse engineer, I guess!



#144 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 13639 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 25 April 2014 - 10:16 AM

I see that you are having a lot of fun :), very, very  good work :thumbsup:

Probably you all know this, but, just in case, that the "Wimboot status" is checked through fsutil:

http://technet.micro...y/dn631790.aspx

fsutil wim enumwims c:

 

seems to me like a confirmation that the thingy is *integrated* in NTFS that has been in some way modified (as also a new data structure - at the moment undocumented - has been introduced):

http://msdn.microsof...5(v=vs.85).aspx

as well as "placeholder files":

http://msdn.microsof...8(v=vs.85).aspx

 

Keep up the good work! :thumbup:

 

:duff:

Wonko



#145 misty

misty

    Silver Member

  • Developer
  • 703 posts
  •  
    United Kingdom

Posted 25 April 2014 - 02:11 PM

@synchronicity
Just tested the --wimboot functions and can confirm another success - thanks.

I have a feature request. Is it possible to apply the windows boot files instead of adding a pointer file? Please refer to my post here.

Fantastic work.

Regards,

Misty

#146 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1920 posts
  • Location:Nantes - France
  •  
    France

Posted 25 April 2014 - 04:07 PM

@synchronicity
Just tested the --wimboot functions and can confirm another success - thanks.

I have a feature request. Is it possible to apply the windows boot files instead of adding a pointer file? Please refer to my post here.

Fantastic work.

Regards,

Misty

 

could not the sector [PrepopulateList] in wimbootcompress.ini fit that purpose?



#147 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 25 April 2014 - 04:22 PM

Wonko --- Thanks for the tips.  I was not aware that fsutil supports those WIM commands now.  But based on their behavior, they're probably just wrappers around the documented ioctls for WOF:

  • fsutil wim enumfiles   =>  FSCTL_ENUM_EXTERNAL_BACKING
  • fsutil wim enumwims  => FSCTL_ENUM_OVERLAY
  • fsutil wim removewim => FSCTL_REMOVE_OVERLAY
  • fsutil wim queryfile      => FSCTL_GET_EXTERNAL_BACKING

I think the USN journal and placeholder files are probably *not* related to WIMBoot.  All file requests to WIMBoot pointer files go through the filter driver, so I don't believe the USN journal would be needed.  And placeholder files seem to just be another feature that happens to be implemented with reparse points.  They don't use the same reparse tag that WOF/WIMBoot does.

 

misty --- As erwan just posted, your could add an appropriate pattern to the PrepopulateList of \Windows\System32\WimBootCompress.ini.  Or make a modified copy of this file and provide it to wimcapture using the --config option.  I think you would need the patterns \bootmgr,  \Boot\*, and \Boot\*\*.  (This pattern matching isn't recursive for directories yet, so I don't think simply "\Boot" would do anything.  I will look into changing this though.)  As to why these aren't in the configuration file that Microsoft provides, I'm not sure.  Probably they expected boot information to be set up separately using the bcdboot utility.



#148 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 13639 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 25 April 2014 - 04:51 PM

@Sinchronicity

Yes, most probably those commands are just wraparounds to those functions, but I was only trying to confirm that the good MS guys are (as always mis-documenting or completely failing to document properly) introducing "new, strange" features to the actual NTFS (not necessarily connected to wims) and that the wim format (in this modified for the nth time version) is acquiring more and more the "dignity" of a filesystem (till now it was IMHO more seen as a "simple" compression format, not entirely unlike ZIP or RAR).

 

:duff:

Wonko 

 



#149 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 25 April 2014 - 04:58 PM

I have most of the information in the WimOverlay.dat files figured out by running some basic tests and examining the binary data in the file.  For each WIM archive registered as a data source on the volume, it contains:

 

- Type of WIM ("OS" or "Not OS", whatever that means)

- Index of image in WIM that was applied

- GUID of WIM file

- Disk format (MBR or GPT) of disk containing WIM file

- Start sector of MBR partition or GUID of GPT partition containing WIM file

- MBR disk ID or GPT disk GUID of disk containing WIM file

- Absolute path to WIM file, excluding drive letter

 

So, the "WIMBoot pointer files" should still work if the drive letter containing the WIM file is changed, since this file apparently doesn't store the drive letter but rather the location of the partition.

 

But there are still a bunch of fields I don't know the meanings of, and I haven't gotten them to change yet.  If anyone wants to, they could send me their WimOverlay.dat and WimOverlay.backup files (they're in "\System Volume Information" on any volume to which a WIM has been applied in WIMBoot mode) and I can check to see if there's anything interesting.



#150 misty

misty

    Silver Member

  • Developer
  • 703 posts
  •  
    United Kingdom

Posted 25 April 2014 - 07:58 PM

could not the sector [PrepopulateList] in wimbootcompress.ini fit that purpose?


misty --- As erwan just posted, your could add an appropriate pattern to the PrepopulateList of \Windows\System32\WimBootCompress.ini....

I'll try this next.
 

...Or make a modified copy of this file and provide it to wimcapture using the --config option. I think you would need the patterns \bootmgr, \Boot\*, and \Boot\*\*...

I copied the contents of the file erwan.l posted here and added entries for \bootmgr and \boot\BCD to the [PrepopulateList]. On running wimlib-imagex with the --config option I received the following error messages -
[WARNING] E:\WimBoot.wimlib\wimbootcompress.ini:5: Unrecognized section "[CompressionExclusionList]"
[WARNING] E:\WimBoot.wimlib\wimbootcompress.ini:72: Unrecognized section "[CompressionFolderList]"
The capture completed, however when reapplied bootmgr was not copied back to the volume and I had to revert to using bcdboot.
 

...As to why these aren't in the configuration file that Microsoft provides, I'm not sure. Probably they expected boot information to be set up separately using the bcdboot utility.

I assume that Microsoft expect most people to have a seperate boot partition - I believe that this is/was the default setup if a hard disk hasn't already been partitioned.

Regards,

Misty




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users