Jump to content











Photo
* * * * * 4 votes

wimlib, with ImageX implementation

wim imagex winpe boot

  • Please log in to reply
365 replies to this topic

#151 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 25 April 2014 - 08:10 PM

The capture completed, however when reapplied bootmgr was not copied back to the volume and I had to revert to using bcdboot.

 

 

Duh, I know what the problem is.  The [PrepopulateList] is only used during extraction, and for that the configuration file used is \Windows\System32\WimBootCompress.ini *within the WIM image*.  I guess I will need to change it so that when a capture configuration file is explicitly specified for a WIMBoot capture, it will be placed in the WIM image at \Windows\System32\WimBootCompress.ini, overriding the copy from the filesystem.



#152 misty

misty

    Gold Member

  • Developer
  • 1033 posts
  •  
    United Kingdom

Posted 25 April 2014 - 08:39 PM

@synchronicity
I tried again, this time using DISM and editing \Windows\System32\WimBootCompress.ini - bootmgr was still not copied back to the volume.
 
I did a quick test using wimlib-imagex (with \Windows\System32\WimBootCompress.ini edited directly) - I aborted on seeing the WARNINGS I mentioned in my previous post. I'll try again and complete the whole process.

Also, just noticed the following in the wimlib documentation -

Specifies a configuration file (UTF-8 or UTF-16LE encoded) for capturing the new image. The
configuration file specifies files that are to be treated specially during the image capture.

Is this required?

#153 misty

misty

    Gold Member

  • Developer
  • 1033 posts
  •  
    United Kingdom

Posted 25 April 2014 - 09:09 PM

Just retested. Edited \Windows\System32\WimBootCompress.ini and saved it in UTF-8 format just to be safe. This time I used Wimlib-imagex to capture and reapply my Windows installation. bootmgr was reapplied successfully and I was able to chainload it via Grub4dos - without having to use bcdboot.

:cheers:

#154 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 25 April 2014 - 11:47 PM

I've updated the v1.6.3 BETA files.  Now, if you capture a WIM image with --wimboot and also specify a custom configuration file with --config, the configuration file will automatically be placed at \Windows\System32\WimBootCompress.ini in the WIM image.

 

I also updated that section of the documentation, but yes the code currently expects the configuration file to be either UTF-8 or UTF-16LE encoded.  I guess this could be a problem if you write the file using a program like Notepad, which uses the current "Windows Code Page" by default, and you use non-ASCII characters.  Of course you'd hope that everything would be using Unicode now...


  • misty likes this

#155 misty

misty

    Gold Member

  • Developer
  • 1033 posts
  •  
    United Kingdom

Posted 26 April 2014 - 10:04 AM

I've updated the v1.6.3 BETA files.  Now, if you capture a WIM image with --wimboot and also specify a custom configuration file with --config, the configuration file will automatically be placed at \Windows\System32\WimBootCompress.ini in the WIM image....

Thanks. It's working for me and the files added to the [PrepopulateList] are now being applied correctly - the same can't be said when using DISM, at least not in the test I did yesterday!

For those interested, I only added the following to the [PrepopulateList] section of my config file -
\bootmgr
\boot\BCD

...the code currently expects the configuration file to be either UTF-8 or UTF-16LE encoded. I guess this could be a problem if you write the file using a program like Notepad, which uses the current "Windows Code Page" by default, and you use non-ASCII characters...

Does this mean that a non UTF-8 or UTF-16LE encoded configuration file will work as long as it doesn't use any non-ASCII characters?

Regards,

Misty

#156 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 14829 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 26 April 2014 - 10:23 AM

For those interested, I only added the following to the [PrepopulateList] section of my config file -

\bootmgr
\boot\BCD

And what happens (or fails to happen) to the other files that normally are in the \boot\ directory? :unsure:

If you prefer, can you compare the result of your "adding those two files to the .ini and apply the wim" against "not adding them and run BCDBOOT after having applied the wim"?

 

:duff:

Wonko



#157 misty

misty

    Gold Member

  • Developer
  • 1033 posts
  •  
    United Kingdom

Posted 26 April 2014 - 11:41 AM

@Wonko
In my case the results are the same - Windows boots without error. I have never needed the other files contained in the boot directory in my setup - so I decided not to add them. For those that do, then use the following entries in the [PrepopulateList] section of your .ini file -
\bootmgr
\boot\*
:cheers:

#158 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 26 April 2014 - 03:55 PM

Does this mean that a non UTF-8 or UTF-16LE encoded configuration file will work as long as it doesn't use any non-ASCII characters?

 

I believe this is true for all non-obscure single byte character sets (e.g. any of the ISO 8859 encodings).  The file would be byte-for-byte identical to the equivalent ASCII text as well as byte-for-byte identical to the equivalent UTF-8 text.  So the current code would treat it as UTF-8 and work fine.

 

@Wonko
In my case the results are the same - Windows boots without error. I have never needed the other files contained in the boot directory in my setup - so I decided not to add them. For those that do, then use the following entries in the [PrepopulateList] section of your .ini file -

\bootmgr
\boot\*

 

Well, bootmgr might need the language files if it *does* fail to boot, so for that you'd need to add "\Boot\*\*".  And I'm also seeing \Boot\memtest.exe --- probably rarely used, but it almost certainly won't work if it's a WIMBoot pointer file.  \Boot\* would catch both it and \Boot\BCD.



#159 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 14829 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 26 April 2014 - 05:42 PM

@Wonko
In my case the results are the same - Windows boots without error. I have never needed the other files contained in the boot directory in my setup - so I decided not to add them. For those that do, then use the following entries in the [PrepopulateList] section of your .ini file -

\bootmgr
\boot\*

Very good :thumbsup:, but what I asked was (slightly) different. (please bear with me :))

 

Let's put aside for a moment the "wimboot". :w00t:

 

If you "install" a Windows Vista :ph34r" or 7 (and I believe this is the same for 8/8.1) the "quick way" is to (simply and rudimentally) apply "normally" the install.wim and then run bootsect.exe (if needed) and bcdboot, like in:

http://reboot.pro/to...external-drive/

 

imagex f:\sources\install.wim 5 c:

bcdboot c:\windows /s c:

bootsect /nt60 c:

 

or, in the case of 8 (or with previous versions as long as you use the 8 DISM), one can also apply the wim with DISM, with something *like*:

DISM /Apply-Image /ImageFile:D:\sources\install.wim /index:5 /ApplyDir:C:\

The install.wim is seemingly NOT containing a BOOTMGR and a \boot\ directory in it's root, and after you have applied the image there is no trace of these files on the "target" disk partition.

 

When you actually run the bcdboot command, the "target" disk partition gets added the BOOTMGR and the \boot\ directory, this latter containing, besides the BCD, a number of files.

From your report I gathered that only the BOOTMGR and the \boot\BCD files are needed for booting, did I get this right?

 

I.e., in theory (generically) one could replace the Bcdboot command with plainly copying a BOOTMGR to root and use a BCDEDIT script (or some of the other tools) to create a new BCD, or am I missing something?

 

:duff:

Wonko



#160 misty

misty

    Gold Member

  • Developer
  • 1033 posts
  •  
    United Kingdom

Posted 26 April 2014 - 06:26 PM

...From your report I gathered that only the BOOTMGR and the \boot\BCD files are needed for booting, did I get this right?...I.e., in theory (generically) one could replace the Bcdboot command with plainly copying a BOOTMGR to root and use a BCDEDIT script (or some of the other tools) to create a new BCD, or am I missing something?
 
:duff:
Wonko

You don't appear to be missing anything. :thumbup:

The trick would be scripting the right commands with BCDEDIT. I haven't tried creating a BCD store for a while - at least not for booting a full OS, only WinPE. I haven't actually checked the Windows 8.1 BCD store and am therefore unsure of the entries. As synchronicity mentioned earlier, memtest.exe may be required in some cases. Also the .mui files might be needed in some scenarios. Whether these are actually needed will depend on the entries in the BCD store - possibly the .mui files are required if there are any errors during the boot process.

If I get some time I'll check out the BCD store later and see if I can script creating a new store and booting Windows 8.1.

Regards,

Misty

P.s. just noticed an emoticon titled jaclaz - :jaclaz: (how long has that been there?)

#161 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2735 posts
  • Location:Nantes - France
  •  
    France

Posted 26 April 2014 - 06:44 PM

I just checked my install.wim, captured with dism /wimboot.

It does contain \boot folder and bootmgr.

 

Thus, whether I use dism or wimlib to apply, I'll miss these 2 files.

The trick for now is either to use bcdboot or add the right entries in wimboot.ini.

 

The below script should in theory create a proper bcd :

@echo off
setlocal
set BCDEDIT=%SYSTEMROOT%\system32\bcdedit.exe
set BCDSTORE=C:\boot\BCD

del %BCDSTORE% /F

rem create store
%BCDEDIT% /createstore %BCDSTORE%

rem create bootmgr entry
%BCDEDIT% /store %BCDSTORE% /create {bootmgr}
%BCDEDIT% /store %BCDSTORE% /set {bootmgr} description "Boot Manager"
%BCDEDIT% /store %BCDSTORE% /set {bootmgr} device boot
%BCDEDIT% /store %BCDSTORE% /set {bootmgr} timeout 20

rem add new windows7/8 entry

for /f "tokens=2 delims={}" %%g in ('%BCDEDIT% /store %BCDSTORE% /create /application osloader') do set guid={%%g}
%BCDEDIT% /store %BCDSTORE% /set %guid% device partition=C:
%BCDEDIT% /store %BCDSTORE% /set %guid% path \Windows\system32\winload.exe
%BCDEDIT% /store %BCDSTORE% /set %guid% osdevice partition=C:
%BCDEDIT% /store %BCDSTORE% /set %guid% systemroot \Windows
%BCDEDIT% /store %BCDSTORE% /set %guid% description "Windows 8"
%BCDEDIT% /store %BCDSTORE% /displayorder %guid% /addlast


#162 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 26 April 2014 - 09:41 PM

I've added experimental support for doing a "WIMBoot extraction" when the WOF driver isn't available.  So it should work on older versions of Windows.  Feel free to test this with the new v1.6.3-BETA.

 

Unfortunately I still wasn't able to determine the meanings of all the fields in the WimOverlay.dat file, but I figured out enough to get something to work.  To be safe, wimlib will not update this file if any fields have unexpected values.


  • misty likes this

#163 misty

misty

    Gold Member

  • Developer
  • 1033 posts
  •  
    United Kingdom

Posted 26 April 2014 - 09:47 PM

I've added experimental support for doing a "WIMBoot extraction" when the WOF driver isn't available.  So it should work on older versions of Windows.  Feel free to test this with the new v1.6.3-BETA.

Sounds great. It's getting late here so I'll test sometime tomorrow and will report back.

:cheers:

#164 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2735 posts
  • Location:Nantes - France
  •  
    France

Posted 26 April 2014 - 09:51 PM

Test in progress :)

 

Edit1 : indeed it mentions that "the wof driver is not available, updating wimoverlay.dat directly.".

 

Edit2: seemed much quicker than before (i.e using the wof driver).

 

Edit3 : this time I cannot bcdboot c:\windows /s c: like I used before.

error code 0x780, failed to open file c:\windows\boot\pcat\bootmgr for read.

 

Edit4 : when copy pasting bootmgr manually, i get error 0x80070780 : file cannot be accessed by the system.

 

Edit5 : i actually get that error with any file on disk.



#165 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 26 April 2014 - 11:46 PM

Is that all from the not-completely-up-to-date version of Windows?  After being created, all the pointer files will be inaccessible without the WOF driver; there's no way around that.  So if you want to make modifications to the filesystem before booting the new operating system (which must itself be Windows 8.1 Update 1, so it has the WOF driver and can read its own files), you need to make sure that any files that need to be read or written are listed in [PrepopulateList], so that they are extracted as normal files.  Probably you should just list "bootmgr" (no leading slash) so that all instances of that file are matched.



#166 misty

misty

    Gold Member

  • Developer
  • 1033 posts
  •  
    United Kingdom

Posted 27 April 2014 - 05:37 AM

@synchronicity
Just tested the new build. I used WinPE 3.1 to capture my Windows 8.1 (Update 1) installation, then formatted the OS partition and then reapplied the .wim file - it works :worship:

Just out of curiosity I then tried replacing bootmgr with an older version (from Windows 7 (SP1)) - Windows 8.1 then failed to reboot. Looks like the WOF driver isn't the only dependency.

Now on to a wimlib question. Is it possible to overwrite a file in a wim? Or do I need to delete it first? If a file can be overwritten please let me know which command to use. I'm basically trying to add a folder that contains updated files. I tried a test using the update command, but received the following error (my.file was a file in the directory I was trying to add back to the wim after updating it) -


[ERROR] Cannot overlay "my.file" onto existing dentry: is not directory-on-directory!
ERROR: Exiting with error code 23:
Conflicting files in overlay when creating a WIM image.

@erwan.l/Wonko
This is becoming slightly offtopic, as it relates more to Windows than to wimlib-imagex. Just wanted to confirm that the batch file that erwan.l posted (here) is working. For the record then, the majority of the files included in the boot folder are not required for booting Windows 8.1 (and probably all other versions of Windows that use the bootmgr boot loader - every OS since XP) - all that's needed is bootmgr and boot\BCD.

Regards,

Misty

#167 synchronicity

synchronicity

    Frequent Member

  • Advanced user
  • 165 posts
  •  
    United States

Posted 27 April 2014 - 05:55 AM

To replace a file in a WIM using 'wimlib-imagex update', you currently have to explicitly delete the file using a delete command, then add it with an add command.  So for your use case the update command file would have to be something like:

delete --recursive \somedir
add c:\blah\blah\blah\somedir \somedir

I think I ought to change that, though.  Probably add should default to replacing existing non-directory files, and there should be a --no-replace option to get the current behavior (issue error when non-directory file already exists in the WIM).

 

 



#168 misty

misty

    Gold Member

  • Developer
  • 1033 posts
  •  
    United Kingdom

Posted 27 April 2014 - 06:04 AM

@synchronicity

To replace a file in a WIM using 'wimlib-imagex update', you currently have to explicitly delete the file using a delete command, then add it with an add command.  So for your use case the update command file would have to be something like:



delete --recursive \somedir
add c:\blah\blah\blah\somedir \somedir

I've been using a workaround by deleting individual files before replacing them. I can't delete the whole folder from the wim as I'm basically updating a few files that were previously extracted from the wim - in my case I'm adding a selection of files and folders back to the Windows directory in a WinPE after modifying them.
 

...I think I ought to change that, though.  Probably add should default to replacing existing non-directory files, and there should be a --no-replace option to get the current behavior (issue error when non-directory file already exists in the WIM).

That would be fantastic if possible.

Many thanks for the quick response - and thanks yet again for sharing your work.

Misty

#169 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2735 posts
  • Location:Nantes - France
  •  
    France

Posted 27 April 2014 - 10:57 AM

Is that all from the not-completely-up-to-date version of Windows?  After being created, all the pointer files will be inaccessible without the WOF driver; there's no way around that.  So if you want to make modifications to the filesystem before booting the new operating system (which must itself be Windows 8.1 Update 1, so it has the WOF driver and can read its own files), you need to make sure that any files that need to be read or written are listed in [PrepopulateList], so that they are extracted as normal files.  Probably you should just list "bootmgr" (no leading slash) so that all instances of that file are matched.

 

Hi Synchronicity,

 

I tried the wimapply with the --config parameter but it seems this parameter is only supported when capturing?

 

What would be the proper method?

So far I managed to recreate a BCD (with bcdedit).

I also extracted bootmgr (without --wimboot to my current folder) but cannot use it to overwrite the c:\bootmgr one...

 

I am bit lost for now :)

Idea being to manage it all from a winpe 3/4, without having to fetch files from another source than the wim file.

 

@Misty : I see you succeeded that scenario, would you be able to give more details?



#170 misty

misty

    Gold Member

  • Developer
  • 1033 posts
  •  
    United Kingdom

Posted 27 April 2014 - 11:26 AM

@erwan.l
I used the following command to capture the image -

wimlib-imagex capture C:\ D:\8.1.capture.wim "Windows 8.1 (x86)" --config=D:\wimbootcompress.ini --wimboot
And this cope to reapply it (after formatting the target drive) -
wimlib-imagex apply D:\8.1.capture.wim C:\ --wimboot
Hope this helps. BTW, the config file specified in the capture command was copied as C:\Windows\System32\wimbootcompress.ini - overwritting the original file. My only addition was to add the following to the end of the [PrepopulateList] list -
\bootmgr
\boot\BCD
Regards,

Misty

#171 misty

misty

    Gold Member

  • Developer
  • 1033 posts
  •  
    United Kingdom

Posted 27 April 2014 - 11:32 AM

Hello again,

Not sure if this will help, however this is a batch I put together for my own use (you will probably need to change the path to wimlib-imagex, and also the paths at the start of the batch file) -
@echo off
SET CAPTURE.DRIVE=C:\
SET WIM.FILE=D:\8.1.capture.wim
SET APPLY.TO.DRIVE=C:\
SET LABEL="Windows 8.1 (x86)"

:_options
cls
echo.
echo Options
echo =======
echo.
ECHO CAPTURE.DRIVE=%CAPTURE.DRIVE%
ECHO WIM.FILE=%WIM.FILE%
ECHO APPLY.TO.DRIVE=%APPLY.TO.DRIVE%
ECHO LABEL=%LABEL%
echo.
echo   1] CAPTURE
echo   2] APPLY
echo   3] Format (hd0,1)
echo   4] ABORT
echo.
set choice=
set /p choice=Select option [1 - 4] and press ENTER.
if '%choice%'=='1' goto _capture
if '%choice%'=='2' goto _apply
if '%choice%'=='3' goto _diskpart
if '%choice%'=='4' goto _end
goto _options


:_capture
cls
IF EXIST %WIM.FILE% (
Echo .wim file already exists - either delete this file
Echo or edit this batch
echo.
echo.
echo.
pause
goto _end
)

%~dp0wimlib\wimlib-imagex capture %CAPTURE.DRIVE% %WIM.FILE% %LABEL% --config=%~dp0wimbootcompress.ini --wimboot
echo.
echo.
echo.
pause
goto _options



:_apply
cls
IF NOT EXIST %WIM.FILE% (
Echo .wim file is missing
echo.
echo.
echo.
pause
goto _end
)

%~dp0wimlib\wimlib-imagex apply %WIM.FILE% %APPLY.TO.DRIVE% --wimboot
echo.
echo.
echo.
pause
goto _options


:_diskpart
cls
diskpart /s %~dp0diskpart.script.txt
echo.
echo.
echo.
pause
goto _options

:_end
My diskpart script contained the following - REMEMBER TO CHANGE THIS TO SUIT YOUR OWN SETUP OR YOU WILL PROBABLY FORMAT THE WRONG DRIVE AND LOSE YOUR DATA
sel disk 0
sel part 2
format fs=ntfs label=HD01 quick
Regards,

Misty

#172 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2735 posts
  • Location:Nantes - France
  •  
    France

Posted 27 April 2014 - 11:33 AM

@erwan.l
I used the following command to capture the image -
 

wimlib-imagex capture C:\ D:\8.1.capture.wim "Windows 8.1 (x86)" --config=D:\wimbootcompress.ini --wimboot
And this cope to reapply it (after formatting the target drive) -
wimlib-imagex apply D:\8.1.capture.wim C:\ --wimboot
Hope this helps. BTW, the config file specified in the capture command was copied as C:\Windows\System32\wimbootcompress.ini - overwritting the original file. My only addition was to add the following to the end of the [PrepopulateList] list -
\bootmgr
\boot\BCD
Regards,

Misty

 

 

Ok so that would be mean my current install.wim is hopeless if I want to use a winpe 3/4 then?

Since I did not use a modified wimbootcompress.ini during the capture phase?

For sure MS should have added these 2 entries (bootmgr & bcd) at first...

 

My current install.wim works from a winpe 8.1u1.

My "only" issue for now seems to be that I cannot overwrite my bootmgr "pointer" by a "plain" bootmgr.

The bcd I was able to delete/recreate thus.



#173 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2735 posts
  • Location:Nantes - France
  •  
    France

Posted 27 April 2014 - 11:43 AM

oki I managed to boot my windows 8.1u1, applied with wimlib, from a winpe 4.0 (i.e non win8u1).

 

but this is a ugly hack :

-i deleted the BCD and recreated manually with bcdedit (using this post),

-i restored bootmgr from my wim file (without the --wimboot parameter),

-renamed it to bootmg_,

-patched my boot sector to use bootmg_ and not bootmgr (since i could not delete it)

 

So in short, if I dont want to redo my WIM file based on a new wimbootcompress.ini, I need to be able to :

-delete \boot\bcd and bootmgr

-restore/recreate them

 

My issue with deleting bootmgr is that it is a "supperhidden" file (+s+h) and WinPE is not very helping with seing/deleting these...

 

EDIT :

-I used a MFT editor to delete bootmgr (a reparse point at this stage)

-I restored the individual file bootmgr from my install.wim (without --wimboot)

-Boot ok :)

 

Still a ugly hack but fun ! :)



#174 misty

misty

    Gold Member

  • Developer
  • 1033 posts
  •  
    United Kingdom

Posted 27 April 2014 - 12:06 PM

...but this is a ugly hack ...

No kidding - the lengths we have to go to sometimes to get something working! I can think of two suggestions that might help -

  • Apply your install.wim without the --wimboot flag then recapture it with the new config file.
  • Use the wimlib-imagex update command and delete /Windows/System32/wimbootcompress.ini, then add your amended file

I suspect that, based on a comment synchronicity made elsewhere in this thread, that the [PrepopulateList] is only used when an image is applied and not during the capture phase - option 2 should work if this is the case.

Good luck!

P.s. In regards to the superhidden files I use the following settings in my WinPE build -

  • HKLM\DEFAULT\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced ShowSuperHidden 1
  • HKLM\DEFAULT\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced SuperHidden 0

These are both DWORD values.



#175 erwan.l

erwan.l

    Platinum Member

  • Developer
  • 2735 posts
  • Location:Nantes - France
  •  
    France

Posted 27 April 2014 - 01:07 PM

I think at this point we need to review the use of wimbootcompress.ini.

 

To me this file is needed only when capturing but I read different understandings on this point.

As a whole if I understand correctly, apart from a different compression scheme (XPRESS 4k chunks) and a wimboot (XML) tag in the wim file, the capture wim file is not much different from an "ordinary" wim file.

Thus, it seems that files inside the wim get some sort of a flag that will make a difference when restoring/applying? (i.e bootmgr, bcd, etc).

 

When applying, the wimbootcompress.ini is not used right?

Still, I may have to restore some of my files (bootmgr, bcd, etc) individually, removing this time the --wimboot flag, otherwise these files will be restored as reparse points?

 

Last, reparse points files cannot be dealt with without the MS WOF driver unless you deal with them at the MFT level.

 

Is my understanding correct?






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users