Jump to content











Photo
* * * * * 1 votes

VHD_WIMBOOT - Apply and Capture of WIM Files for OS in VHD

ramdisk grub4dos wimlib svbus windows 10 ssd usb wim vhd wimboot

  • Please log in to reply
1025 replies to this topic

#76 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 10 April 2019 - 01:07 PM

How does the VHD file know to what WIM file it is connected and what occurs when using Upd WimBoot button ?

 

The relation between VHD and WIM is not in the registry and not in the BCD entry, but it is inside the VHD in file \System Volume Information\WimOverlay.dat

 

You can open the VHD in 7-zip and then Copy the file WimOverlay.dat to a place where you can watch the binary content of this file by using TinyHexer.

 

It shows the WIM File path and the MountedDevices registry entry with the Disk Signature of the Drive where the WIM file is located. 

 

attachicon.gifWimOverlay-7z-2019-04-09_152644.png

Very interesting. :)

 

Can you post a sample WimOverlay.dat?

 

It should be trivial to make a Structure Viewer for Tiny Hexer or a direct parser for that file.

 

:duff:

Wonko


  • wimb likes this

#77 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 10 April 2019 - 01:23 PM

Very interesting. :)

 

Can you post a sample WimOverlay.dat?

 

It should be trivial to make a Structure Viewer for Tiny Hexer or a direct parser for that file.

 

 

Sure, it is certainly very interesting. :)

 

I was looking for how actually VHD WIMBOOT works and then I found this file WimOverlay.dat

 

Attached File  WimOverlay.zip   21.08KB   1876 downloads == WimOverlay-TinyHexer-2019-04-10_153600.png



#78 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 10 April 2019 - 02:38 PM

The good news are that seemingly only you and me were unaware of this mechanism :blush: :w00t:.

The bad news are that Wimlib (that actually does support it BOTH through Windows calls AND through driect editing) has this to say on the matter:

https://wimlib.net/man1/wimapply.html

 

 

In addition, this option works best when running on Windows 8.1 Update 1 or later, since that is the first version of Windows that contains the Windows Overlay Filesystem filter driver ("WOF"). If the WOF driver is detected, wimlib will create the WIMBoot "pointer files" using documented ioctls provided by WOF.

Otherwise, if the WOF driver is not detected, wimlib will create the reparse points and edit the file "\System Volume Information\WimOverlay.dat" on the target volume manually. This is potentially subject to problems, since although the code works in certain tested cases, neither of these data formats is actually documented by Microsoft. Before overwriting this file, wimlib will save the previous version in "\System Volume Information\WimOverlay.wimlib_backup", which you potentially could restore if you needed to.

 

The format is not documented. :(

 

Extracting the filename is still possible, of course but I was fantasizing about the possibility to use a "same" .vhd with different .wim files ...

 

The Wimlib Author did attempt to document the format, however:

https://github.com/C...de/wimlib/wof.h

 

I'll see what I can do with this latter info. :unsure:

 

:duff:

Wonko



#79 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 10 April 2019 - 03:08 PM

 

Extracting the filename is still possible, of course but I was fantasizing about the possibility to use a "same" .vhd with different .wim files ...

 

The Wimlib Author did attempt to document the format, however:

https://github.com/C...de/wimlib/wof.h

 

 

Thanks for the Info  :)

 

WimOverlay.dat uniquely identifies the WIM File.

 

You can have different VHD's combined with the "same" WIM file, but not the other way around .....



#80 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 10 April 2019 - 05:09 PM

Thanks for the Info  :)

 

WimOverlay.dat uniquely identifies the WIM File.

 

You can have different VHD's combined with the "same" WIM file, but not the other way around .....

 

Yes, in fact I have tested it. 

 

But also you can have multiple indexes (wimboot capables) on same wim file (as usually in any other wim file). And one or more VHDs linked to each index, See: http://reboot.pro/to...e-5#entry210406

 

EDIT: But if I attach the VHDs and run in Power Shell: Get-WIMBootEntry -Path "J:\"  Only got the wim file path, not any mention to the respective index.
 

alacran


  • antonino61 likes this

#81 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 10 April 2019 - 06:28 PM

 

EDIT: But if I attach the VHDs and run in Power Shell: Get-WIMBootEntry -Path "J:\"  Only got the wim file path, not any mention to the respective index.
 

 

And what do you get in admin command window with

fsutil wim enumwims J:


#82 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 10 April 2019 - 07:22 PM

I was thinking the other way round. :unsure:

 

Maybe we could edit the  \System Volume Information\WimOverlay.dat on-the-fly or possibly have several pre-made ones by the "normal" procedure, rnamed to *something else*:

\System Volume Information\WimOverlay.dbt

\System Volume Information\WimOverlay.dct

\System Volume Information\WimOverlay.ddt :w00t: :ph34r:

...

\System Volume Information\WimOverlay.dyt

and rename the "current" \System Volume Information\WimOverlay.dat to \System Volume Information\WimOverlay.dzt and any of the other ones back to \System Volume Information\WimOverlay.dat.

 

This would probably be possible through a batch grub4dos script running at boot time.

 

:duff:

Wonko



#83 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 10 April 2019 - 11:56 PM

 

And what do you get in admin command window with

fsutil wim enumwims J:

 

Then I got: I:\Wimboot\10-WB.wim:2   the full path and the image index, 2 in this case, attached images of Power Shell and Command prompt.

 

EDIT: The following command is valid for both VHDs 10x86 and 10x64, created respectively from 10-WB.wim image 1 and image 2, if I want to relocate the coupled Wim-VHDs on primary partition of external device:

 

Run as admin directly on each attached VHD drive:

 

DISM.exe /Update-WIMBootEntry /Path:J:\ /DataSourceID:0 /ImageFile:K:\Wimboot\10-WB.wim

 

Where:

 

J:\    Is the attached VHD located on internal HD, before copy it to USB, latter if you want it can be copied to K:\ root,

 

I made this before copy the VHD because if I want to only Ramboot and then make a LZ4 compressed VHD (only can be loaded in Ram, and only work if made from a fixed size VHD) it is faster to do it on the HD, and latter copy to USB only the 10x64-WB.vhd.lz4 (130 MB) and 10x86-WB.vhd.lz4 (99.2 MB) as required (and not the full 1.5 GB for each fixed size VHD).

Or if for any reason I want to be able to also boot from bootmanager and not only Ramboot the OSs (very improbable for me since Ramboot is IMHO the best and fastest option) then I need to make and/or copy 480 MB for the expandable 10x64.vhd and 360 MB for the expandable 10x86.vhd.

 

K:\Wimboot\10-WB.wim    is the wim file already copied to single primary (NTFS) partition of my external USB device.

 

All BCDs and grub4dos menus were made manually.

 

So just applied this command to 10x86.vhd and also the same to 10x64.vhd to relocate the files.

 

By the way I have a new addition to moded WimBootCompress-2019-03-31.ini (attached).

 

[ExclusionList]
\MSOCache\*  >>> This is the hidden cache folder after installing Office (saving to compress in your wim file from about 300 MB to almost 1 GB, depending on version), sometimes it may be created on other partition if available.

 

 

alacran

Attached Thumbnails

  • Wim location PS.png
  • Wim location Command Promt.png

Attached Files



#84 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 11 April 2019 - 06:32 AM

 

DISM.exe /Update-WIMBootEntry /Path:J:\ /DataSourceID:0 /ImageFile:K:\Wimboot\10-WB.wim

 

 

VHD_WIMBOOT will attach the selected VHD file and then exactly use the same code to Update WimBoot the VHD for the selected WIM file.

 

Thanks for the new WimBootCompress.ini file that also excludes MSOCache folder.



#85 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 12 April 2019 - 10:51 PM

Dear Wimb,

I had a hard time trying to make version 2.0 work as well as version 1.8; started thinking there was something wrong with my vhd+wim baking or my actual files, as 2.0 would not complete bcd tasks and gave a lot of registry copying errors. I resorted to 1.8 back again, and everything went back to normal. no issues at all. do not know if I am the only one to have noticed that, but I can assure u it was a bit of a headache. 

nino



#86 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 13 April 2019 - 06:02 AM

 

I had a hard time trying to make version 2.0 work as well as version 1.8; started thinking there was something wrong with my vhd+wim baking or my actual files, as 2.0 would not complete bcd tasks and gave a lot of registry copying errors. I resorted to 1.8 back again, and everything went back to normal. no issues at all. do not know if I am the only one to have noticed that, but I can assure u it was a bit of a headache. 

 

 

The core of the program is the same for all versions from 1.5 until 2.0 and there are no registry operations involved for the VHD + WIM



#87 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 18 April 2019 - 05:10 PM

The Wimlib Author did attempt to document the format, however:

https://github.com/C...de/wimlib/wof.h

 

I'll see what I can do with this latter info. :unsure:

 

The attached is a tentative-only and more-than-half[1]-@§§ed attempt to a Structure VIewer for TinyHexer

 

There is something "wrong" (or I am completely failing to understand it in the wimlib notes) in the format, I tried to do my best to adapt the Structure Viewer to the sample provided, but it needs some serious checking against a number of different  Wimoverlay.dat files, particularly to check if both the "MBR" and the "GPT" data are parsed correctly.

 

I found a 40 bytes section that is seemingly missing from the wimlib document and - maybe - I found how to determine the offset and size of the path. :unsure:

 

 

 

:duff:

Wonko

 

[1] I would say 3/4 to 4/5, it is 10 years or so I don't touch Tiny Hexer Structure Viewer langauage (and I have never been actually familiar with it, and it has a few quirks for the beginners)

Attached Files



#88 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 19 April 2019 - 06:27 AM

Thanks for making the WimOverlay.mps script for TinyHexer.

 

The mpth_small.exe version of TinyHexer that I normally use, does not have the Structure Viewer.

 

Download mpth_18.exe  uploaded from https://www.softpedi...iny-hexer.shtml has Structure Viewer

 

More Info on using Tiny Hexer Scripts was given by you in http://reboot.pro/to...-hexer-scripts/

 

After mounting VHD as drive P: then use fsutil in admin command window

fsutil wim enumwims P:

The 40 bytes section in WimOverlay.dat being undocumented contains at offset 0x30 a 16 bytes guid that identifies the WIM file.

This guid is also shown in the fsutil output

 

TinyHexer-WimOverlay-2019-04-19_075410.png



#89 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 19 April 2019 - 07:53 AM

The 40 bytes section in WimOverlay.dat being undocumented contains at offset 0x30 a 16 bytes guid that identifies the WIM file.
This guid is also shown in the fsutil output
 
attachicon.gifTinyHexer-WimOverlay-2019-04-19_075410.png

Yep, I suspected it was a GUID, but it is a "strange" one. in the example posted should be 
{D7B18E94-EEB0-D1A5-C33B-325CE977FF83} which decodes as "Microsoft Reserved":
C:\appoggio\UUID>uuid -d D7B18E94-EEB0-D1A5-C33B-325CE977FF83
encode: STR:     d7b18e94-eeb0-d1a5-c33b-325ce977ff83
        SIV:     286705947539520541126998466409562374019
decode: variant: reserved (Microsoft GUID)
        version: 13 (unknown)
        content: D7:B1:8E:94:EE:B0:01:A5:03:3B:32:5C:E9:77:FF:83
                 (not decipherable: unknown UUID version)
And there is no reference to this (that I can see) in the Wimlib attempt to describe the format.
The whole "block" I called jokingly "Here be lions" seems to be not mentioned at all, and it is fun as the field WIM_PROVIDER_CURRENT_VERSION is identified and it is 1 (as in the doc) so it shuold be the same format.
 
Anyway, I will tag that as the Wim GUID and that will be 16 bytes less, making the 40 mistery bytes into the 24 mistery bytes (actually 16, I had some overlapping)
 
Can you try to make new WimOverlay.dat file (for the same .wim) with Wimlib and post it? 
Maybe the program has it fine and it is only the comments about the format that are outdated or prelimiinary.
 
:duff:
Wonko
 
EDIT: OK, see new tentative attachment.

Attached Files



#90 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 19 April 2019 - 09:01 AM

I have used wimlib to make new vhd from same wim file and have saved WimOverlay.dat as file WimOverlay-2.dat

 

The guid is now interpreted OK using new version of WimOverlay.mps

 

Attached File  WimOverlay-2.zip   808.39KB   1841 downloads == TinyHexer-WimOverlay-2.dat-2019-04-19_105428.png



#91 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 19 April 2019 - 09:53 AM

Good.
Both files are identical but for the Disk Signature.
It should mean that Wimlib works just fine and it is just the doc that are missing something /or I am missing something in reading them).

The Structure Viewer remains alpha, it needs to be checked on GPT devices, and of course any further ideas/suggestions on the meaning of the bytes is welcome.

Added/modified (I was using an old reference) the "next available data source ID" at offset 16:
https://wimlib.net/f...topic.php?t=272

Current reference is:
https://wimlib.net/g...de/wimlib/wof.h

:duff:
Wonko

Attached Files



#92 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 19 April 2019 - 07:39 PM

TinyHexer-WimOverlay-GPT-2019-04-19_213331.png

 

Attached File  WimOverlay-GPT.zip   967.3KB   1853 downloads

 

GPT is looking good ....



#93 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 20 April 2019 - 07:18 AM

I have just installed SSD 970 EVO Plus NVMe M.2 500GB in my ASUS Z170i PRO GAMING Motherboard

 

The speed is up to 3 GB / sec for NVMe SSD .... instead of about 500 MB /sec for normal SSD and about 100 MB / sec for SATA harddisk

 

CDM-M2-2019-04-19_193641.png == P1070610.JPG

 

Install of Windows 10 x64 is working OK, but Install of Windows 10 x64 in VHDX or VHD WIMBOOT fail for GPT NVMe M.2 SSD

 

Apparently the NVMe M.2 SSD cannot be used for VHD WIMBOOT ..... :(

 

EDIT:

 

After MBR partitioning the NVMe M.2 SSD Drive and using VHD_WIMBOOT to Apply Captured WIM File to 25 GB Expandable VHD,

 

I can WIMBoot in BIOS mode from VHD + WIM located on MBR NTFS partition of NVMe M.2 SSD Drive   :thumbup:



#94 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 20 April 2019 - 08:43 AM

I am afraid I do not get you. Install wimboot on an nvme m.2 impossible? I have my system on nvme m.2. Is your nvme m.2 bootable?



#95 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 20 April 2019 - 09:20 AM

I am afraid I do not get you. Install wimboot on an nvme m.2 impossible? I have my system on nvme m.2. Is your nvme m.2 bootable?

 

Yes NVMe M.2 SSD has GPT partitioning and is UEFI bootable with Windows 10 x64, but is not bootable with Win10 x64 VHD WIMBOOT ....

 

Good news that wimboot is possible on NVMe M.2

 

Do you have MBR partitioning and BIOS mode booting ?



#96 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 20 April 2019 - 09:50 AM

of course I do. MBR-BIOS, not GPT-UEFI. if the bios sees it, it is bootable and can be wimbooted. here it was never any issue. u have set up a new era. no more setups, no more headaches. U have one 2-file-based (vhd+wim) system, 7-8gb overall in my case, u install some software and have a feeling that u can rewimboot your system, fine, vhd capture wim, wim apply vhd, and so on and so forth. that is the new windows. and, besides, u can keep what u want and get rid of what u don't want. that is the way to go about operating systems!

one piece of controversy, though: many ppl tell me there ain't no use in sparing on mb when I have all this ram and storage space. Well, I tried listening to them, but I have not seen any such improvement as to justify not sparing on space, even the smallest compressed version, with my hardware architecture, is half-a-second slower, as fast as, or even faster than a larger uncompressed vhd+wim combo.

nino



#97 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 20 April 2019 - 10:05 AM

Then I will test MBR partitioning and BIOS mode booting.

 

Do you use a special driver for NVMe M.2 SSD booting of Windows 10 x64 VHD WIMBOOT as FILEDISK ?

 

I use the build in Standard driver stornvme.sys of Microsoft as NVMe Controller



#98 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 20 April 2019 - 01:31 PM

OK  B)

 

After MBR partitioning the NVMe M.2 SSD Drive and using VHD_WIMBOOT to Apply Captured WIM File to 25 GB Expandable VHD,

 

I can WIMBoot in BIOS mode from VHD + WIM located on MBR NTFS partition of NVMe M.2 SSD Drive  :thumbup:

 

Thanks for your help.



#99 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 20 April 2019 - 01:40 PM

of course I do. MBR-BIOS, not GPT-UEFI. if the bios sees it, it is bootable and can be wimbooted. here it was never any issue.

Before anyone else posts this question (and I am pretty sure someone will be whining :ph34r: about this before or later):

 

Q. How do you deal with larger than 2.2 TB nvme devices? :w00t:

A. Up to 4 TB BIOS/MBR is fine with Windows 7 and later if you can accept a small limitation. ;)

https://msfn.org/boa...br-hard-drives/

https://msfn.org/boa...comment=1143244

 

:duff:

Wonko



#100 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 21 April 2019 - 12:51 AM

wonko, we r talking megabytes. we do not have such sizes. at least not yet. wimb only wanted to know whether it booted, as little as that.







Also tagged with one or more of these keywords: ramdisk, grub4dos, wimlib, svbus, windows 10, ssd, usb, wim, vhd, wimboot

2 user(s) are reading this topic

0 members, 2 guests, 0 anonymous users