163 replies to this topic

[milindsmart]

Will reply to rest of stuff in a bit, but PROCESS1_INITIALIZATION_ERROR is (to me) one of the most serious BSODs, where seriousness is an indicator of how deep the problem is. e.g. driver crashes are not as serious as kernel issues, and my experience is that this particular BSOD comes from about as deep into the CORE OS as it gets....

 

Still, Windows has become a lot more modular these days, so please don't give up hope, because I certainly ain't.

[Wonko the Sane]

One last comment - just wimbooted a full Windows 8.1 Update 1 using Windows 7 bootmgr. My first attempt failed - changed the BCD store and it worked! The BCD store was the same one I used for Winre and was created using the following batch -

Sure , but HOW did you change it?
And HOW it was made before?
I.e. provide some details on the "before" (not working) and the "after" (working), otherwise it will be difficult to understand the relevance (if any) of the \boot\BCD.
 
I remember a similar issue (troubles with a Windows 7 vs. Windows 8 made \boot\BCD, maybe this is connected (directly or loosely)?
 
Here:
http://www.911cd.net...showtopic=25596
http://www.911cd.net...ndpost&p=175030
 

Wonko

[erwan.l]

About testing, for the record, I am using grub4dos to boot different bootmgr.

It might be deviating from a standard ms boot sequence (ms mbr->ms pbr->bootmgr->...) but it eases tests a lot.

In my grub4dos menu I can choose from different bootmgr (win8.0, win81, win81u1, etc ...).

 

/Erwan

[misty]

This is just a brief message whlst I wait for my wife to finish packing - time wise this is an unpredictable endeavour!

In regards to the original BCD store - this was the default store created when Windows 8.1 Update 1 was installed to an existing partition - this partition contains the boot files, operating system files and WinRE.

When bootmgr was replaced with an older version the OS failed to wimboot. Replacing the BCD store with one created using the batch file I posted earlier resolved this issue - note that the paths were hardcoded in this batch. I wasn't sure it would work due to use of the following...

/set {GUID} winpe Yes

...in the batch. It did.

Bye for now.

[Wonko the Sane]

This is just a brief message whlst I wait for my wife to finish packing - time wise this is an unpredictable endeavour!

... and most probably you will pay for this comment (if and when she finds it)
 

Bye for now.

Have a nice trip/stay etc.


Wonko

[erwan.l]

Just like Misty, I can confirm I can boot my wimboot O.S with bootmgr from Win8.1preU1.

 

When I try to boot that same O.S with bootmgr from Win7SP1 I get a Winload.exe - 0xc0000428 (windows cannot verify the digital signature for this file).

I am not sure this is related to my O.S being wimbooted.

I could try regenerating my BCD thus like Misty related.

[synchronicity]

Simon asked me to comment here, so I'll try to answer some questions.

 

All "links" on NTFS, other than hard links, are implemented using feature called "reparse points".  A reparse point is a file that contains a "reparse" data stream.  The reparse data stream contains a reparse tag, which identifies a filter driver through which access to the file will be redirected.  The reparse data stream also contains up to 16384 bytes of data which can be interpreted in any way by the filter driver.  The common reparse tags are:

 

0xA0000003: Junction / mount point

0xA000000C: Symbolic link

0x80000017: Windows Overlay Filesystem (WOF) redirection

 

The reparse data for WOF, which is normally interpreted by wof.sys, contains a further indirection, to a specific overlay "provider" implementation.  The code for this might be in wof.sys itself; I'm not sure.  Either way, for the WIM provider, the reparse data contains:

 

- Data source ID (volume unique) of the WIM file.  wof.sys resolves this to an actual WIM file using \System Volume Information\WimOverlay.dat, which contains a sequence of per-WIM binary data structures.

- Byte offset, uncompressed size, and compressed size of the file data in the WIM file.

- Some other fields, which seem somewhat redundant.

 

You can look at include/wimlib/wof.h in the wimlib sources if you want to see the actual declarations.  The total size of each WOF reparse data stream is something like 80 or 90 bytes.  That's what you'll be copying if you actually image the volume *without* the WOF driver running.  (By the way, the reparse data and WimOverlay structs are reverse engineered --- MS doesn't document them.)

 

So yes, strictly speaking "WIMBoot" isn't the right name for this thing.  "WIMBoot" is just a name that MS has assigned to a use of the ability to externally back files using a WIM archive.

 

Another question that came up is the role of Wof.sys in the boot process.  As far as I can tell, it is indeed a real Windows NT driver, so it can only execute after ntoskrnl.exe has been loaded.  This brings up an interesting question.  Since ntoskrnl.exe is not listed in [PrepopulateList] in WimBootCompress.ini, both wimlib and WIMGAPI will extract it as externally backed file, which one would think cannot be read without wof.sys, which cannot be loaded until ntfskrnl.exe is loaded.  So what is going on?

 

The answer seems to be that winload.exe (*and* winresume.exe) both implement the same logic as wof.sys, and can load files from WIM archives through the WOF-tagged reparse points.  (And yes, that means that winload.exe, winresume.exe, wof.sys, and wimgapi.dll all contain full XPRESS and LZX decompressors.  It wouldn't be Windows without software bloat!)

 

The consequences of this are that (a) Windows is designed poorly, and ( b ) normally, not very many files need to be listed in [PrepopulateList] in WimBootCompress.ini, and ( c ) if hypothetically you were to try to use Wof.sys on an earlier version of Windows, if this is even possible, then you would at least have to list in [PrepopulateList] *all* the files that winload.exe actually loads.

 

Coincidentally, the fact that ntoskrnl.exe is listed in [CompressionExclusionList] is irrelevant.  Both wimlib and WIMGAPI ignore this section.

 

Probably bootmgr is not particularly important in this process, other than as being the loader of winload.exe or winresume.exe.

 

[synchronicity]

I probably should add something else about accessing externally backed files.

 

When WOF is running, you can issue FSCTL_GET_EXTERNAL_BACKING using DeviceIoControl().  For WOF-tagged reparse points, this essentially gives you a stripped-down version of the reparse data stream.  MS documents this ioctl at  http://msdn.microsof...1(v=vs.85).aspx.  The "wimboot-analyze" utility I wrote uses this to determine if a file is externally backed or not.

 

When WOF is *not* running --- including when accessing the NTFS volume using NTFS-3g, from non-Windows operating systems --- applications will just see an inaccessible reparse point.  Hypothetically, an appropriately equipped application could read the reparse data stream, resolve the data source ID to a WIM file using "\System Volume Information\WimOverlay.dat", and read the uncompressed data from the WIM file.  I don't believe any applications actually do this (other than wof.sys, winloader.exe, and winresume.exe...).  wimlib does, however, provide some pieces of what would, hypothetically, be necessary to do so.

 

Also, it should be possible to independently implement a real kernel-mode driver compatible with WOF, if for some reason someone really wanted to do so.  One could, hypothetically, also choose their own reparse tag, and implement something similar without needing to be compatible with WOF, or even use WIM files at all.

[Wonko the Sane]

Good.

 

More or less, this could mean, that if ntoskernel.exe is taken outside of the .wim and made "flat" in the NTFS filesystem, there are chances that earlier versions  may  still boot as "wimboot", let's say in a "98.78% wimboot" setup.

 

AFAICU this means however:

http://reboot.pro/to...mboot/?p=187022

 

that the "new" winload.exe and winresume.exe (which are "flat" since they are inside [PrepopulateList]) have code to load ntoskrnl.exe (which is inside the .wim) BUT cannot decompress it (since ntoskrnl.exe is in [CompressionExclusionList]), i.e. it seems like winload.exe (and winresume.exe) can parse the "WOF" link, and access the .wim contents, but have not decompression in them. (or maybe this is a "by design" choice to - say - speed up booting by avoiding the decompression step).

 

As hinted before I see the whole stuff more as a *kind of* unionFS (or *not on same volume hardlinks* or *granular mountpoints*) than anything else, i.e. files are in different "places" (in the case of the .wim on another partition even on another volume) but are exposed to the system as if they were all inside the same volume, and as such the approach (once and if "generalized" and taken outside of the "strict Win8.1u1 wimboot" may provide the base for every kind of nice trick.

 

The "leechPE" I only hinted before could be such a project.

 

You simply create a "special" volume where only the PE specific files actually reside, (including the PE Registry and the tools/apps that you don't want to install to the main OS) while all the rest (which are "common between the PE and the installed OS" are "leeched" from the OS itself through this kind of cross-volume (and possibly also cross-disk) links.

 

Wonko

[simonking]

I'm going to go off-topic here with just one small question/voting poll:

http://www.zipmagic....ic.php?f=4&t=48

I thought I'd ask for the insight of jaclaz, most especially, before I make up my mind on this one...as he knows, I'm pretty touchy about censorship. I find it amazing that even my response on the Wikipedia talk page was not allowed to sit as-is, but censored entirely.

[erwan.l]

Good.

 

More or less, this could mean, that if ntoskernel.exe is taken outside of the .wim and made "flat" in the NTFS filesystem, there are chances that earlier versions  may  still boot as "wimboot", let's say in a "98.78% wimboot" setup.

 

AFAICU this means however:

http://reboot.pro/to...mboot/?p=187022

 

that the "new" winload.exe and winresume.exe (which are "flat" since they are inside [PrepopulateList]) have code to load ntoskrnl.exe (which is inside the .wim) BUT cannot decompress it (since ntoskrnl.exe is in [CompressionExclusionList]), i.e. it seems like winload.exe (and winresume.exe) can parse the "WOF" link, and access the .wim contents, but have not decompression in them. (or maybe this is a "by design" choice to - say - speed up booting by avoiding the decompression step).

 

 

 

Sure to be sure I got it right : I thought Synchronicity mentionned that

-winload and winresume both handled xpress and lxz decompressors

-the CompressionExclusionList is actually ignored

 

but reading Wonko's last thread above, I now read the opposite?

 

Also, why did MS bothered to include extra code in winload/winresume?

Only to be able to include ntoskrnl in the wim file (i doubt it) or are there actually lots of file (and therefore a potential benefit of disk space) needed by winload/winresume?

 

Must be otherwise the simplest path would have been : mbr->pbr->bootmgr->bcd->winload->ntoskrnl->wimboot is not it?

[Wonko the Sane]

Sure, at the moment there is not consensus on this (but - particularly myself - we are still in the thinking out loud phase), we are actually saying opposite things.

 

What I believe (actually what common sense tells me ) would be that winload.exe (or winresume.exe) may be:

1. not capable to understand "winboot links" AND not capable to decompress a  XPRESS and LZX compressed file
2. not capable to understand "winboot links" BUT capable to decompress a  XPRESS and LZX compressed file
3. capable to understand "winboot links" BUT not capable to decompress a  XPRESS and LZX compressed file
4. capable to understand "winboot links" AND capable to decompress a  XPRESS and LZX compressed file

IF #1, then the ntoskrnl.exe MUST be on the filesystem (and NOT inside the.wim)

IF #2, then the ntoskrnl.exe MUST be on the filesystem (and NOT inside the.wim), and the XPRES/LZX compression capabilities of winload.exe/winresume.exe are unrelated to ntoskrnl.exe loading

IF #3, then the ntoskrnl.exe CAN be on the filesystem (but CAN ALSO be inside the.wim) BUT it is NEEDED that it is not compressed

IF #4, then the ntoskrnl.exe CAN be on the filesystem (but CAN ALSO be inside the.wim) AND it CAN be compressed alright

 

Since - right now - the ntoskrnl.exe seems to have been put inside the .wim, BUT including it in a "no compression list, it is more logical that the case is #3.

 

Of course, whenever MS is involved, logical reasoning has not the same strength as in other fields. 

 

What Synchronicity posted (as I understand it) is  that the case is #4 instead.

 

But I wouldn't be at all surprised if in a previous version of winload.exe (winresume.exe) it WAS a case #3, then a later version included compression support (and thus IS NOW a case #4) BUT the "[CompressionExclusionList]" remained because noone thought of removing it.

 

No matter what is the case it would maybe possible to move (on previous OS versions, let's say Windows 7) the ntoskrnl.exe from inside the .wim and place it in the "normal" filesystem, and "benefit" from the WOF driver for the rest of booting (that was my reference to a fictional "98.78% wimboot" setup).

 

@Simon

Why don't you start a new thread about your Wikipedia issue?

Really, it doesn't belong here and it might well hijack/disrupt the present topic (right now when it is really interesting).

Or are you expecting us to join your board  just to comment on that thread there?

 

 

Wonko

[simonking]

Agreed, here's the new thread: http://reboot.pro/to...rship/?p=187123.

[milindsmart]

I am short on time and am stuck with another experiment, so can someone actually verify that replacing wimboot-reparse-tagged files with their flat equivalent results in no change, given a working WIMBoot instance? Specifically, replacing ntoskrnl.exe with it's flat version should result in no change... Does it?

[erwan.l]

I am short on time and am stuck with another experiment, so can someone actually verify that replacing wimboot-reparse-tagged files with their flat equivalent results in no change, given a working WIMBoot instance? Specifically, replacing ntoskrnl.exe with it's flat version should result in no change... Does it?

 

I'll try to experiment this.

 

It means booting

-with a winpe that can deal with reparse points (as reparse point cannot be dealt with otherwise),

-copy paste ntoskrnl to a temp folder (which at this point should produce a flat file),

-delete the original ntoskrnl,

-replace it with the flat version from temp folder

[erwan.l]

I'll try to experiment this.

 

It means booting

-with a winpe that can deal with reparse points (as reparse point cannot be dealt with otherwise),

-copy paste ntoskrnl to a temp folder (which at this point should produce a flat file),

-delete the original ntoskrnl,

-replace it with the flat version from temp folder

 

Test performed and system boots ok.

 

I suspect thus that when modifying my filesystem, wimoverlay.day got updated so it should be no surprise then that it still works.

 

For the record, I checked the "size vs size on disk" before and after : was clearly a reparse point before, was clearly a flat file after (size = size on disk).

[milindsmart]

Test performed and system boots ok.

I suspect thus that when modifying my filesystem, wimoverlay.day got updated so it should be no surprise then that it still works.

Nice to know, one headache less.

But why would wimoverlay need to be modified to reflect this change? Surely, it looks up the wimoverlay.DAT file AFTER encountering this reparse point..? Or are you saying that if there exists a wim-backing entry in the dat file, AND the file itself is flat, the OS will choke? That sounds excessively intentionally self crippling.

[synchronicity]

winload.exe and winresume.exe also load a number of other low-level device drivers.  Start Windows in "safe mode" --- the list of files loaded by winload.exe will scroll by.   The kernel (ntoskrnl.exe) needs to be able to access the filesystem before it can take control, so it needs the appropriate disk and filesystem drivers to be loaded for it.

 

If winload.exe and winresume.exe did not contain the ability to read files that are externally backed in a WIM archive, possibly compressed with XPRESS or LZX compression, then all these low-level drivers would need to be listed in [PrepopulateList].  But they are not, so winload.exe and winresume.exe necessarily have this ability.

 

(By the way: on a Linux system, these low-level drivers would be in a single compressed cpio archive which is provided by the bootloader and unpacked by the kernel.  Potentially a cleaner way to do things.) 

 

WimOverlay.dat is just a list of which WIM files are registered as external backing sources on that volume.  The WIM files themselves need not be located on the same volume --- they are actually identified by the physical partition and path relative to it.  Deleting or adding externally backed files does not, by itself, change WimOverlay.dat.  But in order to even add an externally backed file, you need to know the data source ID, which you can only get by registering a WIM file in WimOverlay.dat (or via the FSCTL_ADD_OVERLAY ioctl, when WOF is running).

[Wonko the Sane]

 

The answer seems to be that winload.exe (*and* winresume.exe) both implement the same logic as wof.sys, and can load files from WIM archives through the WOF-tagged reparse points.  (And yes, that means that winload.exe, winresume.exe, wof.sys, and wimgapi.dll all contain full XPRESS and LZX decompressors.  It wouldn't be Windows without software bloat!)

 

The consequences of this are that (a) Windows is designed poorly, and ( b ) normally, not very many files need to be listed in [PrepopulateList] in WimBootCompress.ini, and ( c ) if hypothetically you were to try to use Wof.sys on an earlier version of Windows, if this is even possible, then you would at least have to list in [PrepopulateList] *all* the files that winload.exe actually loads.

 

Coincidentally, the fact that ntoskrnl.exe is listed in [CompressionExclusionList] is irrelevant.  Both wimlib and WIMGAPI ignore this section.

 

 

 

 

winload.exe and winresume.exe also load a number of other low-level device drivers.  Start Windows in "safe mode" --- the list of files loaded by winload.exe will scroll by.   The kernel (ntoskrnl.exe) needs to be able to access the filesystem before it can take control, so it needs the appropriate disk and filesystem drivers to be loaded for it.

 

So, next step would be to make a list of these files and take them "outside of the .wim" saving them as "flat" in the filesystem, right?

 

Wonko

[synchronicity]

So, next step would be to make a list of these files and take them "outside of the .wim" saving them as "flat" in the filesystem, right?

 

If the goal is to use WOF on an older version of Windows, then yes that would be necessary --- keeping in mind that the actual files which winload.exe loads on a given system might depend on the hardware.

 

I would also not be surprised if wof.sys simply does not run with Windows 7, due to kernel API changes.  (Although maybe I am just used to Linux, where they (for good reasons) don't guarantee a stable API for drivers. )  And is it possible for the driver to be signed only for Windows 8.1 or later, or is that not how signing works on Windows?

 

I guess someone will just have to try it.

 

It's also conceivably possible that you could take a Windows 8.1 winload.exe and use it with Windows 7...

[Wonko the Sane]

 

It's also conceivably possible that you could take a Windows 8.1 winload.exe and use it with Windows 7...

 

Yep but this would probably vanify most of the utility of what I had in (my perverted ,)) mind as "final" goal. (which more or less was "porting back to earlier releases of windows a large part of the "winboot" mechainism").

 

I mean, while I find conceivable (and possible) that someone will be able (before or later) to write an independent replacement for the WOF.SYS driver (just like you did with the wim-lib ) I find very improbable (if possible at all) that someone may be willing (or being capable of) writing a winload.exe replacement, as I believe the former to be a relatively simple "file system filter driver" i.e. something that is documented and for which similar drivers have already been developed, whilst the latter is still an obscure, black, "box" inside which the good MS guys may have put "everything" and the contrary of it.

 

Wonko

[simonking]

I believe I've finally nailed the cause of my slow boot experiences, documented probably everywhere online now.

 

As our friend here responded, he's never seen such a slow performing boot loader, and at an atomic level at that:

 

http://www.msfn.org/...-slowly/page-14

 

This also corroborates my other findings - namely, that the GUI boot screens of 8.x would be replaced by a Win7 style text boot screen.

Clearly, one of those partition management tools is installing its own boot loader, after a disk clone operation, and only sometimes.

 

Now then, the question is - how do I revert to the default boot loader on Windows 8.1?

 

For that matter, how does one change the boot loader on older OS's?

 

One of my NT 5.x kernel OS's, for example, after running Paragon's Partition Alignment tool, started displaying a blinking cursor for 20 times.

 

Slows down the boot - not 5 minutes or 30 minutes, but still annoying. Now that I've nailed it down finally, I'd like to cure them all!

[TheHive]

Can cdob or anyone else in this thread can help in finding a solution on the following thread.Thanks!

http://reboot.pro/to...ackup/?p=191375

[alacran]

Well, just for the fun I made this:

 

60 GB USB mechanical HDD MBR partitioned:

First partition: Primary  active 1GB Fat 32 (Large for future test * ): Boot files for MBR and EFI/UEFI

Second partition: Primary 20 GB NTFS for Windows.

Third partition: Primary 37.x GB NTFS for install.wim (wimboot mode)

 

First Test: Dual Boot MBR & UEFI

 

Using synchronicity wimlib-imagex v1.80: http://reboot.pro/to...implementation/

and ReTokener tool wimlib_clc_150319: http://reboot.pro/to...ta/#entry191777

 

Installed 8.1u1x86 to second partition, boot files on first partition.

 

I captured a 8.1u1x86 system fully updated and syspreped (from second partition) in wimboot mode to install.wim image (located in third partition, same HDD).

 

Then formated second partition and apply captured install.wim wimboot mode to it.

 

It booted without any problem in MBR/legacy, also it is seen as a boot option in UEFI (haven't run it but I see no problem).

 

Second test: 5 Booting Options = Dual UEFI Boot (x86 & x64), & Triple MBR Boot (x86, x64 & grubfordos)

 

Later I formated first, second  and third partition and, using install.wim x64 and install.wim x86, directly from install Iso's, just extracted to third partition, each one in a diferent folder. Then created in second partition two 5 GB VHD's first for 8.1x64 and second for 8.1x86, both wimboot mode install.

 

To make the wimboot install to VHD's I used JFX tool WinNTSetup v3.77 & libwim-15.dll from wimlib-imagex v1.80 (in WinNTSetup3.77\Tools\x86 or x64 \wimlib), using as source each image in third partition, and selecting first partition as boot one, (WinNTSetup makes all changes you need for booting x86 & x64 in MBR or UEFI, also the required VHD's in the selected partition).

 

Later after add grubfordos to the BCD using BootIce I can run x86 or x64 and also grubfordos (to run some Iso's) just selecting it from the first window, when in MBR/legacy.

Or x86 and x64 in UEFI (if machine can use both, some do, other don't), but NOT grubfordos in this case unless deactivating UEFI and Segure Boot.

 

EDIT:

Links added.

 

I'm sure it is also possible to make a dual full install (not wimboot), if you install x64 in one partition and x86 in the next one, as long as the first partition (Fat32 format) has the boot files for both.

[alacran]

Backup with 7-Zip v9.38 Beta

 

After my second test, I uncheck "Hide Protected System Files" and Check "Show hidden files, folders and disks" in "Files Options on my main OS on PC.

 

Then plug the USB Disk and using 7-Zip v9.38 Beta made a Maximum Compress copy of all files and folders on First partition (Boot) and also of each VHD's located on second partition, I didn't do it on third partition because .wim files are already compressed.

 

This is my backup copied on my main drive on PC (Better compression than Ghost).

 

To make sure I formated first and second drives one by one, on the USB HDD and deleted install.wim's on third partition, and just copy again both install.wim to third partition (same file locatios as before), and extracting the 7-zip content of primary partition and both 7-zip files with VHD's to second partition.  Let my tell you just in a matter of minutes, all came to live again when booted.