163 replies to this topic

[simonking]

In contrast, if you run the free tool I posted here, it takes under a minute to extract and update WinRE 5.0 to WinRE 5.1

[Wonko the Sane]

ok I get it.

 

indeed, my procedure was the following :

-build a winpe (any)

-boot from there

-use wimlib to image a win81u1 system using the --wimboot flag

-format system drive

-copy the captured wim to that formatted system drive

-apply the captured wim to that system drive

 

benefits were :

-significant disk space saving

-in case system crash, easily re apply the captured wim (with a bootmanager including a ready to use winpe for instance)

I still don't get it.

You are not using the "recommended" specific "Images partition" as in:

http://blogs.windows...e-boot-wimboot/

http://technet.micro...y/dn605112.aspx

 

 

Still  - provided that there is enough available space on another volume/device - cannot the "Wimboot bootng system" be "copied" (to "flat") on this space and then recaptured (with the /wimboot option) from this "flat copy"? <- this is the part I have issues in understanding why it is not possible

 

Wonko

[simonking]

You cannot recapture an already WIMBoot'ed partition, when the WoF driver is not installed. This is because the files will not be served properly as I have already indicated elsewhere on this forum. Eric is aware of the issue which affects WIMLIB and WIMGAPI (DISM) equally; this is not a bug in either, but simply inability to interpret the file system on part of the OS, as it were, when the driver is missing.

[Wonko the Sane]

Yes, but I was not asking about re-capturing the "wimbooted" partition, I was asking if there is not a way to copy (from the OS on the "wimbooted" partition - that of course has the WOF driver working) the partition contents to another volume (or disk/volume image) "flat", in such a way that capturing this "flat" copy is possible without the WOF driver.

 

As said, it is not like I am recommending or finding of any practical use such an approach, only asking if (and why) it should not be possible.

 

Wonko

[simonking]

I suppose you could do the flat copy even on an OS without the WoF driver, assuming it is doing a byte-by-byte copy of the partition.

[milindsmart]

But that will not give you a non-wimboot/whole/non-pointer set of files. It's just another copy of the pointer files... What Wonko the Sane is asking about, is whether the wimboot OS volume can be copied into a non-wimboot flat normal OS volume.

Which I assume will happen if a normal copy operation is executed from an OS instance with the WoF driver.

Edited by milindsmart, 03 September 2014 - 06:44 PM.

[erwan.l]

I did not

 

Yes, but I was not asking about re-capturing the "wimbooted" partition, I was asking if there is not a way to copy (from the OS on the "wimbooted" partition - that of course has the WOF driver working) the partition contents to another volume (or disk/volume image) "flat", in such a way that capturing this "flat" copy is possible without the WOF driver.

 

As said, it is not like I am recommending or finding of any practical use such an approach, only asking if (and why) it should not be possible.

 

Wonko

 

I am curious : if you use a DD like software (i.e 1:1 copy), I would say that the filesystem driver(s) will hide the the wimboot complexity.

Could depend whether you target the physical drive or the logical drive.

[Wonko the Sane]

No, of course not a dd-like copy on the sectors (physicaldrive vs. logicaldrive won't make a difference), but using COPY, XCOPY, ROBOCOPY or the like, including dd or the dsfok tools operating on FILES, i.e copying commands operating on filesystem objects it should be possible. 

 

A dd copy on physical or logical will make a "clone" or "forensic sound image", if the source is "wimbooted", the target will be "as wimbooted as the source", it will actually be identical.

 

 

As an example (only seemingly unrelated ) most (please read as "all but a few specific ones") "copy" programs cannot (by design) manage "properly" sparse files, i.e. if the source is sparse, the target is NOT sparse but it is expanded to "full size", a specific tool like:

http://www.flexhex.c...rse-files.phtml

is needed.

 

Or, on another example, if you use 7-zip to apply the install.wim on a FAT32 volume  all hardlinks/softlinks, etc, are "resolved":

http://reboot.pro/to...rdlinked-files/

 

Wonko

[milindsmart]

No, of course not a dd-like copy on the sectors (physicaldrive vs. logicaldrive won't make a difference),[...]
 
A dd copy on physical or logical will make a "clone" or "forensic sound image", if the source is "wimbooted", the target will be "as wimbooted as the source", it will actually be identical.

Exactly. (I actually quoted you to disagree, but realized the truth after I typed out "No" )

To reiterate, unless you're addressing FILES, there will be no reparsing/"magic" going on. As relevant to dd, at the sector-level there is no magic/spoofing going on, like with grub4dos or the like... The disk driver will happily give you sectors amounting to zero-length files

Edited by milindsmart, 04 September 2014 - 12:25 PM.

[erwan.l]

Oki that answers my doubts then.

I was wondering at which level the wof driver was acting.

 

I got my answer : at the filesystem level.

 

So a 1:1 DD clone of my 20gb wimboot install system will probably result in a useless 500mb or so image file.

 

So we could use a software that do it then

[milindsmart]

I was wondering at which level the wof driver was acting.
 
I got my answer : at the filesystem level.

Turns out, "WoF" actually stands for "Windows Overlay File System Filter", and it was just added in Windows 8.1.

So a 1:1 DD clone of my 20gb wimboot install system will probably result in a useless 500mb or so image file.

Nice way of putting it.

Edited by milindsmart, 04 September 2014 - 01:41 PM.

[Wonko the Sane]

So a 1:1 DD clone of my 20gb wimboot install system will probably result in a useless 500mb or so image file.

 

NOT really (or again we are misunderstanding each other ).

A 1:1 DD clone of your 20 Gb (wimboot installed) volume will result in a 20 Gb (wimboot installed) cloned volume (otherwise it won't be a 1:1 DD clone, but a 1:40 image) IDENTICAL to the source.

 

Wonko

[simonking]

Actually, that depends on where the WIM file is located.

 

If the WIM file is inside the same partition (like DoubleSpace does it), the clone may be useful.

 

If its in an external partition (like the tutorials out there seem to do it), the clone may be really small, but totally useless without the backing WIM.

[erwan.l]

Actually, that depends on where the WIM file is located.

 

If the WIM file is inside the same partition (like DoubleSpace does it), the clone may be useful.

 

If its in an external partition (like the tutorials out there seem to do it), the clone may be really small, but totally useless without the backing WIM.

 That was indeed the missing information : whether or not the wim file stands on the same drive.

If yes, the clone remains applicable.

If not, your clone will be waste.

[Wonko the Sane]

 That was indeed the missing information : whether or not the wim file stands on the same drive.

If yes, the clone remains applicable.

If not, your clone will be waste.

I wouldn't be completely sure about this - as long as the volume on which the referenced wim is still "accessible" on the machine.

 

I mean I don't know (and I believe noone yet, exception made, maybe, for Synchronicity knows) how exactly the "links" in the filesystem are created.

Is the volume containing the .wim required to be on the same disk drive? <- no, as as seen it is actually advised to use another volume for the images only

Is this volume identified by (say) Disk Signature + Offset? <- on "BIOS" machines or more exactly on "MBR disks"

Is this volume identified by (say) UUID? <- on "UEFI" machines or more exactly on "GPT disks"

 

Further question:

 

Would it be possible to "abuse" of this structure to create a "leechPE" of some kind? (or - if you prefer - a "UnionFS-like" setup)

 

Wonko

[milindsmart]

I have most of the information in the WimOverlay.dat files figured out by running some basic tests and examining the binary data in the file. For each WIM archive registered as a data source on the volume, it contains:

- Type of WIM ("OS" or "Not OS", whatever that means)
- Index of image in WIM that was applied
- GUID of WIM file
- Disk format (MBR or GPT) of disk containing WIM file
- Start sector of MBR partition or GUID of GPT partition containing WIM file
- MBR disk ID or GPT disk GUID of disk containing WIM file
- Absolute path to WIM file, excluding drive letter

So, the "WIMBoot pointer files" should still work if the drive letter containing the WIM file is changed, since this file apparently doesn't store the drive letter but rather the location of the partition.

The answers to your 2 questions are both "Yes". Further, misty had once booted successfully with the WIM being in another disk altogether.

That was indeed the missing information : whether or not the wim file stands on the same drive.
If yes, the clone remains applicable.
If not, your clone will be waste.

I argue that since you want a CLONE, the expectation is that it exists independently of the original arrangements. So the clone is useless irrespective of if the WIM file is accessible : It's still a bunch of pointer files, no longer a clone of the volume(as in a full set of files).

[Wonko the Sane]

Sure , the first two were mostly rhetorical questions, meaning that I know that the answer is "Yes (more or less)", but the accent was on the earlier "exactly".

 

I believe (but as said from this to having an exact, detailed understanding of the behaviour there is still yet a gap) that the WIMBOOT approach (once you stop calling it WIMBOOT - i.e. focusing only on the bootability of the system, and you start calling it more generically Windows Overlay Filesystem ) is not much different conceptually from hardlinks/mountpoints/reparse points/softlinks, and similar, with the not so trifling difference that the "target" of the link is inside a compressed archive.

 

Though similar in concept there are various (sometimes little, sometimes big) differences between the "features" of the mentioned "links", and it is entirely possible that the links that the overlay filesystem driver uses have more (and hopefully better/more useful) features then what we might (yet) now/understand.

 

As an example an only seemingly unrelated old experiment:

http://www.911cd.net...showtopic=23626

it came out that hardlinks (which are limited to "same volume") work (and thus are implemented right at the NTFS filesystem level) whilst "junction points" are "one level higher" and need a "filter driver" (or *something* like that) to work at boot time.

See also:

http://schinagl.priv...lextension.html

Symbolic links "filter driver" for XP:

http://schinagl.priv...nksforwindowsxp

 

The Overlay Filesystem driver must be not-so-different from the above, but is *somehow* integrated in boot time sequence (it is possible that the Win 8.1u1 BOOTMGR includes a native counterpart to it )

 

I believe that once the mechanism is clear, I see no reason why the filesystem filter driver can not be replicated (for pre-8.1u1 OSes) or the structure integrated in tools *like* (example):

http://www.pismotechnic.com/pfm/

or Dokan (or similar)

 

Wonko

[simonking]

Oh that would be amazing. Maybe I should open up a project for that? Then DoubleSpace would run natively on Windows 7.x and maybe even older?

[Wonko the Sane]

Oh that would be amazing. Maybe I should open up a project for that? Then DoubleSpace would run natively on Windows 7.x and maybe even older?

 

Well, there would be still the not-so-trifling issue of rewriting a suitable BOOTMGR with the integrated boot-time functionality, but maybe - just maybe - something loosely similar to the "Xp Kansas City Shuffle" would be possible with just the filter driver.

 

Wonko

[erwan.l]

Actually, you can decide which files get "wimbooted" (i.e pointed at) and which ones do not.

Hence you may not have to rewrite a bootmgr but "simply" a filesystem driver.

Could be that we could re use the skeleton part of some of the drivers we have on this forum like winvblock or firadisk.

 

Just thinking loud thus : I may overlook some basics but I am sure Wonko will correct me

[milindsmart]

Is winload also WIMoverlaid? If not, then bootmgr is likely not special at all. I suspect wof.sys will be part of the critical device database driver list... So maybe : http://reboot.pro/to...evice-database/

Can someone check if this is the case? That would be a start.

If not, that is if bootmgr is in fact special, then again I'm sure we can use the same bootmgr present in WIMBooted systems. (Skip the legal aspects of redistribution until this is actually viable, Wonko)

BTW, I don't remember if anyone has posted a full list of files that are flat in the WIMBoot windows partition... If no one has, I think that is vital to proceed.

Edited by milindsmart, 04 September 2014 - 05:56 PM.

[Wonko the Sane]

Just thinking loud thus : I may overlook some basics but I am sure Wonko will correct me 

Lately on this thread everyone is mostly thinking out loud , which is IMHO a good thing as some ideas may come out of this.

 

But it should be easy to understand if BOOTMGR is involved in it *somehow*, just replace on a wimbooted install the BOOTMGR with a copy of the Vista, 7, 8 or 8.1 before update 1 (or create a set of "Vista boot floppies" with those) and see what happens, MIsty attempted *something* loosely similar but failed:

http://reboot.pro/to...mboot/?p=183968

(using the same Registry entries alacran posted and that were confirmed to be "right" by the use simonking made of them)

 

Since it was a half-hearted (besides possibly half-@§§ed ) attempt (no offence whatever intended to the nice try Misty made, of course) I would consider the negative result as a non-definitive one (still seeing the glass half full ).

 

The proposed experiment (which is more or less the opposite of the one Misty carried), using an older BOOTMGR on the new OS, might give (even if failed) some more hints on what exactly is involved.

 

Whilst (purely theoretical estimation ) the chances of a Windows 8.1u1 "wimbooted" install booting through a Vista or 7 BOOTMGR are more or less around 0%, I would rate the probabilities of it booting with a 8 or 8.1preU1 BOOTMGR as within the range of "possible". (of course IF there is not a "special" provision in BOOTMGR for the WOF loading)

 

@milindsmart

It all depends IMHO (and i have not at the moment a suitable install/machine to have a look at it) on which files are "really-really" on the volume and which ones are instead a link of some kind to files residing in the .wim (shouldn't make a difference if the .wim is on the same volume or on another one).

 

Wonko

[erwan.l]

Looking at the wimbootcompress.ini, I would tend to say (need to test it) that the bootmgr may be not involved in the wimboot process.

Indeed the MBR->PBR->bootmgr->bcd->winload->ntoskrnl sequence is executed outside of the WIM filesystem.

The wof driver is probably the first core device driver loaded.

The system registry is also a plain file (not a pointer).

 

You have a wimbootcompress.ini (where you specifiy files to skip) here.

 

[CompressionExclusionList] contains files that should not be compressed during the apply stage)

[CompressionExclusionList]
ntoskrnl.exe

[PrepopulateList]  contains files that will be extracted as normal files (i.e not as a pointer) during the restore stage.

It does contain winload amongst other files.

[PrepopulateList]
\bootmgr
\boot\BCD
*winload.*
*winresume.*
wof.sys
\Windows\System32\Config\SYSTEM

[Wonko the Sane]

you have a wimbootcompress.ini (where you specifiy files to skip) here.

Yep , and more or less or "approximately" the list represents *any* file that could not be "theoretically" Read Only or, seen the other way round, *any* file that "belongs" to the Windows Operating System AND that by design is changed/modified at boot time or in "normal operation" of the OS OR that needs to be accessed in the REALLY early stage of booting.

I.e., sets aside the [Exclusionlist], what remains (and get a "special treatment") is just:

[CompressionExclusionList]
ntoskrnl.exe

[PrepopulateList]
\bootmgr
\boot\BCD
*winload.*
*winresume.*
wof.sys
\Windows\System32\Config\SYSTEM

 

 

What actually happens is to be found through experiments.

 

For all we (or at least myself) know the normal boot sequence is (more or less):

1. BIOS/EFI/*whatever* loading BOOTMGR (or it's corresponding EFI file) that loads:
2. \boot\BCD (or corresponding EFI BCD) and (the BOOTMGR) accesses the Registry and loads:
3. winload.exe that loads:
4. ntoskrnl.exe that loads;
5.  the "rest" of the OS

now, as I see it, you have to insert *somewhere* a:

• Wof.sys that loads:

and append a number of "through WOF.SYS" on each following item.

You can insert the:

• Wof.sys that loads:

a. between 2 and 3,

or

b. between 3 and 4

or

c. between 4 and 5

 

At least IF the "insertion point"  is a. it depends on what exactly the BOOTMGR looks for in the Registry (or in the BCD)  

 

What may be "good news" is that the FLTMGR service (which needs to be running in "normal operations") is not seemingly among the "special treatment" filelist.

 

 

Wonko

[erwan.l]

I thought that only ntoskrnl could load a driver?

I was conforted in that idea by the fact that the system registry hive is also a plain file as indeed this hive contains the list of drivers to be loaded by ntoskrnl.

 

So I would bet on between 4 and 5 in your previous post