Jump to content











Photo
- - - - -

ramloading Windows 10 vhd-core

windows linux window10 vhd ramloading

  • Please log in to reply
188 replies to this topic

#51 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 11 February 2019 - 10:05 PM

@wonko,

I have just checked the files and folders remaining on c: as far as my experience is concerned. What I found that would be relevant to the point of moving them off of the vhd is the content of system32: almost 2gb of files; as for c:\windows, even if all the files could be safely moved off, they would not amount to anything even 1/4 as substantial to justify their being anywhere else (just a few megs).



#52 quarky42

quarky42

    Member

  • Members
  • 38 posts
  •  
    United States

Posted 11 February 2019 - 10:22 PM

Firadisk is working with the new grub4dos-0.4.6a-2018-12-23 on my old Win10x64 12.8GB image with the addition of the --top flag.  Thanks, Wonko!

 

I change the image over to my new Win10x64 16.0GB image and get a INACCESSIBLE BOOT DEVICE message.  I've gone through all the advice given in the thread and done a fair amount more reading.   It looks like the size of the image was the culprit.  I think this bit me in the butt a little over a year ago when I was first trying to get this working.  (That and I had problems even then finding the right version of Grub4DOS)

 

Thanks to the help and advice in this thread, I successfully booted my 16.0 GB image by removing Firadisk and installing SVBus V1.1.  I sincerely thank you for the help, Wonko.   Also, thank you antonino61 (if that really is your username...kidding) for your added comments and help.  After I am happy with my notes on documenting the install process, I think I will write up a simple guide on how to do all of the Windows image setup in HyperV.   It can be quite simple.

 

On the other topic of moving major windows components to a different partition, that is an excellent discussion.   For the time being, one of the things I need to maintain for my image is the ability for me to load windows updates.   So I can't move too much out of the Windows directory that might be needed elsewhere.  On the other hand, I don't see why I couldn't just mount a VHD image to (hd1) in the Grub4Dos bootloader so that Windows would load to RAM and everything else would be part of the VHD image but it would be mounted and ready at boot time.  Suffice it to say: This discussion has given me a number of ideas on how I'll ultimately want my boot disk setup for my environment.

 

Anyway back on the issue of loading Windows Updates:  The VHDX file that I have loaded in HyperV is plenty large enough to allow Windows Update to complete.  After it is done, I shutdown the VM, copy the VHDX file over to a new location where I can modify it.  I mount the VHDX file, resize the operating system partition to make it smaller and then use the Edit Disk option in the HyperV Manager to shrink the entire VHDX file down to it's minimum size.   Then I use the same Edit Disk option in HyperV Manager to convert the VHDX to a Static Disk VHD file.   This is all being done on an M.2 based SSD, so it is plenty quick.   The resulting VHD file goes onto my Grub4DOS bootable media and it's done.   I could create a boot entry to boot the VHD file directly.  That would work well for small updates, but for any large updates that require more free space I think my original image is probably best.  Plus that gives me a good working backup in case I screw over the other image.


Edited by quarky42, 11 February 2019 - 10:30 PM.


#53 quarky42

quarky42

    Member

  • Members
  • 38 posts
  •  
    United States

Posted 11 February 2019 - 10:37 PM

Good. :)
 
Time to go back to the (endless) issue on root vs. rootnoverify.  :frusty:

 

(...)

 
:duff:
Wonko

 

 

It may feel like beating your head against the wall.  Please know that your efforts are appreciated, and I have a much better understanding of the difference between the two.  In fact, I may want to try booting to an encrypted partition in the future which may require the rootnoverify at that time, but for now I have plenty on my plate and will make sure that going forward I use the more specific root command instead.    Your efforts to teach are not in vain!  ;?)



#54 quarky42

quarky42

    Member

  • Members
  • 38 posts
  •  
    United States

Posted 11 February 2019 - 11:22 PM

Dear Quarky, 

there musta been a misunderstanding here. I am not in the least upset; I really meant what I said - I do not know how to automate the loading of a second vhd in time for windows booting requirements (viz, if windows needs a certain file on c:\ in the booting routine, how can it be fetched to suit its liking if the d:\ still has to be recognized and it has not yet been?), nor do I know whether it would be quicker that way. one step forward would be to move the files normally in the windows and windows\system32 root subfolders off of the c:\ vhd into d:\windows and d:\windows\system32 respectively in order to shrink the c:\ vhd even further.

As wonko adds, the d:\ path or paths in the environment variable should be added as well.

 

No worries. I wasn't sure if you thought I was saying my way is better.   I am trying to say my way lets me solve some minor challenges in my environment where I would be using this.

 

You can load a second VHD several ways:

1> Using the Grub4Dos Menu, you should be able to map a second VHD file to (hd1) or (hd2) or any other hd number that isn't already in use. You could mount it as a disk in ram or you could mount the vhd file directly without the --mem and --top flags.  In that last case, you wouldn't have to wait for it to load into RAM and changes would be saved to the VHD file.

2> Using ImDisk or other RamDisk software you could mount a volume after windows loads as a ram drive.

3> You can mount VHD files directly in Windows.  There is very likely a script or one could be made that would do this for you.  Then the only challenge becomes calling that script at boot time.  There are mechanisms for that.  I just don't have the documentation in front of me.

 

I suspect there are other ways a VHD file could be mounted, but it depends on what your purpose is and what you're trying to achieve.

 

For my purposes, I would really like to maintain the upgradability of my base windows image, so that I can easily updated it with minimal hassle.  If I can script some space saving operations that move some things to other locations and remaps them properly, that could work, pretty well.   If your purpose is to make the image as small as possible and you don't mind recreating it whenever you need to, that's good too.  We likely value things differently.



#55 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 11 February 2019 - 11:57 PM

yes, perfect, everybody has their own priority, but I still do not understand the advantage of having the loading-of-c:\-as-a-filedisk in g4d's charge, when bootfiles could be created on the host drive and inside the vhd for booting smoothly thru bootmgr. 

bcdboot c:\windows /s c:

and

bcdboot c:\windows /s d:

like this, u would have no persistence problems.

having g4d load the vhd as a filedisk is something I have tried just for the fun of it, but whatever software installation I tried would never be retained.

so I limited myself to using g4d only for ramloading, and of course I could not expect any software installation to be kept across reboots.

As for the windows updates, I do not think it matters to the system whether the files and folders are really on c: or elsewhere, as long as the junction to c: is there to guarantee the logical position the sofware expects to find whatever in.

Regarding a secondary vhd to be loaded which would contain the "non-core" part of windows (all the stuff we have moved off of c:\ in the previous examples) would the routine have enough time to recognize this second vhd as a useful volume (say d:, as in the examples) that the c:\ junctions point to, and safely continue the routine toward the windows interface? if so, is it any quicker than leaving the non-core stuff "unincapsulated" on the host drive?

The idea of having a vhd instead of a plain disk is that it could easily be preloaded into ram, because it is a file to g4d.

now, the substantial breakthrough, as wonko puts it, would be to see if other stuff could be moved off the c:\ vhd to make it even smaller, not only in terms of preramloading times, but also in terms of quick booting and neat working, as it would only keep the files and directories windows expects to find exactly there during the boot process. then again, let us make an example, along with all those subfolders from the windows directory, i might as well try moving the windows\inf directory to d:\ and junction it to c:\. when I reboot, windows might (and surely does) need a file from this directory - there is a junction to c:\ of the d:\windows\inf, but windows does not see it because it knows nothing of junction at that very stage, so to windows the inf folder is at that time inexistent. the other junction folders do the trick just because the system does not need anything in them at that very stage, but later on, when it has already learned of their existence thru acknowledgement of the junctions that make files and folders logically appear on c:\ event though they are on d:\. 

Now if we had an exact or near exact list of the files called by the system in windows and windows\system32 during the bootup routine, we could tailor the c:\vhd more precisely to suit windows' bootup liking and have a much lighter vhd. this is strictly necessary before arriving at an up-and-running session with the windows interface on. once windows gets to that stage, almost everything can be placed anywhere.

Sorry about my logorrhoic accounting.


Edited by antonino61, 12 February 2019 - 12:06 AM.


#56 quarky42

quarky42

    Member

  • Members
  • 38 posts
  •  
    United States

Posted 12 February 2019 - 12:53 AM

For some needs, having an OS return to a fresh state on reboot is a major plus.   Having things attach as a ramdrive after the boot saves boot time.  

 

I think the key is to have stuff that isn't loaded during boot be loaded from a static location whether it be a different partition, a vhd file, a ram disk...what ever.    Also, from something else I read, it sounds like the System32 folder isn't fully "real"... a lot of the stuff that's in there is composed of hardlinks that come out of the WinSXS which is where the bulk of the Windows installation is located.   To maintain system performance, I suspect that the WinSXS should stay loaded in RAMDISK.   Stuff like the Install subdirectory could be moved out because the install files aren't used much.



#57 cdob

cdob

    Gold Member

  • Expert
  • 1469 posts

Posted 12 February 2019 - 05:06 AM

There is a 'Grub4DOS AHCI / NVMe speed patch for memory maps'
http://reboot.pro/to...or-memory-maps/

Try it and compare VHD file load time to RAM.

#58 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 12 February 2019 - 07:32 AM

i have tried it and used it so far. 4gb vhds load in no time. even though it is really quick also with 10-or-over-gb vhds, it is still convenient to keep vhd as tiny as possible, not only in preramloading terms, but also in booting and operational terms.


Edited by antonino61, 12 February 2019 - 07:33 AM.


#59 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 12 February 2019 - 08:24 AM

The point is that only a fraction of the files in the c:\windows\system32 folder are actually needed at boot time, and some (namely most if not all the command line ones) are simply never used unless they are invoked, and then you have a number GUI .exe's, and of dll's that are "linked" to these files, the .chm files, etc.,

If you think about it, a PE (the Pre-installation Environment) boots just fine even if it is a fraction of the size of the corresponding "full-fledged" OS, so, even if a minimal PE might be "too little", the base OS can be greatly slimmed down.

Compare with a minimal XP (as it is nlited or even more aggressively slimmed down), the typical install is 1.5 GB of space required, actual needed size, if I recall correctly 800 or 900 MB, that can be easily shrinked to 200-250 MB with nlite and to much less (50-200 MB) using one of the "mini-XP" projects we have around, although these are made with a rather heavy substitution of built-in programs with alternatives and limiting the functions).

Typically one should log (a complete log) the booting of the system and then move anything that is not in use during that time.
The tools of the trade were Filemon and Regmon (Sysinternals/Mark Russinovich) that have been replaced by a single tool, Procmon, and then (to look and hopefully find dependencies of the logged files, Dependency Walker.

:duff:
Wonko

#60 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 12 February 2019 - 08:43 AM

It may feel like beating your head against the wall.  Please know that your efforts are appreciated, and I have a much better understanding of the difference between the two.  In fact, I may want to try booting to an encrypted partition in the future which may require the rootnoverify at that time, but for now I have plenty on my plate and will make sure that going forward I use the more specific root command instead.    Your efforts to teach are not in vain!  ;?)

Don't worry, in this case the head banging was more a figure of speech than anything else :), the intended target was - besides the generic idea of calling things with their names and making things as simple as possible (but not simpler) - our good friend Wimb, which of course is familiar with grub4dos since some 10 years or so and should have known ....  :whistling:

 

@antonino61

RE: Hard Links

You might find of interest this seemingly UNrelated topic:

http://reboot.pro/to...rdlinked-files/

where it was *somehow* managed to have Windows 7 running from a FAT32 filesystem/volume, where a few info on the hardlinking of recent windows is talked about.

 

@both

Only for the record there is a difference between having the "rest of the files" on a "physical" hard disk and having them on a "virtual" disk or volume, typically the "real" disk is mounted/becomes accessible earlier in the booting phase whilst the "virtual" one may be mounted only after the booting is complete, so at least in theory, "more" files could be moved to a physical disk than those that could be moved to a second image, but of course it depends on the actual virtual disk driver used and anyway I believe that these differences do not overall influence greatly the size of the main image, which IMHO should remain "self-standing" for the booting phase.

 

:duff:

Wonko



#61 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 12 February 2019 - 01:24 PM

and I don't think anyone could be more surgically accurate about the fact of the matter than wonko has already been. this is exactly the path I will tread by following his hints.



#62 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 12 February 2019 - 07:45 PM

About 730 files in windows\system32, if everything goes ok. I'll tell you in a bit.



#63 cdob

cdob

    Gold Member

  • Expert
  • 1469 posts

Posted 12 February 2019 - 10:04 PM

Another hints:

First hard link duplicate files as indicated already.
Most winsxs files are hard linked by default, some are not.

And Windows 10 supports wofadk.sys compactos compressing.
Do this.

And early boot files can be NTFS compressed (compare bootlog.txt and add parent *.sys drivers).
Bootmgr supports NTFS compression, but no compactos compression.
May spare some mb of data after countless of hours. Yes, a insane task, consider this as a nice challenge.

#64 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 12 February 2019 - 10:10 PM

mmm, compression? why? right now I am adding files one by one - I'm halfway thru. is there a way to automate this process? what shall I do in your view? Pls tell me, but spare me the compression. Is that necessary? as wonko says, what u gain in space, u lose in time.

nino



#65 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 13 February 2019 - 01:57 AM

Ladies and Gentlemen, I am dreadful sorry, but ... no dice! Stuck at boot with 0x0000000 error or something.



#66 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 13 February 2019 - 12:50 PM

here we are, left with my attempt yesterday and a lot of perplexity today. When trying to get my head together, I happen to think the other way round. (reverse engineering principle?) What files in c:\windows\system32 can we be sure or certain that windows will not use, so as to start moving just these files to d:\windows\system32, see if it boots like that and work our way up (or out, for that matter) from there. Suggestions are more than welcome.


Edited by antonino61, 13 February 2019 - 12:51 PM.


#67 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 13 February 2019 - 01:44 PM

here we are, left with my attempt yesterday and a lot of perplexity today. When trying to get my head together, I happen to think the other way round. (reverse engineering principle?) What files in c:\windows\system32 can we be sure or certain that windows will not use, so as to start moving just these files to d:\windows\system32, see if it boots like that and work our way up (or out, for that matter) from there. Suggestions are more than welcome.

Sure, you removed "too much".

 

As said the "right" way is to log a complete "normal" booting.

Then:

1) leave alone *anything* that appears in the log

2) leave alone - for the moment - all .dll's
3) remove ONLY files that surely do not belong to the boot process (again surely all command line .exe's but also those GUI ones that clearly does not belong to the boot process, let's say - only as an example - dxdiag.exe - if it exists on Windows 10 :unsure: - is the typical GUI .exe that is used only "on demand)

 

If the thingy boots, good, and then you will need to trace the dependencies of the remaining .exe's (those NOT mentioned in the log) and see what you can move.

 

Ideally you should make a number of "sets", each set containing no more than -say - 20 files, you remove one "set", and if it boots, move to the next "set", if it doesn't boot you re-add the first half of the "set" and try again, if it doesn't boot you remove again the half of the set that you re-added and re-add instead the other half, etc., etc. 

 

And, as hinted before a good reference is a (corresponding) PE.

Build a Windows 10 PE (one of the biggest builds but with no or very few "third party" tools), whatever files are in that PE are very likely to be the "bare minimum" and you shouldn't move them from the Windows 10 "reduced" build.

 

:duff:

Wonko



#68 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 13 February 2019 - 05:41 PM

thank you ever so much, wonko.

One more question now: will any pe do, or will I have to build my own from my own machine?

nino



#69 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 13 February 2019 - 07:53 PM

thank you ever so much, wonko.

One more question now: will any pe do, or will I have to build my own from my own machine?

nino

You will need to build one (actually two, see below) not so much "from your own machine" but rather "from your own source" files AND (most important) because you thus have control on what goes in the PE.

You want to build a PE that has only some basic functionalities (not too many) and one that has all (or nearly all) the functionalities, so that you can compare them and decide what to attempt stripping from the "full install". 

Wimb is already "promoting" the newish (booting in RAM)  XPE:

http://reboot.pro/to...stick/?p=209305

you could start with that.

 

 

 

:duff:

Wonko



#70 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 14 February 2019 - 02:57 PM

thanks again, wonko, i will try to apply your suggestions.

by perusing the content of windows\system32, though, I have found that there is a lot more to it than just exes and dlls. namely, *.acm, *.ocx, *.ax, *.bin, *.cfg, *.conf, *.config, *.cpl (control panel entries they must be), *.dat, *.drv, *.din, *.dtd, *.h, *.iec, *.h, extentionless files and a bit more. what shall I do with those?

nino



#71 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 14 February 2019 - 05:25 PM

what shall I do with those?

 

It is a good question :) , and I have a very good answer for it (besides the obvious "it depends" :w00t: :ph34r:), or - more exactly - a very good answer to another question that could be adapted to the present question.

 

Chesterton's Fence:

https://en.wikipedia...sterton's_fence

 

If I were you, I would leave them alone for the moment.

 

You see, most (but not all) of them may represent (besides either a "standalone" needed or movable file) a dependency of either:

1) something that you actually already moved elsewhere successfully
2) something that is needed and that you cannot move

3) something that is not actually needed in the booting phase, and that can thus be moved BUT that needs to be unregistered/re-registered (not entirely unlike some .dll's) for later operation/functionalities of the OS

 

Compare with (shameless plug):

http://reboot.pro/to...10-mb/?p=154017

 

 

:duff:

Wonko



#72 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 14 February 2019 - 05:40 PM

Ok, I'll do as u say. I'll proceed score by score. but, ... sorry for my ignorance: do I create a win pe if I apply install.wim by imagex to a disk? u mention win pe for good comparison, even 2 of them (min+max), that is the reason for my asking.

nino 



#73 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 14 February 2019 - 08:44 PM

Ok, I'll do as u say. I'll proceed score by score. but, ... sorry for my ignorance: do I create a win pe if I apply install.wim by imagex to a disk? u mention win pe for good comparison, even 2 of them (min+max), that is the reason for my asking.

nino 

No.

If you apply an install.wim (via imagex or otherwise) you are installing.

 

The boot.wim (present on Windows install media) is actually a very minimal PE (or Pre-install Environment).

 

What - by means of Winbuilder (and a specific project) or through other builders you "build" or "assemble" is a "custom" PE that usually has many more capabilities up to (nearly) the full functionality of the installed OS.

 

Since you are "bulding" or "assembling" a PE, you can choose (within limits) what to add (or not add) to the build.

 

If you ask this question, however, you have a loong way before you, follow the given link, try replicating what Wimb suggests:

http://reboot.pro/to...stick/?p=209305

 

It won't probably be easy at first try, although most things are automated you will need to get the hang of it.

 

What happens (hopefully) is that a number of files are automatically extracted from the source .iso (either from the boot.wim or install.wim) and re-assembled, possibly integrated by third party programs and or with modifications in such a way that a new bootable (and "portable") Pre-installation Environment is created with most of the features of the corresponding "installed" Operating System.

 

Maybe (better) ignore the above for the moment, and INSTEAD read about the MistyPE :

http://mistyprojects...ocs/readme.html

(which is simpler, but the general ideas and concepts are the same), download it:

http://mistyprojects....2018.01.21.zip

and try building it at first.

 

:duff:

Wonko 



#74 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 15 February 2019 - 12:24 AM

boot.wim deployed on vhd with gimagex will not boot

it will only yield a bsod



#75 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 15 February 2019 - 08:54 AM

boot.wim deployed on vhd with gimagex will not boot

it will only yield a bsod

Sure it won't.

 

That doesn't mean that i is not a PE, only you used not the "right" (or the "complete") method to deploy it.

 

Also the install.wim simply deployed with gimagex won't boot, there are other needed steps, *like*:

http://reboot.pro/to...external-drive/

 

Check this if you want to apply a "standard" boot.wim to a disk (or vhd):

https://docs.microso...boot-or-non-ram

 

 

:duff:

Wonko







Also tagged with one or more of these keywords: windows, linux, window10, vhd, ramloading

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users