Jump to content











Photo
* * * * * 1 votes

VHD_WIMBOOT - Apply and Capture of WIM Files for OS in VHD

ramdisk grub4dos wimlib svbus windows 10 ssd usb wim vhd wimboot

  • Please log in to reply
1025 replies to this topic

#451 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 23 March 2021 - 11:40 AM

SATA faster than USB is something we should expect.

 

Loading time of 10 XPE WIM file as MBR from NVME is incredibly slow, it seems as bad design of MBR Windows driver for NVME device (could be intentional to promote/force to use UEFI).

 

To load on Ram a Mini-10x64 VHD file the NVME is a piece of junk, it seems grub4dos reads the drive very slow in both cases UEFI and MBR, but especially on MBR.

 

EDIT: Remember Kai Schtrom made: Grub4DOS AHCI/NVMe speed patch for memory maps  but unfortunatelly it is for grub4dos v0.45c (MBR only).

 

Only good score for NVME is on 10 XPE WIM file when UEFI booting, so this UEFI Windows driver for NVME is working fine.

 

alacran


  • wimb likes this

#452 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 23 March 2021 - 01:25 PM

 

As Bootdisk type were used M2 NVME SSD 970 EVO and SATA SSD 870 EVO and USB SSD T5 all of Samsung
 

 

@gbrao

As Bootdisk type were used M2 NVME SSD 970 EVO Size 500 GB and SATA SSD 870 EVO Size 1 TB and USB SSD T5 Size 500 GB all of Samsung

 

@alacran

Indeed NVME SSD is often disappointing for loading Boot Image into RAMDISK

Surprisingly MBR Grub4dos booting from SATA  SSD is the winner for loading VHD into RAMDISK



#453 gbrao

gbrao

    Frequent Member

  • Advanced user
  • 474 posts
  •  
    India

Posted 23 March 2021 - 03:38 PM

Reading a 4 GB VHD from my 1TB WD EZEX hard disk takes 25 secs - using BIOS + MBR. I'm using a old g4d (grub4dos-0.4.5c-2016-01-18 I think).

 

I'm assuming the times you mentioned were to just read the VHD to memory, and not the time to fully boot Windows.

 

The NVMe timings are quite strange.

 

edit : i'm using slow ddr3 1333 memory btw,


  • antonino61 likes this

#454 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 23 March 2021 - 07:07 PM

Pls do not challenge alacrán on this: there's nothing doing from end-of-ramloading onwards!

It must be said, however, that if a quicker bootmgr than the microsoft one could be made, I would not relinquish it, would any of u? Svbus isn't, by one or two seconds. there is a Chinese research thread on the primo driver, which they claim would be 4 times as fast as svbus; unfortunately, primo is not freeware, and the development of the project is still not so clear to us, also owing to the language.


  • alacran likes this

#455 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 23 March 2021 - 11:24 PM

@ wimb

 

If you compress your Mini-10x64 VHD file located into the M2 NVME SSD 970 EVO, It is very possible you can get a good improvement on the time required to load it on RAM, as the size of the file to be readen from the drive will be smaller, it could compensate the grub4dos (UEFI/MBR) slow reading speed of NVME devices.

 

You could use 7-zip to make a Mini-10x64.vhd.gz file, but it is very slow process, I recommend to use lz4_compressor with default option.

 

Please try it and let us know the reduction on time consumed to load on RAM the resulting Mini-10x64.vhd.lz4 file, usually it is more notorious for Wimboot VHDs as they have more unused space and also many pointer files cero size, but I think compressing the VHD file can make a difference also in this case of slow drive reading.

 

You can make the compression on another drive and latter just copy the Mini-10x64.vhd.lz4 file to the NVME drive, this way, as long as there is enought free space on target drive, it will be also unfragmented.

 

alacran



#456 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 24 March 2021 - 12:10 AM

Pls do not challenge alacrán on this: there's nothing doing from end-of-ramloading onwards!

It must be said, however, that if a quicker bootmgr than the microsoft one could be made, I would not relinquish it, would any of u? Svbus isn't, by one or two seconds. there is a Chinese research thread on the primo driver, which they claim would be 4 times as fast as svbus; unfortunately, primo is not freeware, and the development of the project is still not so clear to us, also owing to the language.

 

You have a general idea of the concept, but I would like to clarify this:

 

Grub4dos loads to RAM the VHD, only things we can do to improve the time required for this is reduce the size of the VHD, and also compress the VHD file.

 

EDIT: We have recently found that in some cases just by updating the (Bios/UEFI) firmware of the motherboard we can get an impressive increase in speed, more info about this on following posts.

 

SVBus and Primo are only drivers (not bootmanagers) that allow Windows to read/write from/to the VHD loaded on RAM, booting is then made always by means of usual Windows boot files/folders.

 

In case of SVBus and Primo their main effect is during booting the VHD already loaded on RAM, the respective driver in use is loaded during first stages of boot, It seems here we could get a time improvement using Primo, it is said Primo is faster than SVBus when reading/writing to the VHD loaded on RAM, but haven't tested it myself as it is not freeware.

 

After booting from RAM: But I don't think Primo speed improvement can be even notorious on standard tasks as Windows has a pre-established timing for almost all their processes, including open the menu from Start button, so I really do not care very much about Primo driver speed improvement, and I'm not convinced to buy it with the only idea to get a few seconds improvement during boot stage, which by the way using SVBus driver is good enough for my taste.

 

alacran


Edited by alacran, 01 April 2021 - 02:04 AM.
New info.

  • antonino61 likes this

#457 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 24 March 2021 - 06:44 AM

@gbrao

I just measured the time in seconds to load VHD to RAM

 

@alacran

I have used your  lz4_compressor with default option to make Mini-10x64.vhd.lz4 Size 1831 MB instead of originally 3900 MB

 

Now in case of NVME SSD and MBR Grub4dos the time to load to RAM is reduced to 42 seconds instead of originally 83 seconds

 

CrystalDiskMark was used to measure speed of NVME SSD and SATA SSD and USB SSD

It is clear that NVME SSD is superior in that case.

 

    NVME SSD    ==   SATA SSD        ==     USB SSD

M2_NVME_SSD_2021-03-24_064629.jpg == SATA_SSD_2021-03-24_064956.jpg == USB_SSD_2021-03-24_065410.jpg


#458 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 24 March 2021 - 12:58 PM

@ alacrán (and co.)

What I had in mind when I mentioned svbus was booting as filedisk, not as ramdisk: what is the 1-2secs lag due to?



#459 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 24 March 2021 - 07:28 PM

 

@gbrao

I just measured the time in seconds to load VHD to RAM

 

@alacran

I have used your  lz4_compressor with default option to make Mini-10x64.vhd.lz4 Size 1831 MB instead of originally 3900 MB

 

Now in case of NVME SSD and MBR Grub4dos the time to load to RAM is reduced to 42 seconds instead of originally 83 seconds

 

CrystalDiskMark was used to measure speed of NVME SSD and SATA SSD and USB SSD

It is clear that NVME SSD is superior in that case.

 

 

Thanks for the info.

 

With a lz4 compressed VHD file to 46.95 % of its original size the time required to load it on RAM is almost 50 %, which is a very good improvement, but it is still almost 3X the time it takes to load the full size VHD as MBR from a SATA SSD (15 sec.).

 

From: http://reboot.pro/in...=19#entry217960

 

To load on Ram a Mini-10x64 VHD file the NVME is a piece of junk, it seems grub4dos reads the drive very slow in both cases UEFI and MBR, but especially on MBR.

 

There is no doubt the trouble resides on the very slow tool or drivers used internally by grub4dos to read a NVME drive.

 

In fact we can be sure the NVME drive is working fine as expected, as it is demonstrated by your CrystalDiskMark tests, but this was obvious just seen the time for the UEFI WinPE on same drive was only 2 seconds.

 

IMHO only thing you can do is make an issue report on: https://github.com/c...grub4dos/issues

 

It could be good if you test MBR booting the uncompressed VHD file (grub4dos v0.45c do not support lz4 compression) using Kai Schtrom's Grub4DOS AHCI/NVMe speed patch for memory maps  for grub4dos v0.45c (MBR only), and include also that info on the issue report, Kai Schtrom's AHCI/NVME speed patch could give them some ideas to improve the NVME reading speed of grub4dos v0.46a

 

alacran



#460 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 25 March 2021 - 07:04 AM

@alacran

 

Thanks for your help and advice.

 

I have made grub4dos issue report: NVME SSD very slow for grub4dos loading VHD to RAM


  • alacran likes this

#461 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 30 March 2021 - 01:36 AM

@ wimb

 

Well, in case of grub4dos low speed reading of a NVME drive, it seems only thing the user can do is go to the MB manufacturer page and look if there is a new Firmware (Bios/UEFI) made especifically to improve its internal drivers to read NVME devices.

 

@ All, JFYI

 

The low speed reading the NVME drive is not caused by grub4dos internal code, it is caused by the drivers contained in the Firmware (Bios/UEFI) of respective MB, as our friend A1ive explained here:

 

From: https://github.com/c...mment-809116053

 

grub (and other bootloaders) use the API provided by the firmware (BIOS/UEFI) to read the disk, and the speed is determined by the firmware's drivers.

 

Also our fellow steve6375 gave a more detailed explanation here:

 

From: https://github.com/c...mment-809164631

Spoiler

 

I was wrong thinking grub4dos uses some kind of internal drivers, when in fact it uses the drivers contained in the Firmware Bios/UEFI of respective MB, but it was good to know this info.

 

alacran


  • wimb likes this

#462 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 30 March 2021 - 08:49 AM

 

Time to Load 10XPE WIM file Or Mini-10x64 VHD file into RAMDISK varies a lot with way of booting and type of bootdisk.
 
Loading RAMDISK was measured in seconds for 10XPE WIM of 867 MB and Mini-10x64 VHD of 3900 MB
As Bootdisk type were used M2 NVME SSD 970 EVO Size 500 GB and SATA SSD 870 EVO Size 1 TB and USB SSD T5 Size 500 GB all of Samsung
 
The way of booting from RAMDISK is for 10XPE WIM file to use latest Windows 10x64 Boot Manager in UEFI Or MBR BIOS mode
The way of booting from RAMDISK is for Mini-10x64 VHD file to use UEFI Grub4dos Or MBR Grub4dos in MBR BIOS mode
 
bootdisk type  PE WIM  Mini VHD
 
NVME SSD UEFI      2     65
NVME SSD MBR    23     83
 
SATA SSD UEFI       2     38
SATA SSD MBR       4     15
 
USB  SSD UEFI       8     55
USB  SSD MBR       8     32
 
The factor 4 in MBR LoadTime between WIM and VHD into RAMDISK is explained by their Size - WIM = 867 MB and VHD = 3900 MB
 
Remarkable is for NVME SSD that loading WIM or VHD into RAMDISK is rather slow in case of MBR BIOS mode 
Remarkable is that MBR Grub4dos loading VHD into RAMDISK is very fast for SATA SSD and very slow for NVME SSD
 
SATA  SSD is always superior as compared to USB SSD or NVME SSD for loading WIM or VHD into RAMDISK
NVME SSD is for most cases inferior for loading WIM or VHD into RAMDISK
 
More Info in VHD_WIMBOOT.pdf == NVME_SSD == SATA_SSD == USB_SSD
 

 

 

Good News !

 

After BIOS/UEFI firmware Update the NVME SSD speed to Load PE WIM and Mini-10x64 VHD to RAM is very much improved.

Now NVME SSD UEFI is clearly the winner in loading 3900 MB VHD in only 3 seconds to RAM instead of 65 seconds before firmware Update.

Also NVME SSD MBR Grub4dos is much better in loading 3900 MB VHD in 20 seconds to RAM instead of 83 seconds before firmware Update.

 

 PE WIM  Mini VHD Before and After BIOS/UEFI firmware Update
bootdisk                 PE   VHD      PE   VHD
NVME SSD UEFI      2     65             1     3
NVME SSD MBR    23     83            4    20
 
SATA SSD UEFI       2     38             2    38
SATA SSD MBR       4     15             3    13
 
USB  SSD UEFI       8     55              8    55
USB  SSD MBR       8     32              8    32
 
Disks_2021-03-22_185258.jpg == NVME_MBR_UEFI_2021-04-06_071342.jpg

 

 


  • gbrao and alacran like this

#463 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 30 March 2021 - 09:26 AM

@ wimb

 

Well, in case of grub4dos low speed reading of a NVME drive, it seems only thing the user can do is go to the MB manufacturer page and look if there is a new Firmware (Bios/UEFI) made especifically to improve its internal drivers to read NVME devices.

 

@ All, JFYI

 

The low speed reading the NVME drive is not caused by grub4dos internal code, it is caused by the drivers contained in the Firmware (Bios/UEFI) of respective MB, as our friend A1ive explained here:

 

From: https://github.com/c...mment-809116053

 

Also our fellow steve6375 gave a more detailed explanation here:

 

From: https://github.com/c...mment-809164631

Spoiler

 

I was wrong thinking grub4dos uses some kind of internal drivers, when in fact it uses the drivers contained in the Firmware Bios/UEFI of respective MB, but it was good to know this info.

 

 

Thanks alacran for your advice to Update BIOS/UEFI Firmware  :)

 

The previous post shows a very impressive improvement for NVME SSD in the speed to Load PE WIM and Mini-10x64 VHD to RAM


  • alacran likes this

#464 gbrao

gbrao

    Frequent Member

  • Advanced user
  • 474 posts
  •  
    India

Posted 30 March 2021 - 02:25 PM

 

SATA SSD MBR       4     15                  3    13

 

^ quite good too.

 

if you don't mind post your processor, mobo and memory.



#465 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 31 March 2021 - 06:40 AM

if you don't mind post your processor, mobo and memory.

 

Intel i7-6700K CPU @ 4.00 GHz 4 Cores 8 Threads on ASUS Z170I PRO GAMING mobo with 32 GB RAM Kingston DDR4-2400 (1200 MHz)

 

SSD 970 EVO Plus NVMe M.2 500GB

 

P1070610.JPG == NVME_MBR_UEFI_2021-04-06_071342.jpg == Disks_2021-03-22_185258.jpg  


  • gbrao likes this

#466 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 31 March 2021 - 03:38 PM

Well, folks, I really cannot fail to enthuse at the religious, Easter-like, value of Wimb the Flying Dutchman's creation of the past 2 decades - vhd_wimboot, thanks to which we can have a 2-way generation and successive regerenations of *.wim  and *.vhd files. So many alternatives (all working, btw), so many children that father other children or get fathered by later versions of brothers which will become children that will possibly father yet other children. All this is to celebrate the mutual versatility of the software. No matter how confusing and counterintuitive my explanation might read, I don't remember any windows installs being more stable than these ones that were created with vhd_wimboot. Sorry again for the intricate presentation and, attaboy wimb ... and a few more as well!! 



#467 wimb

wimb

    Platinum Member

  • Developer
  • 3756 posts
  • Interests:Boot and Install from USB
  •  
    Netherlands

Posted 31 March 2021 - 06:02 PM

Thanks for your kind words.

 

I can say it was an interesting development which is based on interaction of all of us at this forum reboot.pro

 

:cheers:


  • alacran and antonino61 like this

#468 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 31 March 2021 - 06:31 PM

that is why I added ...and a few more as well!! but, if I may, u r the pragmatic clinical finisher of the team, in that you have concocted the instances of all into this wonderful concretion, so much so that vhd_wimboot can be considered the European Onderdonk House!!! It is a pity the software gets much less recognition than it deserves, for being, among other things, "paradoxically" more stable than the original, without rhetoricism. I keep talking about it at work, where they tell us to get our files off of the pc's before they re-format and re-install the system all over, which takes place every half-year to every year. I find it ludicrous.



#469 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 31 March 2021 - 06:36 PM

btw, I am not a fan of UEFI, but I could not help seeing the comparison charts above in this thread. Since my mobo supports both BIOS and UEFI,  would u suggest I try the latter and, if you do, would that be one hell of a change (is the MBR-->GPT transition necessary)?



#470 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 01 April 2021 - 04:52 AM

wimb, on 31 Mar 2021 - 12:02 PM, said:

Thanks for your kind words.

I can say it was an interesting development which is based on interaction of all of us at this forum reboot.pro

:cheers:


It has been a very long journey to arrive to the present development of RAMOS from VHD.

Certainly your programing skills have made possible to create your programs to let non advanced users make use of this achievements.

Also you have developed and improved several new approaches for this subject.

I'm also totally agree many people (but not only from reboot.pro) has contributed during last decades to make this possible, just to mention some of the many people and steps that had been followed to arrive to present achivements:

 

Following is some kind of historic list, of all previous developments that have been necessary to arrive to present state of RAMOS.

 

grub4dos developers that made this possible in first place (who developed, mantain and improve it): tinybit, karyonix, chenal, yaya, a1ive, and many others who contibuted to this fantastic boot loader program.

steve6375 author of RMPrepUSB, EasytoBoot and his many tutorials to use grub4dos.

 

Shao Miller author of WinVBlock driver (to avoid old USB booting limitations).

karyonix author of Firadisk driver, Diskmod driver (to avoid old USB booting limitations), and his contributions to solve Wimboot VHD entries on (MBR) menu.lst

 

JFX author of WinNTSetup.

 

wimb for his programs to create VHD_XP_Setup - Install XP in VHD, VHD_W7_Compact - Make Mini 7, and VHD_W8_Compact - Make Mini 8 very good to create small size VHDs using NT Compression, mainly used for filedisk booting, as at the time it was very rare to have enought RAM for RAMBOOT, I remember the Mini-XP was fantastic, and it made me start having interest on this subject.

erwnan.l and misty for the first experiments on first topic on reboot.pro about OS Wimboot.

Cdob for his contributions to USB booting, and to let us understand and manage the target WIM file on Wimboot, and for his batch file to reduce winsxs folder.

Eric Biggers author of wimlib-imagex.

Kai Schtrom author of SVBus driver, (very useful for all versions of RAMOS), that has replaced Firadisk and WinVBlock drivers.

a1ive for his great grub2 version and for create the first topic on reboot.pro about grub4dos for UEFI.

All people on wuyou.net forum who made possible to let make use of SVBus driver on grub4dos for UEFI.

liuzhaoyzz for his very informative topics and posts on wuyou.net forum and here on reboot.pro

Wonko the Sane for his interventions in almost all topics related to this subject.

erwan.l for the technical maintenance of reboot.pro forum and his help to improve the code of some wimb's programs.

Nuno Brito for start, keep alive and maintain this wonderful forum reboot.pro

And finally gbrao, antonino61 and alacran (myself) for contributions with some topics, ideas and tests.

I'm sure the list is not complete, please excuse me for any involuntary omission.

 

NOTE: as I was thinking on VHD_WIMBOOT program I forgot to mention, that just a few time after the development of WinVBlock and Firadisk drivers, wimb made his program to create Mini-XP, and some time latter Mini-7 and Mini-8 VHDs, I apologize for this involuntary mistake, now they were added to the post in approximate chronological order.

alacran


  • wimb and antonino61 like this

#471 gbrao

gbrao

    Frequent Member

  • Advanced user
  • 474 posts
  •  
    India

Posted 01 April 2021 - 05:28 AM

...

And finally gbrao, antonino61 and alacran (myself) for contributions with some topics, ideas and tests.

I'm sure the list is not complete, please excuse me for any omission.

alacran

I don't deserve to be on that list.

And, that alacran should be somewhere higher up the list :-)



#472 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 01 April 2021 - 06:08 AM

I don't deserve to be on that list.

And, that alacran should be somewhere higher up the list :-)

 

You made several good contibutions to help reduce the OS foot print on this topic, (wich is part of the present subject): http://reboot.pro/in...showtopic=22383

 

I haven't developed any driver or program to make me deserve a better place, anyway I tried to make some kind of historic list, of all previous developments that have been necessary to arrive to present state of RAMOS.

 

Your friend

 

alacran



#473 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 01 April 2021 - 11:09 AM

Wonderful historical tracing by Alacrán! Higher aims ahead! e.g., now I am looking to further reduce vhd footprint by removing any possible duplicates. I will tell you more in a bit.



#474 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 01 April 2021 - 11:33 AM

well, folks, one more point for the team - no duplicates on i:\; I will see if the same results can be obtained on the same drive as c:\. in a bit.



#475 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 01 April 2021 - 12:49 PM

Wonderful historical tracing by Alacrán! Higher aims ahead! e.g., now I am looking to further reduce vhd footprint by removing any possible duplicates. I will tell you more in a bit.

 

The history was nor complete, as I was thinking on VHD_WIMBOOT program I forgot to mention just a few time after the development of WinVBlock and Firadisk drivers, wimb made his program to create Mini-XP, and some time latter Mini-7 and Mini-8 VHDs, all of them having the option to use NT compression.

 

I still keep them on my backups, my previous post was updated to include them.

 

About duplicate files, don't waste your time, Windows uses a lot the Symbolic Links and Directory Junctions by design, so even if files or folders seem to appear more than once they are written to the disk or virtual disk only once, all other instances are zero size files.  For more info you can see this topic: http://reboot.pro/in...showtopic=22331

 

Also I know you prefer to use Wimboot installations, where majority of files on the VHD are mainly pointer files (zero size) to the linked WIM file, well let me tell you: also on WIM files, the files are written only once, even if they seem to appear several times on the partition/drive/virtual drive used as source to capture the WIM file, so you will not reduce the used size of the VHD or the size of the WIM file.

 

By the way, JFYI  7-zip also uses same approach.

 

Only files that could appear several times are those added by the user, usually on his Documents and Downloads folders, but if captured in a WIM file, same apply to them.

 

alacran







Also tagged with one or more of these keywords: ramdisk, grub4dos, wimlib, svbus, windows 10, ssd, usb, wim, vhd, wimboot

5 user(s) are reading this topic

0 members, 5 guests, 0 anonymous users