Jump to content











Photo
- - - - -

Reducing wimboot source wim file using LZX Compression, and VHD using gzip or LZ4 Compression, to save room, and also load faster on RAM

wimboot ramboot lzx gzip lz4 vhd

Best Answer alacran , 26 March 2019 - 10:05 PM

We could say that this thread is a continuation of: Making the smallest Win10 install (Wimboot mode) on 512 MB VHD

 

It was necessary to run some additional test using the info from cdob post No. 95: http://reboot.pro/to...e-4#entry210231

 

 

lz4.exe -12  --content-size --favor-decSpeed e:\win.vhd e:\win.vhd.lz4

 

LZ4 compression is better and faster than using GZ, or any other option, it gave me the faster loading to Ram times, also noticed that when loading from an slow device (as my MicroSD) it has the best efect.

 

All test ran on same I3 3225, 3.3 GHZ, 8 GB Ram at 1333 MHZ, Rambooting from SSD Adata SU650 into Adata XPG enclosure 3.0 USB.

 

This are my findings:

 

Speed:

VHD								Time to load on RAM

1 - 10x64-WB.vhd Ramboot using g4d  grub4dos-0.4.6a-2018-12-23		12 seconds

2 - 10x64-WB.vhd.gz Ramboot using g4d  grub4dos-0.4.6a-2018-12-23	7 seconds

3 - 10x64-WB-EXP.vhd Ramboot using g4d  0.4.5c Modd by kyrionix		4 seconds

4 - 10x64-WB.vhd.lz4 Ramboot using g4d  grub4dos-0.4.6a-2019-03-25	3.67 seconds

Saved espace:

VHD				Used	Saved espace

10x64-WB.vhd			1536 MB	0 % saved espace

10x64-WB.vhd Expandable		480 MB	68.75 % saved espace

10x64-WB.vhd.gz			233 MB	84.83 % saved espace

10x64-WB.vhd.lz4		135 MB	91.21 % saved espace

It is very possible I am getting close of maximum loading speed possible for this PC and this is influencing results.

 

On a single test I ran from my MicroSD, the 10x64-WB.vhd.lz4 loaded to Ram on 14.68 seconds wich is really a very good time for that device, and it was plugged to USB 2.0

 

Conclusions:

 

If Rambooting from a fast device as an SSD use expandable VHDs, they have very good speed and a resonable saved espace, do no require any additional compression, they are capable to Ramboot and also can be booted from Windows bootmanager, so it is not a big issue if they can't boot as Filedisk using grub4dos.

 

If Rambooting from a not so fast device as an SSD, we have the alternative to use LZ4 compressed VHDs, wich are the faster option and also save more espace than any other option, but they are only usefull for Rambooting, we can not boot them as Filedik using grub4dos or from Windows Bootmanager, this limitations also apply to GZ compression.

 

Thanks to all members of the forum who collaborated with this thread with info and suggestions, for the realization of all this tests:

 

antonino61, Wonko the sane, karyonix, tinybit, cdob and yaya for fast fixing grub4dos-0.4.6a-2019-03-25

 

 

Just to make a summary of all we have learned/tested/proved/improve on this thread:

From previous Post: http://reboot.pro/to...e-4#entry210209

 

LZX compressed wim files for Wimboot VHDs do not fail anymore during boot or Ramboot. See: http://reboot.pro/to...am/#entry210021

 

With this hight compression I have found If we don't add those items on [PrepopulateList], especially the first two we will have troubles during Ramboot

 

Saving 20 % espace using LZX compressed wim files. See: http://reboot.pro/to...am/#entry210021

 

Only things we have to do in a different way to reduce about 20 % the size of our source win file is: ..........

 

Tested GZ compresed VHDs and found they load faster on Ram (almost about 1/2 time).See: http://reboot.pro/to...am/#entry210031

 

I compressed 10x64-WB.vhd with 7-zip GZ Ultra as 10x64-WB.vhd.gz , it took 15 min, but file is now 233 MB

 

Saving about 85 % espace if using GZ compresed VHDs, See. http://reboot.pro/to...am/#entry210031

 

....but file is now 233 MB so it reduced a lot the size 84.83 %,......

 

Improved speed using GZ compression on VHDs See: http://reboot.pro/to...am/#entry210031

 

So there is a big improvement in time to load in Ram (43.75 % less time)

 

Testing expandable VHDs Ramboot capability, and found it was failing.See: http://reboot.pro/to...e-2#entry210068

 

It fail to load on Ram, still verifiying to try to find if I made a mistake before say it do not work,

 

Detect the last grub4dos 0.46a version capable to Ramboot expandable VHDs. See: http://reboot.pro/to...e-2#entry210089

 

grub4dos-0.4.6a-2017-12-20 is last version capable of Ramboot dynamic VHDs, the next version grub4dos-0.4.6a-2017-12-23 Fails.

 

Thanks to karionix who detected where was exactly the two lines disabled on grub4dos 0.46a after 2017-12-20, See: http://reboot.pro/to...e-2#entry210101

 

The codes to read hard disk footer in line 185 and read dynamic disk header in line 209 were just "disabled".

 

Improved again time to load on Ram , using Expandable VHDs. See: http://reboot.pro/to...e-2#entry210074

	VHD							Time to load on ram

1 - 10x64-WB.vhd Ramboot using g4d  v0.46a Dic 2018            		12 seconds

2 - 10x64-WB.vhd.gz Ramboot using g4d  v0.46a Dic 2018            	7 seconds

3 - 10x64-WB-EXP.vhd Ramboot using g4d  0.4.5c Modd by kyrionix		4 seconds

Saved espace using expandable VHDs compared to 1.5 GB fixed size VHDs. See http://reboot.pro/to...e-3#entry210115

VHD			Used	Saved espace

10x64-WB.vhd		480 MB	68.75 % saved espace

10x86-WB.vhd		360 MB	76.56 % saved espace

8x64-WB.vhd		416 MB	72.91 % saved espace

W864ESP1-WB.vhd		364 MB	76.30 % saved espace

Got the attention of tinybit and yaya (grub4dos development team) and thanks to it, get in about 2 days a new fixed release. See: http://reboot.pro/to...e-3#entry210171

 

yaya fixed it.

 

And this new posts related to LZ4 Compression:

 

karyonix info about LZ4, See: http://reboot.pro/to...e-4#entry210229

cdob info and commands for LZ4, See: http://reboot.pro/to...e-4#entry210231
 

Analyzing why the difference of speeds: http://reboot.pro/to...e-5#entry210265

 

Comparing the 2 options suggested by cdob for LZ4 Compression: http://reboot.pro/to...e-5#entry210278

 

When having x86 and x64 versions you can make a wimboot source wim file with multiple indexes to save espace, see: http://reboot.pro/to...am/#entry210406

 

And for all lazy people wimb just made a program to make almost all this, things not available on his program are compressions of VHDs and multi-index wimboot source wim files:

 

VHD_WIMBOOT - Apply and Capture of WIM Files for OS in VHD by wimb: http://reboot.pro/to...-for-os-in-vhd/

 

And finally now, there is also available this lz4_compressor GUI: http://reboot.pro/to...lz4-compressor/

 

I only hope this thread has helped some of us to learn something.

 

alacran

Go to the full post


  • Please log in to reply
165 replies to this topic

#76 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 25 March 2019 - 10:33 AM

If u wanna compare my preramloading with his, it takes even less than 4 secs here. Most of the loading time is taxed afterwards here.

#77 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 25 March 2019 - 10:35 AM

Sorry Wonko when I typed loading time I meant booting time

#78 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 25 March 2019 - 12:28 PM

As said I don' think any of you two are actually lying :), but surely you are completely failing to communicate between you two and to a third party reader/observer (yours truly).

 

Maybe till now you didn't notice the above (which is, has been and will remain an S.E.P. for me) but now you know.

 

Up to you to find *somehow*  a common language to communicate and hopefully understandable by third party readers.

 

:duff:

Wonko



#79 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 25 March 2019 - 12:51 PM

r u so sure that the communication failure rests entirely on the sender's side and not on the recipient's? e.g., I suggested u make some tests, and alacran hinted at it as well, i am sure of that because of his wording, did u check the logical discrepancy (in terms of communication only, not in terms of technology, of course) between the counter at the preloading stage stopping at end of used space and resmon.exe showing that the vhd occupies the whole size, free space included? just take good care to replicate the same on ur end, u will understand what I say better, I am sure. on the other hand, if u want a quicker approach, u might as well keep thinking that what I write here is perforce of little or no value inasmuch as I confuse ppl. it is entirely up to u. me, if I am told something, i first check, just because it might even by mistake be true. perhaps because I have one hell of a lotta time, and if I haven't, i make it. 

nino



#80 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 25 March 2019 - 01:07 PM

I  am saying this not to talk of discrepancy because I love the word, but because this is where alacran might fail to understand. he claims that no matter how big the dynamic vhd file, the preloading stops at the end of the used space in it. so if u have 2 same-used-spce dynamic vhds, 1 of say 2gb and another of 20gb, they take the same time to load in ram. only to load in ram at the outset, because the counter stops counting at the same time, when used space is reached. but this does not mean that they don't occupy different ram space in the end. when all is loaded at the windows interface (end of booting, let's say), if u call resmon.exe and check the memory occupied, u will see at the extreme left a grey space indicating the amount of ram occupied by the vhd, which is the full size of the vhd, which is also what alacran seems to overlook from his wording. he says to me, since the preramloading time is the same,  why do u (which is me), as u can afford it, not have a confortably large dynamic vhd so as to be on the safe side spacewise for a long time?. who cares if it occupies so much ram since u have plenty? this is his suggestion to me. my reply, after having tested what he says, is hell no. that space to no avail is no sense, the computer does not show to be any quicker, more efficient or more responsive than with a small size dynamic vhd. what is so difficult to understand on account of communication, honestly, I have a degree + a phd, but still fail to understand what it is,

nino



#81 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 25 March 2019 - 04:42 PM

to whom it may concern, i have just rebooted with 131mb free space on c:\ (the dynamic vhd). At reboot completion, I got 207mb free space on c:\ (still the vhd), which has shrunk to 196 at the time of writing (now), and will shrink even further to the 131mb already recorded. why? windows has always done so, I am not the first who has seen this, for simply writing to disk, for ur surfing, for just existing, anything might be the reason. The fact remains, and even if I claimed that the space that is getting occupied is the ram, because i have booted it as ramdisk it is only a burocratic detail --> windows does the same even if the vhd is booted as filedisk, namely with conventional bootmgr, given the dynamic character of the vhd. The same would have been if it had been a fixed vhd, also if booted thru g4d as filedisk. now the "bulge" as I call it, is under control, so no prob, at least so far. but I could by no means say that the used space stays intact. even less so if I happen to install new software, in which case I claimed by supposition on an earlier post (almost, not completely, unheeded, btw) the information would not enter the wim, but reside in the vhd. that was the only reason why I wondered if it would be more proper to re-bake wim and vhd, since it takes 2 minutes to do one thing and 30 seconds to do the other here. If u wanna think otherwise, of course, whatever suits u best.

nino



#82 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 25 March 2019 - 07:20 PM

yaya fixed it.

download the latest build http://grub4dos.chenall.net

the 2 lines were commented out by mistake.

added a line of bytesread = bytesread to suppress gcc unused variable warning.

 

Just to let you know after testing it Rambooting expandable VHDs, I can tell you with no doubt Rambooting expandable VHDs is working very fine now.

 

alacran



#83 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 25 March 2019 - 08:43 PM

This is the last of several post about comparing results:
 

As said I don' think any of you two are actually lying :), but surely you are completely failing to communicate between you two and to a third party reader/observer (yours truly).

 

Maybe till now you didn't notice the above (which is, has been and will remain an S.E.P. for me) but now you know.

 

Up to you to find *somehow*  a common language to communicate and hopefully understandable by third party readers.

 

:duff:

Wonko

 

 I'm sure my reports about my tests are clear enough for anybody who can read, and all of them talk about the time to load on RAM the info contained into the VHD, trying to found the best/faster option to suggest to users. 

 

Why only the time to load on Ram? Well, because that is a time we can influence or modify by means of certain actions.

 

After that point once the VHD is loaded on Ram there is nothing we can do to improve booting speed. So why bother us with something we can't change?

 

Please DO NOT compare my reports with other people reports since:

 

1 - Using a different PC: CPU, Ram and Mass Storage Device speeds are very different and affect a lot the results.

 

2 - Other people may not use same criteria on his/her tests: As when I report only loading to Ram time, he/she may report full booting time.

 

3 - I am in no way responsible or capable to change how other people like to do their things.

 

4 - Unless other people specifically says he's/she's tests and reports are made following exactly same criterias as mine in order to establish a comparative, If you try to compare them, you are comparing oranges and apples, and only a fool does this.

 

I perfectly know you are not a fool, on the contrary I consider you a very knowgeable man, but this time I can't understand why you blame me on something out of my control, especially when even yourself after trying several times can't make other people do what you want, maybe you are only frustated and that was the only way you found to let it go out.

 

You usually apport good ideas to all, but this time your comments are out of focus.

 

Just to make a summary of all we have learned/tested/proved/improve on this thread:

 

LZX compressed wim files for Wimboot VHDs do not fail anymore during boot or Ramboot. See: http://reboot.pro/to...am/#entry210021

 

With this hight compression I have found If we don't add those items on [PrepopulateList], especially the first two we will have troubles during Ramboot

 

Saving 20 % espace using LZX compressed wim files. See: http://reboot.pro/to...am/#entry210021

 

Only things we have to do in a different way to reduce about 20 % the size of our source win file is: ..........

 

Tested GZ compresed VHDs and found they load faster on Ram (almost about 1/2 time).See: http://reboot.pro/to...am/#entry210031

 

I compressed 10x64-WB.vhd with 7-zip GZ Ultra as 10x64-WB.vhd.gz , it took 15 min, but file is now 233 MB

 

Saving about 85 % espace if using GZ compresed VHDs, See. http://reboot.pro/to...am/#entry210031

 

....but file is now 233 MB so it reduced a lot the size 84.83 %,......

 

Improved speed using GZ compression on VHDs See: http://reboot.pro/to...am/#entry210031

 

So there is a big improvement in time to load in Ram (43.75 % less time)

 

 

Testing expandable VHDs Ramboot capability, and found it was failing.See: http://reboot.pro/to...e-2#entry210068

 

It fail to load on Ram, still verifiying to try to find if I made a mistake before say it do not work,

 

Detect the last grub4dos 0.46a version capable to Ramboot expandable VHDs. See: http://reboot.pro/to...e-2#entry210089

 

grub4dos-0.4.6a-2017-12-20 is last version capable of Ramboot dynamic VHDs, the next version grub4dos-0.4.6a-2017-12-23 Fails.

 

Thanks to karionix who detected where was exactly the two lines disabled on grub4dos 0.46a after 2017-12-20, See: http://reboot.pro/to...e-2#entry210101

 

The codes to read hard disk footer in line 185 and read dynamic disk header in line 209 were just "disabled".

 

Improved again time to load on Ram , using Expandable VHDs. See: http://reboot.pro/to...e-2#entry210074

1 - 10x64-WB.vhd Ramboot using g4d  v0.46a Dic 2018            12 seconds

2 - 10x64-WB.vhd.gz Ramboot using g4d  v0.46a Dic 2018            7 seconds

3 - 10x64-WB-EXP.vhd Ramboot using g4d  0.4.5c Modd by kyrionix        4 seconds

Saved espace using expandable VHDs compared to 1.5 GB fixed size VHDs. See http://reboot.pro/to...e-3#entry210115

VHD			Used	Saved espace

10x64-WB.vhd		480 MB	68.75 % saved espace

10x86-WB.vhd		360 MB	76.56 % saved espace

8x64-WB.vhd		416 MB	72.91 % saved espace

W864ESP1-WB.vhd		364 MB	76.30 % saved espace

Got the attention of tinybit and yaya (grub4dos development team) and thanks to it, get in about 2 days a new fixed release. See: http://reboot.pro/to...e-3#entry210171

 

yaya fixed it.

 

Best Regards

 

alacran



#84 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 25 March 2019 - 08:56 PM

Yep :), I wonder who exactly suggested the ideas of .trying to gz compress the .vhd and of trying dynamic .vhd's.... :whistling:

 

And also the (evidently completely failed) idea of having a "common" testing procedure, so that results could be actually comparable ... :frusty:

 

Anyway, as usual, go trying helping people  ...  :wacko:

 

 

:duff:

Wonko



#85 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 25 March 2019 - 09:06 PM

Once again you read, but you understand what you want. I said WE:

 

 

Just to make a summary of all we have learned/tested/proved/improve on this thread:

 

As far as I know that means all of us, not me.

 

Sorry to hurt your feelings, this was an exercise made by all who contributed to this thread.

 

alacran


  • antonino61 likes this

#86 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 25 March 2019 - 09:29 PM

do not worry alacrán, wonko does not understand me, not u.

there is still one thing pending, lz4. wonderful compression, but no dice on loading. who introduced the idea of compressing? who introduced the idea of dynamic vhds? I honestly do not know. I presume someone or other did. to me, they were just out there, even though I remember alacran mentioning them at different times, but if it had been pamelo, i would certainly not have refused it.

for lz4, I remember karyonix mentioning it, but I do not know if it was with a view to loading them for booting. I consider it interesting, if we could. A couple hours ago I contacted the lz4 man, asking how we could have g4d load (which it does, but unfortunately stops) the lz4-compressed vhd and allow the booting, and he said to contact the g4d crew, as it was not on lz4's part. 

back to the previous testing discrepancy, any comparison is possible, I believe, only provided we knew what we were comparing. Now i do not know what hardware or software architecture alacran has, but I am sure he has good reason to concentrate on the preloading stage, the remnant stage presumably being sort of easy and smooth for him. me, instead, I have no issues in preramloading, as it is almost instant; my issues are on the next stage instead, as my architecture must for some reason slow things down. When I insist that here the only thing that is slow is booting, i am not only practicing rhetorics.

nino



#87 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 25 March 2019 - 09:37 PM

LZ4 is part of the algoritm used to decompress (read/open/load/decode) expandable VHDs, it was added by karyonix to grub4dos long time ago, and was working fine, but latter on Dec 2017 some one commit a mistake and dissabled the 2 lines required for it to work,  and it is fixed and working on the very last grub4dos 046a just released: http://reboot.pro/to...e-3#entry210171

 

EDIT: I missunderstud the info I read, karyonix has already clafified the info in post No. 93

 

 

LZ4 is a compression algorithm. Its functionality is comparable to gzip. LZ4 decompression is faster than gzip.
GURB4DOS can decompress .lz4 files like .gz files, with the condition that lz4 files were created with --content-size option.

LZ4 is unrelated to dynamic VHD. What is broken in dec 2017 is dynamic VHD. LZ4 is not affected.

 

Excuseme for wrong info, but at least it is clear now.

 

alacran


Edited by alacran, 26 March 2019 - 09:25 PM.


#88 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 25 March 2019 - 09:39 PM

oh yes? wow. let us try then, I did this afternoon and it did not work. decompress u said? i thought it was compress, as it shrinks mine by 1/8 or so. let me try again compressed then I will let u know asap.

.



#89 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 25 March 2019 - 09:46 PM

no alacrán, it compresses fine (932mb --> 239mb), the counter shows loading, but it stops there with a wrong table message that has always been there after the (hd0,0) sign on the top left.

i do not know if I have made myself understood, I meant compressing the dynamic vhd, as in:

 

title Windows 10 - RAMDISK1
find --set-root --ignore-floppies /vhd.vhd.lz4
map --top --mem /vhd.vhd.lz4 (hd)
map --hook
root (hd-1,0)
chainloader /bootmgr
 
I have been able to boot without the lz4 so far, so I am still at yesterday.


#90 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 25 March 2019 - 09:47 PM

NO, DO NOT compress anything, they call it decompress expandable VHD where I read it, but AFAIK it is read/open/load/decode something like that the meaning we can give to the expression, so again we are not talking in compress anything, Okay?

 

Included this info on previous post now.

 

For more info read:

 

is there no one who can look into replacing lzma with LZ4?

EDIT: LZ4 is now implemented...

 

EDIT: I missunderstud the info I read, karyonix has already clarified the info in post No. 93

 

LZ4 is a compression algorithm. Its functionality is comparable to gzip. LZ4 decompression is faster than gzip.
GURB4DOS can decompress .lz4 files like .gz files, with the condition that lz4 files were created with --content-size option.

LZ4 is unrelated to dynamic VHD. What is broken in dec 2017 is dynamic VHD. LZ4 is not affected.

 

But one thing remains valid if you compress an expandable VHDs it CAN NOT read/load.

 

Excuseme for wrong info, but at least it is clear now.



#91 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 25 March 2019 - 10:23 PM

I did not know compressing was a taboo now. ok, dynamic vhd fast enough.



#92 tinybit

tinybit

    Gold Member

  • Developer
  • 1175 posts
  •  
    China

Posted 26 March 2019 - 04:11 AM

Just to let you know after testing it Rambooting expandable VHDs, I can tell you with no doubt Rambooting expandable VHDs is working very fine now.

 

alacran

I am grateful. And heartfelt thanks to everyone here. And thank yaya.


  • alacran and antonino61 like this

#93 karyonix

karyonix

    Frequent Member

  • Advanced user
  • 481 posts
  •  
    Thailand

Posted 26 March 2019 - 01:13 PM

do not worry alacrán, wonko does not understand me, not u.
there is still one thing pending, lz4. wonderful compression, but no dice on loading. who introduced the idea of compressing? who introduced the idea of dynamic vhds? I honestly do not know. I presume someone or other did. to me, they were just out there, even though I remember alacran mentioning them at different times, but if it had been pamelo, i would certainly not have refused it.
for lz4, I remember karyonix mentioning it, but I do not know if it was with a view to loading them for booting. I consider it interesting, if we could. A couple hours ago I contacted the lz4 man, asking how we could have g4d load (which it does, but unfortunately stops) the lz4-compressed vhd and allow the booting, and he said to contact the g4d crew, as it was not on lz4's part. 
back to the previous testing discrepancy, any comparison is possible, I believe, only provided we knew what we were comparing. Now i do not know what hardware or software architecture alacran has, but I am sure he has good reason to concentrate on the preloading stage, the remnant stage presumably being sort of easy and smooth for him. me, instead, I have no issues in preramloading, as it is almost instant; my issues are on the next stage instead, as my architecture must for some reason slow things down. When I insist that here the only thing that is slow is booting, i am not only practicing rhetorics.
nino

LZ4 is a compression algorithm. Its functionality is comparable to gzip. LZ4 decompression is faster than gzip.
GURB4DOS can decompress .lz4 files like .gz files, with the condition that lz4 files were created with --content-size option.
 

LZ4 is part of the algoritm used to decompress (read/open/load/decode) expandable VHDs, it was added by karyonix to grub4dos long time ago, and was working fine, but latter on Dec 2017 some one commit a mistake and dissabled the 2 lines required for it to work,  and it is fixed and working on the very last grub4dos 046a just released: http://reboot.pro/to...e-3#entry210171

LZ4 is unrelated to dynamic VHD. What is broken in dec 2017 is dynamic VHD. LZ4 is not affected.
 

no alacrán, it compresses fine (932mb --> 239mb), the counter shows loading, but it stops there with a wrong table message that has always been there after the (hd0,0) sign on the top left.
i do not know if I have made myself understood, I meant compressing the dynamic vhd, as in:
 
title Windows 10 - RAMDISK1
find --set-root --ignore-floppies /vhd.vhd.lz4
map --top --mem /vhd.vhd.lz4 (hd)
map --hook
root (hd-1,0)
chainloader /bootmgr
 
I have been able to boot without the lz4 so far, so I am still at yesterday.

GRUB4DOS does not support double decompression/decoding, so compressed dynamic disk VHD will not expand in GRUB4DOS.
Compressed fixed VHD can be decompressed in GRUB4DOS, and fixed VHD does not require further decoding, so it should work.
  • alacran and antonino61 like this

#94 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 26 March 2019 - 04:58 PM

so what is the typical lz4 command line for the *.lz4 file to work with g4d?

lz4 -9 ? d:\vhd.vhd?

nino



#95 cdob

cdob

    Gold Member

  • Expert
  • 1469 posts

Posted 26 March 2019 - 05:54 PM

so what is the typical lz4 command line for the *.lz4 file to work with g4d?


Mount the vhd file and zero free space first. Dismount the vhd file.
https://docs.microso...wnloads/sdelete
sdelete64.exe -z f:
.
Compress the vhd file
https://github.com/lz4/lz4/releases
lz4.exe -9 --content-size e:\win.vhd e:\win.vhd.lz4
.

Should you aim for best possible decompression speed, it's possible to request LZ4 to actively favor decompression speed, even if it means sacrificing some compression ratio in the process.

lz4.exe -12  --content-size --favor-decSpeed e:\win.vhd e:\win.vhd.lz4

  • alacran likes this

#96 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 26 March 2019 - 06:07 PM

thank you ever so much



#97 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 26 March 2019 - 06:51 PM

no dice, cdob, i did exactly as u said, sdelete and lz4 -12, it cleaned and compressed fine respectively, but yields wrong table at boot with g4d. I am still booting uncompressed dynamic vhd. 



#98 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 26 March 2019 - 09:29 PM

@ kyrionix

 

LZ4 is a compression algorithm. Its functionality is comparable to gzip. LZ4 decompression is faster than gzip.
GURB4DOS can decompress .lz4 files like .gz files, with the condition that lz4 files were created with --content-size option.
 
LZ4 is unrelated to dynamic VHD. What is broken in dec 2017 is dynamic VHD. LZ4 is not affected.
 
GRUB4DOS does not support double decompression/decoding, so compressed dynamic disk VHD will not expand in GRUB4DOS.
Compressed fixed VHD can be decompressed in GRUB4DOS, and fixed VHD does not require further decoding, so it should work.

 

Thanks for make it clear to me, I already make some notes on my previous post related to this subject with a link to your post, for future reader.

 

alacran



#99 alacran

alacran

    Platinum Member

  • .script developer
  • 2710 posts
  •  
    Mexico

Posted 26 March 2019 - 10:05 PM   Best Answer

We could say that this thread is a continuation of: Making the smallest Win10 install (Wimboot mode) on 512 MB VHD

 

It was necessary to run some additional test using the info from cdob post No. 95: http://reboot.pro/to...e-4#entry210231

 

 

lz4.exe -12  --content-size --favor-decSpeed e:\win.vhd e:\win.vhd.lz4

 

LZ4 compression is better and faster than using GZ, or any other option, it gave me the faster loading to Ram times, also noticed that when loading from an slow device (as my MicroSD) it has the best efect.

 

All test ran on same I3 3225, 3.3 GHZ, 8 GB Ram at 1333 MHZ, Rambooting from SSD Adata SU650 into Adata XPG enclosure 3.0 USB.

 

This are my findings:

 

Speed:

VHD								Time to load on RAM

1 - 10x64-WB.vhd Ramboot using g4d  grub4dos-0.4.6a-2018-12-23		12 seconds

2 - 10x64-WB.vhd.gz Ramboot using g4d  grub4dos-0.4.6a-2018-12-23	7 seconds

3 - 10x64-WB-EXP.vhd Ramboot using g4d  0.4.5c Modd by kyrionix		4 seconds

4 - 10x64-WB.vhd.lz4 Ramboot using g4d  grub4dos-0.4.6a-2019-03-25	3.67 seconds

Saved espace:

VHD				Used	Saved espace

10x64-WB.vhd			1536 MB	0 % saved espace

10x64-WB.vhd Expandable		480 MB	68.75 % saved espace

10x64-WB.vhd.gz			233 MB	84.83 % saved espace

10x64-WB.vhd.lz4		135 MB	91.21 % saved espace

It is very possible I am getting close of maximum loading speed possible for this PC and this is influencing results.

 

On a single test I ran from my MicroSD, the 10x64-WB.vhd.lz4 loaded to Ram on 14.68 seconds wich is really a very good time for that device, and it was plugged to USB 2.0

 

Conclusions:

 

If Rambooting from a fast device as an SSD use expandable VHDs, they have very good speed and a resonable saved espace, do no require any additional compression, they are capable to Ramboot and also can be booted from Windows bootmanager, so it is not a big issue if they can't boot as Filedisk using grub4dos.

 

If Rambooting from a not so fast device as an SSD, we have the alternative to use LZ4 compressed VHDs, wich are the faster option and also save more espace than any other option, but they are only usefull for Rambooting, we can not boot them as Filedik using grub4dos or from Windows Bootmanager, this limitations also apply to GZ compression.

 

Thanks to all members of the forum who collaborated with this thread with info and suggestions, for the realization of all this tests:

 

antonino61, Wonko the sane, karyonix, tinybit, cdob and yaya for fast fixing grub4dos-0.4.6a-2019-03-25

 

 

Just to make a summary of all we have learned/tested/proved/improve on this thread:

From previous Post: http://reboot.pro/to...e-4#entry210209

 

LZX compressed wim files for Wimboot VHDs do not fail anymore during boot or Ramboot. See: http://reboot.pro/to...am/#entry210021

 

With this hight compression I have found If we don't add those items on [PrepopulateList], especially the first two we will have troubles during Ramboot

 

Saving 20 % espace using LZX compressed wim files. See: http://reboot.pro/to...am/#entry210021

 

Only things we have to do in a different way to reduce about 20 % the size of our source win file is: ..........

 

Tested GZ compresed VHDs and found they load faster on Ram (almost about 1/2 time).See: http://reboot.pro/to...am/#entry210031

 

I compressed 10x64-WB.vhd with 7-zip GZ Ultra as 10x64-WB.vhd.gz , it took 15 min, but file is now 233 MB

 

Saving about 85 % espace if using GZ compresed VHDs, See. http://reboot.pro/to...am/#entry210031

 

....but file is now 233 MB so it reduced a lot the size 84.83 %,......

 

Improved speed using GZ compression on VHDs See: http://reboot.pro/to...am/#entry210031

 

So there is a big improvement in time to load in Ram (43.75 % less time)

 

Testing expandable VHDs Ramboot capability, and found it was failing.See: http://reboot.pro/to...e-2#entry210068

 

It fail to load on Ram, still verifiying to try to find if I made a mistake before say it do not work,

 

Detect the last grub4dos 0.46a version capable to Ramboot expandable VHDs. See: http://reboot.pro/to...e-2#entry210089

 

grub4dos-0.4.6a-2017-12-20 is last version capable of Ramboot dynamic VHDs, the next version grub4dos-0.4.6a-2017-12-23 Fails.

 

Thanks to karionix who detected where was exactly the two lines disabled on grub4dos 0.46a after 2017-12-20, See: http://reboot.pro/to...e-2#entry210101

 

The codes to read hard disk footer in line 185 and read dynamic disk header in line 209 were just "disabled".

 

Improved again time to load on Ram , using Expandable VHDs. See: http://reboot.pro/to...e-2#entry210074

	VHD							Time to load on ram

1 - 10x64-WB.vhd Ramboot using g4d  v0.46a Dic 2018            		12 seconds

2 - 10x64-WB.vhd.gz Ramboot using g4d  v0.46a Dic 2018            	7 seconds

3 - 10x64-WB-EXP.vhd Ramboot using g4d  0.4.5c Modd by kyrionix		4 seconds

Saved espace using expandable VHDs compared to 1.5 GB fixed size VHDs. See http://reboot.pro/to...e-3#entry210115

VHD			Used	Saved espace

10x64-WB.vhd		480 MB	68.75 % saved espace

10x86-WB.vhd		360 MB	76.56 % saved espace

8x64-WB.vhd		416 MB	72.91 % saved espace

W864ESP1-WB.vhd		364 MB	76.30 % saved espace

Got the attention of tinybit and yaya (grub4dos development team) and thanks to it, get in about 2 days a new fixed release. See: http://reboot.pro/to...e-3#entry210171

 

yaya fixed it.

 

And this new posts related to LZ4 Compression:

 

karyonix info about LZ4, See: http://reboot.pro/to...e-4#entry210229

cdob info and commands for LZ4, See: http://reboot.pro/to...e-4#entry210231
 

Analyzing why the difference of speeds: http://reboot.pro/to...e-5#entry210265

 

Comparing the 2 options suggested by cdob for LZ4 Compression: http://reboot.pro/to...e-5#entry210278

 

When having x86 and x64 versions you can make a wimboot source wim file with multiple indexes to save espace, see: http://reboot.pro/to...am/#entry210406

 

And for all lazy people wimb just made a program to make almost all this, things not available on his program are compressions of VHDs and multi-index wimboot source wim files:

 

VHD_WIMBOOT - Apply and Capture of WIM Files for OS in VHD by wimb: http://reboot.pro/to...-for-os-in-vhd/

 

And finally now, there is also available this lz4_compressor GUI: http://reboot.pro/to...lz4-compressor/

 

I only hope this thread has helped some of us to learn something.

 

alacran



#100 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 26 March 2019 - 10:43 PM

Sorry to be a stickler and contradict alacran, but here lz4 is not worth the trouble speedwise. probably in terms of saving space, it is. my test was made on a fixed vhd from hard disk, not from usb. the post loading stage is real slow, and the preloading stage is no faster, or rather, slightly slower than the dynamic vhd, which remains by far the best practicable solution to me. 





Also tagged with one or more of these keywords: wimboot, ramboot, lzx, gzip, lz4, vhd

3 user(s) are reading this topic

0 members, 3 guests, 0 anonymous users