Jump to content











Photo
- - - - -

Reducing wimboot source wim file using LZX Compression, and VHD using gzip or LZ4 Compression, to save room, and also load faster on RAM

wimboot ramboot lzx gzip lz4 vhd

Best Answer alacran , 26 March 2019 - 10:05 PM

We could say that this thread is a continuation of: Making the smallest Win10 install (Wimboot mode) on 512 MB VHD

 

It was necessary to run some additional test using the info from cdob post No. 95: http://reboot.pro/to...e-4#entry210231

 

 

lz4.exe -12  --content-size --favor-decSpeed e:\win.vhd e:\win.vhd.lz4

 

LZ4 compression is better and faster than using GZ, or any other option, it gave me the faster loading to Ram times, also noticed that when loading from an slow device (as my MicroSD) it has the best efect.

 

All test ran on same I3 3225, 3.3 GHZ, 8 GB Ram at 1333 MHZ, Rambooting from SSD Adata SU650 into Adata XPG enclosure 3.0 USB.

 

This are my findings:

 

Speed:

VHD								Time to load on RAM

1 - 10x64-WB.vhd Ramboot using g4d  grub4dos-0.4.6a-2018-12-23		12 seconds

2 - 10x64-WB.vhd.gz Ramboot using g4d  grub4dos-0.4.6a-2018-12-23	7 seconds

3 - 10x64-WB-EXP.vhd Ramboot using g4d  0.4.5c Modd by kyrionix		4 seconds

4 - 10x64-WB.vhd.lz4 Ramboot using g4d  grub4dos-0.4.6a-2019-03-25	3.67 seconds

Saved espace:

VHD				Used	Saved espace

10x64-WB.vhd			1536 MB	0 % saved espace

10x64-WB.vhd Expandable		480 MB	68.75 % saved espace

10x64-WB.vhd.gz			233 MB	84.83 % saved espace

10x64-WB.vhd.lz4		135 MB	91.21 % saved espace

It is very possible I am getting close of maximum loading speed possible for this PC and this is influencing results.

 

On a single test I ran from my MicroSD, the 10x64-WB.vhd.lz4 loaded to Ram on 14.68 seconds wich is really a very good time for that device, and it was plugged to USB 2.0

 

Conclusions:

 

If Rambooting from a fast device as an SSD use expandable VHDs, they have very good speed and a resonable saved espace, do no require any additional compression, they are capable to Ramboot and also can be booted from Windows bootmanager, so it is not a big issue if they can't boot as Filedisk using grub4dos.

 

If Rambooting from a not so fast device as an SSD, we have the alternative to use LZ4 compressed VHDs, wich are the faster option and also save more espace than any other option, but they are only usefull for Rambooting, we can not boot them as Filedik using grub4dos or from Windows Bootmanager, this limitations also apply to GZ compression.

 

Thanks to all members of the forum who collaborated with this thread with info and suggestions, for the realization of all this tests:

 

antonino61, Wonko the sane, karyonix, tinybit, cdob and yaya for fast fixing grub4dos-0.4.6a-2019-03-25

 

 

Just to make a summary of all we have learned/tested/proved/improve on this thread:

From previous Post: http://reboot.pro/to...e-4#entry210209

 

LZX compressed wim files for Wimboot VHDs do not fail anymore during boot or Ramboot. See: http://reboot.pro/to...am/#entry210021

 

With this hight compression I have found If we don't add those items on [PrepopulateList], especially the first two we will have troubles during Ramboot

 

Saving 20 % espace using LZX compressed wim files. See: http://reboot.pro/to...am/#entry210021

 

Only things we have to do in a different way to reduce about 20 % the size of our source win file is: ..........

 

Tested GZ compresed VHDs and found they load faster on Ram (almost about 1/2 time).See: http://reboot.pro/to...am/#entry210031

 

I compressed 10x64-WB.vhd with 7-zip GZ Ultra as 10x64-WB.vhd.gz , it took 15 min, but file is now 233 MB

 

Saving about 85 % espace if using GZ compresed VHDs, See. http://reboot.pro/to...am/#entry210031

 

....but file is now 233 MB so it reduced a lot the size 84.83 %,......

 

Improved speed using GZ compression on VHDs See: http://reboot.pro/to...am/#entry210031

 

So there is a big improvement in time to load in Ram (43.75 % less time)

 

Testing expandable VHDs Ramboot capability, and found it was failing.See: http://reboot.pro/to...e-2#entry210068

 

It fail to load on Ram, still verifiying to try to find if I made a mistake before say it do not work,

 

Detect the last grub4dos 0.46a version capable to Ramboot expandable VHDs. See: http://reboot.pro/to...e-2#entry210089

 

grub4dos-0.4.6a-2017-12-20 is last version capable of Ramboot dynamic VHDs, the next version grub4dos-0.4.6a-2017-12-23 Fails.

 

Thanks to karionix who detected where was exactly the two lines disabled on grub4dos 0.46a after 2017-12-20, See: http://reboot.pro/to...e-2#entry210101

 

The codes to read hard disk footer in line 185 and read dynamic disk header in line 209 were just "disabled".

 

Improved again time to load on Ram , using Expandable VHDs. See: http://reboot.pro/to...e-2#entry210074

	VHD							Time to load on ram

1 - 10x64-WB.vhd Ramboot using g4d  v0.46a Dic 2018            		12 seconds

2 - 10x64-WB.vhd.gz Ramboot using g4d  v0.46a Dic 2018            	7 seconds

3 - 10x64-WB-EXP.vhd Ramboot using g4d  0.4.5c Modd by kyrionix		4 seconds

Saved espace using expandable VHDs compared to 1.5 GB fixed size VHDs. See http://reboot.pro/to...e-3#entry210115

VHD			Used	Saved espace

10x64-WB.vhd		480 MB	68.75 % saved espace

10x86-WB.vhd		360 MB	76.56 % saved espace

8x64-WB.vhd		416 MB	72.91 % saved espace

W864ESP1-WB.vhd		364 MB	76.30 % saved espace

Got the attention of tinybit and yaya (grub4dos development team) and thanks to it, get in about 2 days a new fixed release. See: http://reboot.pro/to...e-3#entry210171

 

yaya fixed it.

 

And this new posts related to LZ4 Compression:

 

karyonix info about LZ4, See: http://reboot.pro/to...e-4#entry210229

cdob info and commands for LZ4, See: http://reboot.pro/to...e-4#entry210231
 

Analyzing why the difference of speeds: http://reboot.pro/to...e-5#entry210265

 

Comparing the 2 options suggested by cdob for LZ4 Compression: http://reboot.pro/to...e-5#entry210278

 

When having x86 and x64 versions you can make a wimboot source wim file with multiple indexes to save espace, see: http://reboot.pro/to...am/#entry210406

 

And for all lazy people wimb just made a program to make almost all this, things not available on his program are compressions of VHDs and multi-index wimboot source wim files:

 

VHD_WIMBOOT - Apply and Capture of WIM Files for OS in VHD by wimb: http://reboot.pro/to...-for-os-in-vhd/

 

And finally now, there is also available this lz4_compressor GUI: http://reboot.pro/to...lz4-compressor/

 

I only hope this thread has helped some of us to learn something.

 

alacran

Go to the full post


  • Please log in to reply
165 replies to this topic

#151 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 09 April 2019 - 08:58 AM

here are the files u told me to extract:

Hmmm. :dubbio:

 

There is something that is not "right", it seems like you have a number of duplicated metadata.

Some are "found one", i.e.marked with an "f", which should mean that you ran a recovery or a reconstruction before making the filelist :unsure:.

 

To make sure, please try again.

 

This time:

1) Start DMDE
2) Choose in the left pane "Logical disks"

3) Choose in the right pane the drive letter of the mounted image

4) Press OK

5) in the window you should see a line with the drive letter,  "Logic disk", the size of the image and a few letters like EBCF in green, select this line

6) click on Open Volume

7) in the left pane select $Metadata, then right click ->choose Recover Files ->click on the List... button -> File List ... and save the file as .txt,

8) repeat from #1 for the second image

 

:duff:

Wonko



#152 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 09 April 2019 - 04:49 PM

Dear wonko, in the meantime, last night, before u wrote, I managed to transfer all log directories containing 128mb-or-so *.etl files to z:\ (the temp ramdisk I have) and symbolically linked them back to their respective original places in the dir structure of c:\. so most of them did not show up at next reboot, pls don't ask me why because I know u know. 

now that I found ur post, I did as u said, so, here are the 2 files

Attached Files



#153 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 09 April 2019 - 05:43 PM

Good :)

now the files look fine.

 

There is no difference in size of these $Metadata, so the difference shown above must have been due to *something* else. :dubbio:

 

Does the .vhd still "grow" now that you have moved those .etl files off the volume?

 

The $MFT seems a bit largish, maybe using some *strange and dangerous* technique it can be reduced, but I don' t know if it is worth the time/effort :unsure:.

 

More or less the approach (that might need to be adapted to .vhd's and particularly to dynamic ones) for a plain device/image is to create the smallest possible NTFS filesystem (with the same cluster size of the final one) and then enlarge it, as seen here (only seemingly completely unrelated thread):

http://reboot.pro/to...disk-emulation/

 

The problem with this approach may be that then the $MFT is too d@mn small and will most probably become fragmented, so the initial size needs to be found by converging attempts.

 

If roughly you have 15000 files and 4000 directories on that volume, even using a 50% "slack", 20000*1.50*1024=30,720,000 a $MFT around 30 MB should be good enough, saving some 70 MB.

 

 

:duff:

Wonko



#154 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 09 April 2019 - 06:20 PM

Dear Wonko, I also played with cluster size even before the "vhd era", and went from smallest to largest, as my usual mindset suggests (no way to find the happy medium without first going thru extremes), and I realized that the best of all possible worlds is what microsoft envisaged, default cluster size, for your long standing theory (what u gain in space u lose in time). so the trade-off is just the present cluster size. then, if u tell me that the gain is 70 megs, I honestly do not know if it is worth the trouble.  The only thing I can think of in these respect is ... can u  recommend a filesystem that works better than ntfs in windows?

As for the free space difference that u were asking me about,  the temp ramdisk is doing the trick; so far, I have moved, all *.etl file dirs + softwaredistribution (the update culprit dir) to z:\ and symbolically linked them back to their respective positions in the c:\ file structure. the only one that seems to reside or be recreated in z:\ is the wdi dir. all others are missing to my eyes, so I can tell u that the symlinks on c:\ seemingly point to nothing visible anywhere. the difference is definitely less than 70megs between reboot and many hours later. Tomorrow morning I will be able to tell u if this happy situation has persisted overnight.

Besides, how r u? r u back out of the asylum in the end?



#155 blackbalfur

blackbalfur

    Member

  • Members
  • 82 posts
  •  
    Netherlands

Posted 09 April 2019 - 07:21 PM

@antonino61 why do you make the vhd space an issue if you have 32 gb of ram?

 

When you have your optimal fixed size vhd with all the software you ever need you capture it to a wimboot file.

 

If you use that optimall wimboot file to create a 12 gb dynamic vhd file and only use it to boot into ram it can grow whatever size it wants.

 

those 12gb will boot into ram in seconds.

 

the next reboot all bulging files are gone.

 

There is no problem.


Edited by blackbalfur, 09 April 2019 - 07:23 PM.


#156 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 09 April 2019 - 10:39 PM

My dear blackbalfur, 

I do not have 32gb of ram, but twice as much ram. I did try what u proposed and timed it, only to find out that it takes half a second less time to boot than, and the same time to run apps as my original 1.5gb vhd. is this what u consider worth it? I honestly do not, unless there is something else I am missing out on. 

nino

Ps: the test above was made by booting with bootmgr; I have just made the same test, this time with g4d: the 12gb vhd takes one and a half seconds more to boot than, and the same timing to operate apps as my original 1.5 vhd. 

the used space in the 12gb vhd does not seem change at all after running some apps or doing some web browsing, but I do not know whether this is due to real space consolidation or to the used/free space ratio which has changed on account of the greater overall vhd size. by way of comparison,  the original vhd changes by 2-10 bytes under the same conditions. if my findings are so far away from the expected ones, pls let me know.



#157 blackbalfur

blackbalfur

    Member

  • Members
  • 82 posts
  •  
    Netherlands

Posted 09 April 2019 - 11:32 PM

If you made your 12 gb dynamic vhd file with the wimboot file you should never again boot it up with bootmanager.

 

dynamic vhd normally booting with bootmanager gives you growing that will stay.

 

You say it yourself and you do not see it.

 

 

 

 

the used space in the 12gb vhd does not seem change at all after running some apps or doing some web browsing, but I do not know whether this is due to real space consolidation or to the used/free space ratio which has changed on account of the greater overall vhd size.

 

If you make your dynamic vhd 12 gb there is no worry that it will slow your os down because there is no lack of space.

 

1 and a half seconds is your price for letting applications run as they want and you do not have to bother about it.

 

But again you should only use dynamic vhd's in combination with ram booting. 

 

When you reboot the vhd all growing stuff has gone away and you start fresh again.

 

1 time booting your dynamic vhd with bootmanager and you have ruined your smallest size dynamic vhd.

 

But then again you can create a new one in seconds with your wimboot file.


Edited by blackbalfur, 09 April 2019 - 11:38 PM.


#158 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 10 April 2019 - 12:50 AM

My dear blackbalfour, 

symbolic links of critical directories to temp ramdisk have done the trick in the first place, so much so that space no longer grows with bootmgr either (the amount of space occupied never did change in either file in either booting way (only a negligible difference in the 1.5gb vhd and none at all in the 12gb vhd) (my doubt was purely hair-splitting)).

the substantial thing that I was at pains trying to point out is that the meager performance gain does not justify the amount of space on disk (u would not even notice a 1-and-a-half-second difference, so it is better to save space (1.5gb vs 12gb)).

nino



#159 blackbalfur

blackbalfur

    Member

  • Members
  • 82 posts
  •  
    Netherlands

Posted 10 April 2019 - 02:17 AM

If you have done it right the amount of disk space is smaller with the 12 gb dynamic vhd then with your 1,5 gb fixed size vhd.

The 12 gb only lives in ram.

#160 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 10 April 2019 - 05:37 AM

my 1.5gb one is dynamic as well



#161 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 10 April 2019 - 08:26 AM

Dear Wonko, I also played with cluster size even before the "vhd era", and went from smallest to largest, ....

 

Well, the proposed (to be tested) approach has nothing to do with changing cluster size as a matter of fact it is about keeping the cluster size exacty as is and shrinking to the minimum needed the $MFT, but as said I doubt it has any relevance in this case.

 

About filesystems, you (we) are limited to NTFS anyway (besides being IMHO the "best" filesystem) it is the only one available to Windows that allows hardlinks/softlinks/junctions and mountpoints, so in this case it is actually *needed*.

 

:duff:

Wonko


  • antonino61 likes this

#162 blackbalfur

blackbalfur

    Member

  • Members
  • 82 posts
  •  
    Netherlands

Posted 10 April 2019 - 10:51 AM

my 1.5gb one is dynamic as well

 

Fist of all you choose whatever setup you think is best.

 

but you mention the size of the vhd's on disk and they are both dynamic so the 12gb will be a little bigger.

 

You choose a cramped box where every test you want to do with can/will fill up your free space.

 

Lets say you want to try to run a game with long loading scenes in to ram and see if it has influence (it will).

 

Without moving parts the game will be at its fastest but you will have no space to try something like this.

 

And the funny part is you have so much space.

 

Everyone makes his/her own choice and do what he/she thinks works best.



#163 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 10 April 2019 - 11:46 AM

games here do not affect free disk space on c:\ so much as text files "paradoxically" do.



#164 blackbalfur

blackbalfur

    Member

  • Members
  • 82 posts
  •  
    Netherlands

Posted 10 April 2019 - 11:51 AM

You can not test a game into ram that is 2gb in size on your 1.5gb vhd.

 

But it seems that we have different believes on the power of ram. 

 

You think in space i think in speed.



#165 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 10 April 2019 - 02:03 PM

oh now I get u when u say "test a game", I did not think u meant to test it in ram, I dont run games in ram, the few times I did was thru an ad hoc ramdisk and it was unreal fast. no need. I simply meant launching a game the conventional way. 



#166 antonino61

antonino61

    Gold Member

  • Advanced user
  • 1525 posts
  •  
    Italy

Posted 10 April 2019 - 05:01 PM

BTW, I can confirrm, after a few hours of intense variegated activity, that with this symbolic link system whereby directories containing "spontaneously created *.etl files" are "diverted" to ram (z:\ is my temp ramdisk), used space and free space in the vhd are totally under control even though the vhd has been booted thru bootmgr.





Also tagged with one or more of these keywords: wimboot, ramboot, lzx, gzip, lz4, vhd

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users