17 replies to this topic

[RandomUser]

I came across with this old thread: [GRUB4DOS] Any alternate solution to map 7GB.vhd.GZ on RAM? and while I am not looking to compress my RAM disk for loading into RAM, but found Wonko the Sane, reply seems interesting.

 

I cannot even imagine WHY exactly one would want/need an 8 Gb image loaded in RAM , but the "right" approach IMHO is to separate the setup in a small, efficient image dedicated to boot the system and another smallish image mounted with IMDISK and expanded to the size you need it.

You can use a mount point and thus the "system" drive will be as large as you need it.

 

Wonko

This sounds like the solution what I am looking for. I have tried pooling the RAM disk (Firadisk booted) in attempt to make C: bigger by expanding it to another RAM disk using Softperfect RAM disk (a clean disk). However pooling made another drive letter with both drives compined, and this does not work as most software install parts of itself in system drive regardless which drive you chose to install the program in. I could create one big firadisk image and have grub4dos load the intire thing in RAM, but that seems like a waste of disk space, espicially if most the image will contained free (empty spaces) not ideal when wanting 64 to 128GB of RAM disk space.

So my question is there a tutorial or could a tutorial be written to allow expanding the C: drive using a smallest possible FiraDisk, to boot the system, and then expanding the partition to take up other RAM disk like IMDISK or such? Windows does not like to be installed on Dynamic disk, so cannot spann it's partition over to the next disk.

Edited by RandomUser, 14 May 2015 - 01:10 AM.

[Wonko the Sane]

I am not sure that the hinted solution in that thread may apply to your case.
Can you try expanding on what are your requisites/goals?
Which OS/Environment?
What is the intended use?
etc.


Wonko

[RandomUser]

I am tring to see if I can expand my RAMdisk which is drive C:\. Instead of having grub4dos load one big 64GB or 128GB image to make C:\ drive 64GB or 128GB in size. I want to take a smaller image (OS installed) 20GB and let grub4dos load that, and then somehow use anything else to expande the C:\ drive up to 64GB to 128GB, so that way I don't have to spend all the time needed for grub4dos to load gigs of empty space (for temparary installs of application and among other things). I don't need applications to be perminantly installed. I got Windows Server 2012 to load in RAM successfully using Firadisk driver so that would be the invironment. The problem is that some application, even if you have another RAMDisk mapped with a different letter say drive D:\ it still install part of itself on drive C:\ regardless on which drive you choose. So I am wanting to create one big C:\ RAMdisk drive. As 20GB does fill up pretty fast, and I have all those RAM just waiting to be used as RAMdisk.

So I guess I want to compine Firadisk RAMdisk with, say Softperfect or IMDisk RAMdisk together.

Hope this is clearer.

Edited by RandomUser, 14 May 2015 - 05:11 PM.

[Wonko the Sane]

Now I better see, so basically you "start fresh" every time, reinstalling after having booted *whatever* program you wish to install.

The given hint was to have a secondary (or tertiary) RAMDISK of some kind mapped instead of a drive letter (thus another volume) to a mountpoint.

 

This is more or less what happens when using *any* WinPE 2.x or later, the "normal" boot.sdi is just a "placeholder" 3 Mb in size (but it can even be reduced) hosting a NTFS filesystem to which folders (residing on a "real hard disk" or inside a .wim, etc.) are mapped via junctions.

 

The key here is understanding which are the folders that "grow" in your usage and see if they are used also in the early part of booting, or more generally if - since you are talking about Server 2012, which means more or less Windows 8/8.1, if you can take advantage of the .wim format features, including possibly the capabilities of the WOF/Wimboot driver. (the Wimboot is only available according to MS for the 8.1 and not for the server version, but it is possible that it can be made working nonetheless).

 

But if you have that much RAM (enough to actually be able to use 64 or 128 Gb of it for RamDisk(s)) and the issue is the time needed by grub4dos to load it, maybe you can workaround the issue by trunncating the "base" image and gzipping it.

The only thing that cannot be fitted in it is the "$BootSectMirr", the mirror copy of the bootsector, not something that would be actually needed in a "volatile" setup like you describe.

 

Wonko

[RandomUser]

Out of couriosity, does the new WIM file makes permenent changes to C:\ and has its own drive letter? It has been a while since I looked into WIM, and this WOF/Wimboot driver I am new at. So I guess they upgraded how WIM works? Seems like another steep learning curve fo

[Wonko the Sane]

About the truncating, yes that is what I meant, of course it needs to be tested, as it depends on grub4dos mapping the virtual partition by the size "declared" in the MBR/PBR or on actual size but I presume one could make a (md) "manually" and then dd to it the (partial) data, not entirely unlike the early "hmload.com" approach works.

You may want to create as "special" NTFS filesystem with the $MFT and the other structures "anticipated" as near to the beginning as possible, see this (seemingly unrelated) thread:

http://reboot.pro/to...disk-emulation/

 

The .wim (very basically and AFAICU) is more like a "compressed filesystem archive" than anything else, you could think of it more like a .zip file and the way it can be "integrated" in Explorer through zipfldr.dll, it is "read only", and the Wimboot feature (which is one of the very few things IMHO worth of notice in Windows 8.1) will most probably be removed/deprecated in the upcoming Windows 10 exactly because it is not "serviceable" (or not easily serviceable).

The Wimboot was born to allow the stupid mass of bloat which is 8.1 to run on low-end netbooks and tablets but in practice initially it works fine, but then each and every update pushed to the OS is stored outside it, and given the amount of updates and their size this in practice will end to have half the Wimboot contain superseded files and the actual files stored outside it.

Still, if appliable, could be something useful for "strange, new, setups" like the one you describe.

 

Yet another approach (really hard to say if doable/applicable) to reach your goal could be maybe that of *somehow* porting the "XP Kansas City shuffle" to your system, JFY the idea of it is/was to pre-boot a very minimal OS and have it (through it's faster drivers) load the rest, it proved to be useful in a couple situations (motherboards that have USB 2.0 chips but provide only 1.1 speed at BIOS level or that had not USB booting support) this would workaround to both the lowish speed of the grub4dos file copy/decompression and the 4 Gb size limit, I don't think anyone ever tested it on a Vista or later OS, though, given the tendency of the good MS guys the mechanism may still work, but really cannot say.

 

The approach originally hinted should be (if sufficient) the easiest to test, you simply build your "Base" OS and then try moving the folder (that is not really-really needed at booting time and that it is likely to grow upon installation) say \Program Files\ to another ramdisk, re-mounting it in the "main" filesystem as a mountpoint or a junction.

 

No need to say how YMMGV.

 

 

Wonko

[RandomUser]

Sorry for such a late reply, I tried to truncate the image, by removing all zeros, however failed to boot, even failed to see what format the image was ie NTFS. I tried snipping the bottom off, it works, however there is a limit you can snip off reguardless on the numbers of zeros. For instant, a 64GB image, I was able to secessfully snip off 21.3GB off the end of the image, anything more then that, fails to boot or even be recognize, regardless if there is zeros at the new ending of the snipped off images. Yet there is only 12GB of used space and the rest of the image is empty free space. I think I may be truncating the image wrong or something. I tried WinImage, however the size remains the same as original image even with truncate unused part of image is checked. I am or was currantly researching on parse image, but not a whole lot of result on how to create one in Windows. Mostly Mac OS X and dmg based image. Perhaps Parsing the image could be the key? Or perhaps I am not truncating the image properly?  I tried defraging the image in attempt to bring all the files taward the biginning of the image and away from the ending of the image and used SDelete -z to write zeros to the free space of the image. Oh well, it's been fun experimanting, perhaps someone can write an utility that can properly truncate the images. I think you may be on to something here.

Edited by RandomUser, 27 May 2015 - 06:57 PM.

[Wonko the Sane]

Well, as said, the image (if NTFS) needs to be specially crafted (do check the given thread) in order to have the NTFS filesystem structures "as early as possible", and very likely besides what is described in that thread one will need to manually move the $MFT Mirr to gain some more "space that can be truncated".

Sparse images might be OK, but ONLY IF (and WHEN) the grub4dos will be able to deal with them.

 

Wonko

[RandomUser]

Okay, so if I understood the thread correctly is to create a small NTFS size partition, like 3MB and that should force the $MFT mirr towards the beggining of the drive, if so, this does seems to work as far as $MFT mirr is located. However even after shaving off more then 21.3GB off the image, it surprisangly still failes to boot or recoqnized. Even if the image is emtpy (has nothing on it) besides the usual NTFS files like $MFT and $MFT mirr, which both were bretty close at the beggining of the disk image. I was quite surprised this didn't work. It's too bad grub4dos does not support Sparse images yet, you have save me hours of trying. Perhaps one day it will and by then maybe a way to create one easily in Windows. However I am happy that grub4dos can do what it currantly do, along with Firadisk. I just may have to put up with longer loading time, or store the image on a faster HDD like SSD.

Thank you for your help.

Edited by RandomUser, 30 May 2015 - 12:27 AM.

[Wonko the Sane]

Well, the fact that any given sector is actually full of 00's doesn't in itself mean that it won't be needed (if it is pointed at by some structure) and NTFS has quite a few "special" files and "area pointers", it's hard to say what specifically creates the issue.

 

Try playing it "the other way round".

Create the 3 Mb volume and expand it to the minimum size you need (let's say 10 Gb).

Make a copy of it.

Expand the copy to the "full size needed" (let's say 30 Gb).

Compare the differences.

 

Still I am pretty sure that you can use mountpoints/junctions to greatly limit the size of the main image, it all depends on the way you install programs (or whatever needs to be ready at boot time), I really cannot see how/why this should not be largely possible.

 

Making an managing (properly) sparse files is not something "common".

I am attaching a few related files/tools (Open Source) that I collected (if you want to play with those).

 

Wonko

 

Attached Files

•  MakeSparseFile.zip   17.77KB   555 downloads
•  mksparse.zip   23.15KB   582 downloads
•  sparse_checker.zip   14.35KB   554 downloads
•  file-utils-mingw-bin-stripped.zip   255.96KB   476 downloads
•  SparseCopier_SourceAndApp.rar   113.68KB   661 downloads

[RandomUser]

Thank you for that uploads, the tools made it so much easier to create sparse images. I did not even knew sparse images existed, let alone what they were, so that would explain why they're uncommon, how long have they been around? As for your suggestion, I got some interesting result in the MFT zone placement.
Started out as 3MB partition, then expanded it to 10GB:

Then on after that I copied that 10GB partition on another volume then expanded that to 20GB:

As you can see the mft zone is created and is located closer at the beginning of the drive as opposed to the middle. Just wanted to show you this.
An interesting thing happened after I used the sparse checker tool, at the very last of the results of possible sparsing it shows more than 21.3GB at X offset. I decided to go to that offset with an hex editor then select starting with the stated offset to the end of the supposed offset relative to the size mentioned. Of course they all were zeros selected, and I proceeded to delete the selection and saved the image and that reduced the size beyond the 21.3GB limit, now mounting the image via filedisk, Windows ask to format.
Here is where it gets interesting, Grub4DOS was able to load the image and mapped it, and Windows even booted off of the RAMdisk, while still retaining the original sized image 64GB in RAM, however, it was corruption galore. While Windows booted, but there was many corruptions and in the end Windows BSOD every time after a few minutes of runtime. The deletion part of the image has nothing but zeros, and somehow files still managed to get corrupted even when all the important NTFS files resides towards at the beginning of the volume and the cutoff was towards the end.
As for mountpoints/Junction, would I point that to an actual hard drive? It sounds like any program installed to it would make permanent changes to the file system, and a reboot will just clear the program installed list in the uninstall program option. Not much applications are needed at boot, just mostly drivers.

Again thank you for your help.

[Wonko the Sane]

For some reasons I cannot see the images you posted.
Sparse files exist since NTFS, only they are rarely used.
 
But your procedure to truncate the image (which is NOT exactly connected to making it sparse) is improper, in the sense that you have no guarantee that the tool you are using is correctly identifying the area where to truncate.
 
Forget (temporarily) about "sparse" files and tools  (they are another thing) and concentrate on the "truncate".
 
Get DMDE:
http://dmde.com/
play with it with the images in the attachment (look at the "Cluster Map").
The only file in these filesystems is a Lorem Ipsum 4096 bytes long.
The 8700kb.img is an image created as 25 Mb in size and partitioned/formatted with Partition Guru Free:
http://www.eassos.com/download.php
then shrinked to the minimum size (in that program) of 8700Kb. (somehow Partition Guru makes "more compact" filesystem structures when shrinking) and then truncated.
The 25Mb.img is the full sized image.
The  Trunc25MB.img is the same but truncated to the same size of 8700kb.img (actually at one cluster in excess to allow DMDE to see the file).
If you use (say) FSZ (part of the DSFOK package http://members.ozema...eezip/freeware/) to change the size of a copy of the Trunc25MB.img to it's full size of 26.214.400 bytes) you will obtain a perfectly working image (only without the $BootMirr at the end),
 As well if you dd the Trunc25MB.img to any device it will be exactly as if instead of 3.714.560 bytes  you had transferred the whole 26.214.400 bytes.
 
The whole point here is after having made sure that the NTFS system files are "compact enough" to place a file (the LoremIpsum4096.txt in this test) in the filesystem, and then truncate right at the end of that file.
 
I would presume that this approach would "scale" even better on GB sized images, all in all your create a 10 Gb image (or whatever minimum size can contain your needed files) , you fill it to the brim with the files you need, then you expand it to (say) 40 Gb and hopefully nothing but the $BootMirr (which as said you can ignore) should be anywhere past the 10 Gb initial size, then you can safely truncate the image to it's initial size and once restored to the device (or to a mem device in grub4dos) it will be to all effects the same as the whole 40 Gb image.
 
You might want also to test the effectiveness of compressing the image (if you are mapping it to a mem device in grub4dos) cannot say if decompressing it on-the-fly will be faster than transferring the whole image.
 

Wonko

 

Removed attachment, see post below.

[RandomUser]

This is interesting, problem is I cannot replicate the result as the one in the attachment, I think I am missunderstanding something. Perhaps maybe a video uploaded on youtube or something that shows step by step process? While I can easily shrink the partition size within the image using Partition Guru Free, which does modifies the image directly and no need to mount it. However the image remains the same size, not truncating. Do I need to manually truncate it? Then I admit that I am lost at how to resize the partition of the volume without increasing the image size. I feel that I am getting closer, to the proper truncation goal.

Thank you.

[Wonko the Sane]

 Do I need to manually truncate it? Then I admit that I am lost at how to resize the partition of the volume without increasing the image size. I feel that I am getting closer, to the proper truncation goal.

Thank you.

Sure , you have to truncate the image using (say) fsz (see above post).

 

BUT I re-checked the example files I posted and *somehow* I provided a "defective" file.

 

Try following me (with the files in the new file attached on this post)

Make a copy of Trunc25Mb.img as New25Mb.img <- this file size is 3714560

Make a copy of 25Mb.img as NewTrunc25Mb.img <- this file size is 26214400

 

In a nutshell,

fsz  New25Mb.img 26214400

will resize the (now truncated) New25Mb.img to it's "full size" (identical, with the exception of the $BootMir at the end to the "full sized" image)

Then:

fsz  NewTrunc25Mb.img 3714560

will truncate it to the minimal size of the "truncated" image.

Both images when dded/dsfoed to a device (larger than 25 Mb) will be to all practical matters identical.

 

Now if you:

FC /B 25Mb.img New25Mb.img>diff.txt

you will see how the only differences found will be from offset 0x018FFE00 and offset 0x018FFFFF, i.e. the $BootMir which is the last sector of the image (also last sector of the partition/volume and first sector outside the filesystem)

 

 

The "difficult" part is to calculate where exactly it is possible to truncate the "full sized" 25Mb.img.

If you run DMDE and open the 25Mb.img (and the NTFS volume in it) you will see in green "EBCF", whilst if you open the Trunc25Mb.img you will see "EBxF" and you will be prompted to "use this virtual volume size" (choose this) or "use this reduced volume size". The ""E" means that there are issues with the Partition Table (and the Extents in it), while the "x" instead of the "C" means that the copy of the bootsector (the $BootMir) is missing, this is "normal" given that the image is truncated.

Now go to "Tools->Cluster Map" and click OK to allow the updating of the Cluster Map. Use the Arrow Keys to navigate the Cluster Map in the lower portion of the screen, you will soon find out that the file I copied to use as "delimiter", LoremIpsum4096.txt is right after $INDEX_ALLOCATION on cluster 897/6392.

If you double click on the vertical bar | representing the file, it will open in a hex view in which at the top you have:

LBA:7239 vol.sec:7176 Clus:897 Sec:0

The LBA represents the absolute LBA of the beginning of the file, which has been intentionally created 8 sectors=1 4kb cluster=4096 bytes in size.

So 7239*512=3706368+4096=3710464 this is the minimum size at which you can truncate the image, in this test I added another cluster, i.e. 3710464+4096=3714560 (it was empty or 00's anyway and it is needed to let DMDE see the "last file" LoremIpsum4096.txt when the image is truncated).

 

Try repeating yourself the above, you will soon get the hang of the idea and of the programs involved.

 

Wonko

Attached Files

•  Truncated.zip   736.7KB   544 downloads

[RandomUser]

Your instruction is much better, and I understood it much better. I was able to follow it right to the "T" and the results works the same as the images in your attachment (I was able to replicate it). Also discovered another way of approaching this is using a hex editor, by searching the last few strings from the text file (LoremIpsum4096.txt) in the image. Then add 4096 bytes from the last byte offset in the search result, then selected from the calculated offset to the end of the image file offset, then press delete and it produced the same exact image as in the attachment. So I guess this way is for the lazy whom do not want to do formula calculation and have a nice GUI method for truncating the image. I then tried this on my RAMdisk image, and it booted up great with zero file corruptions. This truncation saved me 48.6GB of hard disk space, and even drastically reduced the loading time. Turns out I was truncating the image wrong previously in the eleventh reply. I was truncating at the beginning of the file string when using the hex editor and thus deleting the sectors where the file resides, and that is a no no LOL. That is what caused corruptions galore. However not with your instruction. Again Windows booted wonderfully without any fuss with your instructions. I think the mod or admin probably should sticky this topic and rename the title to something like "Grub4DOS reduce loading time and save HDD space via truncated image" or something of that nature and maybe move it to a more appropriate sub form if need be for better exposure. Again thank you sooo much for your help. I appreciate it a lot.

[Wonko the Sane]

Very good , happy that it works for you.

 

"As is" I doubt it can be a very useful solution as your "RAM booting" is AFAIK/AFAICT not very common and objectively it is a bit complex to replicate (though all in all you do it just once when you "prepare" the base image).

 

To automate it, It would be possible to use (say) myfragmenter:

http://www.mydefrag....Fragmenter.html

to locate the "last file" position and use the result (with a minimal automated calculation in a batch) to determine where to truncate the image, but the issue would be of course to find out which is the "last file", I can make a few tests, but I fear that it will take forever to list all files and have the last file address, while using DMDE cluster map is easy and all in all fastish, most probably the "right" way would be to analyze the $MFT.

 

I will see if I can invent some practical automated solution.

An approach could be that of making a (say) 10 Gb image, fill it with all needed, then resize it to (say) 40 Gb and re-truncate it to it's original size (minus one sector to get rid of the $BootMirr, this way there is no need to calculate anything and under Windows 7 and later diskpart can probably also be used (scripted) to make the expansion, though most probably the image mounted with a virtual disk needs to be "full size" (as a file) and can be truncated once unmounted.

 

Wonko

[Wonko the Sane]

Just for the record karyonix just implemented a grub4dos fork that can map --mem Dynamic VHD's (which are in a nutshell the VHD version of a sparse file):
http://reboot.pro/to...mic-vhd-to-ram/

 

Wonko

[RandomUser]

I didn't think of the not common aspect of this project LOL. You are very knowlegable, heck you attacted it head on regardless if it is common or not, and the end results I'd say bretty good . If you want you could still try creating an automated software, if not, that is okay too, as most likely used once and forget. I do know that imdisk reads truncated images without a problem.

That is awsome work karyonix did there. Thanks for the link. Wonder if chenell will have it permenantly implemented in his future versions of Grub4DOS. This seems like a coincidence to this topic LOL.