32 replies to this topic

[MedEvil]

I look for some input and opinions, on what's possibly the best setup for a PE.
Especialy the advantages and disadvantages of certain solutions.

Early PE1 projects (XP) used for the most part, because of the spotty performance of FBWF, a user directory and temp dir on a ram disk.

Later PE1 projects, which used a HDD image in RAM, put the user directory and temp dir back onto the ramloaded HDD image.

Win7 projects rely completely, to my knowledge, on the use of FBWF and have the user directory and the temp dir both in the FBWF overlay.


So far the biggest drawback to pure FBWF usage, imo, is that it's size is rather limited and clutters easily, because it is hard to tell, what files exist in the wim and which exist in the overlay.
Advantages: driver installation possible, possibly advantages when installing software

The drawback to pure HDD usage is, that the size of the HDD has to be set at build time and can not adapt to the available resources. (If anyone knows, how to make a expanding HDD, i'm interested!)
Advantagea are the same as for the above.

The drawback to ramdisk usage is that it's impossible to install drivers, which weren't included at build time.
Main advantage of this setup, is that the ramdisk can adapt to available hardware without size limits and it is very easy to keep an eye on resource consumption, as well as freeing of resources, when needed.

And then there is of course the ramdisk vs. filedisk question.
Filedisk offer faster booting on the cost of worst overall performance, while ramdisk offers great performance on the cost of longer boot times.
Filedisks can be huge without impact on performance. Also they can be writable, which is helpful on RAM challended systems.
RamDisks on the other hand, have to be kept small, to keep the boot times at a usable level. Advantage the boot media can be removed and used on another computer. Ramdisks give also the best user experience on any hardware, given there's enough RAM available to load and use them.

Did i miss any advantages or disadvantages? What's your opinion on the ideal setup?

I go at the moment with a combination of user directory and temp dir on a ramdisk and a write protected filedisk, with a ton of preinstalled drivers.

[misty]

@MedEvil
My own preference is for a writable system drive - I'm not overly bothered about which method is used to achieve this and have various builds with different methods. At the moment I prefer to use WinPE 3.0 (default Microsoft Windows Automated Installation Kit build with minor customisation) or LiveXP (ramload using a RAW hard disk image format).

I had to dig out an old FBWF version of BartPE recently when fixing an old computer with only 128MB of RAM though. Never had any issues with my FBWF files - although I'm not sure of their source.

The reason I prefer a writable system drive is due to some programs (and obviously drivers as you mentioned) attempting to write to the system. I remember reading about one old BartPE plugin (can't remember which one unfortunately) that the author only managed to get working by accident as he was using a writable system drive as standard.

Regards,

Misty

[Wonko the Sane]

@Medevil
I am not sure to get your question.
What about a Firadisk directly mapped HD sparse image?
Has it ever been attempted?
Would it be supported or it would just crash the system?
Or would it be automatically expanded to max size during the loading? (I don't think so if "directly" - not RAM - mapped)

If not possible, the "right" way to boost booting speed (as I see it) is still going back to the basics and create a very small "main" PE which is actually booted and have a second image/filesystem that is loaded later and that contains all programs, etc.

AFAICR, this approach has never been widely used/adopted, but still it is the one that most (still IMHO) makes sense: if you have something that is slow at loading and you cannot find a way to make that loading faster, load LESS amount with the slow method and load the rest with a faster method.


Wonko

[Wonko the Sane]

I made a double post .
To be hopefully forgiven for this brain fart , I will use the space already occupied by it to re-vamp this:
http://reboot.pro/11340/

One of the n nice things left in an UNfinalized state....


Wonko

[MedEvil]

What about a Firadisk directly mapped HD sparse image?

That's exactly, what i was thinking.
Unfortunately, i have never seenm, heared or read about any such bootable ramdisk being used.
You say it should work with firadisk?

If not possible, the "right" way to boost booting speed (as I see it) is still going back to the basics and create a very small "main" PE which is actually booted and have a second image/filesystem that is loaded later and that contains all programs, etc.

AFAICR, this approach has never been widely used/adopted, but still it is the one that most (still IMHO) makes sense:

Can't speak for everyone else, but the reason i never used this kind of setup is, that Windows is not Linux. In Windows one can not have a floppy sized boot image.
A Windows boot image needs to include the whole %systemroot%, which in my PE is 75% and more of the whole size of the PE.
If one would be able to mount additional images to non-empty system folders, i would switch to this setup right away.

But the way things are, it just doesn't offer enough benefit, imo.
With full USB2.0 bootspeed the boot time reduction over a single Ramdisk is neglectable and with USB1.1 boot speed, it doesn't improve boot speed enough to be considered a viable alternative to a filedisk or running the PE straight of the media.

If anyone would run a PE with 100MB %systemroot% and 1GB Program files folder, i would agree, however.

[pscEx]

That's exactly, what i was thinking.
Unfortunately, i have never seenm, heared or read about any such bootable ramdisk being used.
You say it should work with firadisk?

I do not know whether I understood correctly, but IMO multiPE does that.

Contents of ISO:

Datentr„ger in Laufwerk F: ist multiPE_XP
Volumeseriennummer: 74FF-D28F

Verzeichnis von F:\

07.12.2011 18:20 <DIR> I386
07.12.2011 18:20 128 menu.lst
1 Datei(en) 128 Bytes

Verzeichnis von F:\I386

07.12.2011 18:20 <DIR> .
07.12.2011 18:21 <DIR> ..
07.12.2011 18:19 1.072 ProjectInfo.ini
07.12.2011 18:20 115.153.920 multiPE_RAM_XP.IMG
2 Datei(en) 115.154.992 Bytes

Anzahl der angezeigten Dateien:
3 Datei(en) 115.155.120 Bytes
3 Verzeichnis(se), 0 Bytes frei

menu.lst:

timeout 0
default 0
debug off
hiddenmenu

map --mem /I386/multiPE_RAM_XP.IMG (hd0)
map --hook
chainloader (hd0,0)/NTLDR

Peter

[Wonko the Sane]

That's exactly, what i was thinking.
Unfortunately, i have never seen, heared or read about any such bootable ramdisk being used.
You say it should work with firadisk?

I suspect it cannot work with "ramdisk" (as the image would be expanded during loading) but it may work for a "direct" mapped image.
As said I have never attempted experiments with this approach nor I have seen anyone interested enough to do that.

Can't speak for everyone else, but the reason i never used this kind of setup is, that Windows is not Linux. In Windows one can not have a floppy sized boot image.
A Windows boot image needs to include the whole %systemroot%, which in my PE is 75% and more of the whole size of the PE.
If one would be able to mount additional images to non-empty system folders, i would switch to this setup right away.

Here I beg to disagree.
As we have seen with the "kicker CD" and with the "XP Kansas City Shuffle/Fake Signature" methods, the amount of actually *needed* files at boot time is actually relatively little.
The %systemroot% directory is more (IMNSHO) a myth than an actual *need*.
I mean, the good MS guys have created a "monolithic" directory structure made (I am talking of "full" XP but the principle/ideas can be - at least partially - exported to a PE 1.x) like:
C:\WINDOWS
C:\WINDOWS\system
C:\WINDOWS\system32
where they "dumped" *everything*.
That is very convenient for some tasks, but it is not for *every task*.
What I have been experimenting in XPCLI (with only very partial results, since I play that game alone and it's just something I do as a past-time) is to have the least possible files in the three folders above.
This can be done by making a folder for the app example for chkdsk:

where I put, besides the file, also ALL it's dependencies (.dll's) that are NOT in C:\Windows\System32\ as usual (and that are not needed for *very basic* booting).
This way, each time I add an app, I have some (please read as *many*) duplicate files (a copy in each folder of the app that needs the given .dll) but I effectively reduce the size of the C:\Windows.
Once the system is "complete", nothing prevents from substituting the folder with junctions or mountpoints and de-duplicate the redundant .dll's with hard-links.


Wonko

[MedEvil]

I do not know whether I understood correctly, but IMO multiPE does that.

Sorry, but what part exactly, of your post, should tell me, that you used a spatse file?

[pscEx]

Sorry, but what part exactly, of your post, should tell me, that you used a spatse file?

Whan you explain to me, what "spatse" means, perhaps I can explain.

But understanding Wonko's post, your requirement is not fullfilled with a mapped RAM drive, like multiPE does.

Peter

[MedEvil]

Here I beg to disagree.
As we have seen with the "kicker CD" and with the "XP Kansas City Shuffle/Fake Signature" methods, the amount of actually *needed* files at boot time is actually relatively little.

Yes it is. The problem is, that we can't have pre kernel and post kernel files seperated. Win7 tries to do something like this, but it's boot.wim is still huge compared to Linux initdisk.

What I have been experimenting in XPCLI (with only very partial results, since I play that game alone and it's just something I do as a past-time) is to have the least possible files in the three folders above.
This can be done by making a folder for the app example for chkdsk:
where I put, besides the file, also ALL it's dependencies (.dll's) that are NOT in C:\Windows\System32\ as usual (and that are not needed for *very basic* booting).

The problem with that 'DOS approach' is, that it won't work, if one want's or needs a more full blown Windows/PE.

However, if it would be possible to load and mount a second image soon enough and then create hardlinks for all the files from the second disk in the initial ramdisk, before continue booting, then we would have a winner.

[MedEvil]

Whan you explain to me, what "spatse" means, perhaps I can explain.

A sparse image is an image of a disk, which does not have a static size, but grows upon need, until it reaches it's max. size.

But understanding Wonko's post, your requirement is not fullfilled with a mapped RAM drive, like multiPE does.

There is no requirement, per se. I hope to discuss the advantages and disadvantages of different approachs, in order to find a better solution, than what i have at the moment.

I think, we all want the same, the best of both worlds. The fast booting of file backed disks and the superior performance of ramdisks.

Who knows, maybe we can come up with something completely new and revolutionary!

[Wonko the Sane]

The problem with that 'DOS approach' is, that it won't work, if one want's or needs a more full blown Windows/PE.

I again beg to disagree, the "separation line" between "strictly needed for booting" and "can be mounted slightly later" is at a much smaller level then what you seem like thinking.
As I see it, the "bare minimum" are the files in the "Kicker CD".
Of course the approach I described is - for obvious reasons - much more "integralist" then *any other* one, but I still think that you can remain below or around 50 Mb.

@psc
The general concept of a sparse file - simplified - (not necessarily a file that represents a disk image) is that any sector in it which is 00 filled is NOT in the actual file, but only "indexed as empty".
Most copying tools - including of course copying it to RAM by grub4dos (provided that grub4dos supports sparse files, something I never tested) - will expand the file to it's full size.

From time to time you may want to actually READ into the links I provide:
http://reboot.pro/5000/page__st__3
http://www.opalapps....se_checker.html
http://reboot.pro/6745/page__st__29


Wonko

[MedEvil]

I again beg to disagree, the "separation line" between "strictly needed for booting" and "can be mounted slightly later" is at a much smaller level then what you seem like thinking.
As I see it, the "bare minimum" are the files in the "Kicker CD".

Do you have (by chance ) a link to this kickerCD or it's explaination?

Of course the approach I described is - for obvious reasons - much more "integralist" then *any other* one, but I still think that you can remain below or around 50 Mb.

50MB flat or already compressed?

[Wonko the Sane]

I would say around 50 Mb "flat".

Do you have (by chance ) a link to this kickerCD or it's explaination?

Are you actually doubting that I have not a handful of links up my sleeve?
http://www.911cd.net...showtopic=24763
http://www.911cd.net...showtopic=21242
http://www.911cd.net...showtopic=21702
http://www.911cd.net...showtopic=21939
http://www.911cd.net...pic=24742&st=26

And, completely UNRELATED to the above, loading a PE thorough NTLDR (re-tested and seemingly working on HD-like device - hit a roadblock on CD ):
http://reboot.pro/12339/
The times reported in Alexei32's original posts are impressive (adapted from):
http://forum.ru-boar...18&start=380#16

		   setupldr.bin ntldr

PE build #1:	  2:10	  1:00

PE build #2:	  3:20	  1:35

PE build #3:	  0:55	  0:20

Wonko

[MedEvil]

Thanks for the links!

I would say around 50 Mb "flat".

Looked a bit through my old papers.
PicoPE - with CMD shell ~ 50MB
NativeEx - with explorer shell ~ 70MB
NaughtyPE - with explorer shell and sound and ... ~ 100MB

And, completely UNRELATED to the above, loading a PE thorough NTLDR (re-tested and seemingly working on HD-like device - hit a roadblock on CD

Well, there is a Longhorn ntldr, which is XP compatible and enumerates CDVD drives by order of detection, with single digit numbers, instead of using those strange 3 digit ident numbers.
But since we do not redistribute M$ files and the iso couldn't any longer be downloaded, we never made this public.

The times reported in Alexei32's original posts are impressive

I think we talked about this before. Aside from setupldr using txtsetup.sif, in addition to the registry, i can not see, how this improvement can possible work.
And getting rid of txtsetup.sif is really simple, even with setupldr.

[Wonko the Sane]

I think we talked about this before. Aside from setupldr using txtsetup.sif, in addition to the registry, i can not see, how this improvement can possible work.
And getting rid of txtsetup.sif is really simple, even with setupldr.

I don't think we did.
In any case, that is a possible way to shave off some relevant time off booting and - though of course solving the CD issue would be interesting - I guess that (say) 90% of PE booting is done from HD like media.
The thingy is now fully repeatable, why don't you try it?
If it works for you also, then I think that yes it can be "ported back" to SETUPLDR.BIN.
From the little I can understand, it works by "subtracting" some devices from the "normal" PE/NTDETECT.COM routine, possibly "as is" it is not much portable, but I think it can have it's merits even if you re-instate some detection tothe "normal" PE booting.


Wonko

[MedEvil]

I might would have tryed, but the fact that this methode gives faster boot time, by being less usable, discourages me from actually doing so.

I mean, i could disable HwPnP and empty out txtsetup.sif and regsetup.hiv too to make NaughtyPE boot in less than a minute, but that would also make it pretty useless as a portable system.

For a customized PE or Mini Windows, that runs from HDD and works just on that computer, the time to beat is still ~15 seconds, with all devices working.

[MedEvil]

Ok, i have read now all the threads you linked to and they all have the same idea.

- boot from drive A - the textbased setup
- confuse Windows to continue from drive B

For this to work, both A snd B need to be accessable.

For use with ramdisks, drive B would first need to get created and loaded before Windows can switch.
On top of that, Windows would already have to have switched from BIOS routines to own drivers, or the whole procedure would not give any speed increase, for the loading, over using a single image.

I don't think the switching happens at the end of the textbased setup, but at the beginning of the graphical setup.
Please correct me someone, if i'm wrong!

[cdob]

- boot from drive A - the textbased setup
- confuse Windows to continue from drive B

Or boot from drive A and continue at drive A too.

Read whole textmode part at one continuous read.
This is faster in comparison to several single reads.

[Wonko the Sane]

Or boot from drive A, continue at drive A and have "all the rest" in drive B (that can be read "later" in the boot process).

I presume that you already shaved off (with the axe ) everything that you could have removed with the axe, now is probably time to use a razor instead and scratch even tiny bits of time.

I mean, I guess that the scope is to try and combine together several different approaches in the hope of trimming down the boot time.


Wonko

[MedEvil]

Or boot from drive A and continue at drive A too.

Read whole textmode part at one continuous read.
This is faster in comparison to several single reads.

Very interesting idea! On which media does it give an advantage? Or does all media benefit from it?

I mean, I guess that the scope is to try and combine together several different approaches in the hope of trimming down the boot time.

Yes it is. The question still is though, if this approach can be used with ramdisks too or just with a plain build for the second stage?

[Wonko the Sane]

Yes it is. The question still is though, if this approach can be used with ramdisks too or just with a plain build for the second stage?

No answer yet.
I am still convinced, as I am since years, that the ramdisk approach is "wrong" when compared with the "filedisk" one, if the scope is portability, for the known issue with low RAM machines.


Wonko

[MedEvil]

Imo, the filedisk approach, just like the plain build, have the advantage that the image does not need to be loaded, hence it's size has no bearing on the boot time. Which makes it great for big images and the only way, imo, for really huge images.

On the other hand, ramdisk builds offer a fantastic performance, once up and running, that can not be matched by any filedisk or plain build. And with the average amount of RAM, that can today be expected in computers in the wild, the ramdisk builds are no longer limited to be used on gamers computers.

From all my tests, the biggest problem for portable OS, are the USB boot speeds. They vary so widely that they can deliver a way faster boot, than from CD, as well as a way slower boot.
I have recorded boot times for the same ramdisk based build on the same stick, varying between 30 seconds and 20 minutes on different computers, while boot times from CD stayed pretty much the same with 2 minutes.

A filedisk or plain build, which boots from USB, though can still be faster than a similar CD based one, because it can make good in the gui phase of the boot, what it lost in the text based phase.

What's missing most, imo, is a way to tackle the USB boot speed issue, so that one can expect a more similar boot time on all computers, also for ramdisk based builds.

[Wonko the Sane]

What's missing most, imo, is a way to tackle the USB boot speed issue, so that one can expect a more similar boot time on all computers, also for ramdisk based builds.

The big "gap" is between USB 1.1 and USB 2.0.
If the hardware is USB 1.1 there is nothing to do.
If the hardware is USB 2.0 but BIOS boots at 1.1, the "kicker" (or PLoP) will make it much faster, while PLoP (when it works) will operate faster "since the start" the kicker will have the initial loading anyway driven by BIOS.
The point is - since the PLoP is not yet "universal" - whether adopting as "standard" the kicker will slow down a few seconds boot on USB 2.0 (when compared to "direct" booting) .
If using the Ramdisk, if I recall correctly the fastest was booting .is_ with the Server2003 SP1 ramdisk....:
http://www.911cd.net...showtopic=19737
but then you are againg hitting RAM usage limtis....


Wonko

[MedEvil]

I guess we can forget about hardware USB1.1 as USB2.0 was released 10 years ago.

While plop is a great idea, it hasn't worked once properly for me, on all the computers i tested it on.
It offers USB boot ability on computers which don't by default.
It offers to boot from addon cards.

It does not offer any decent speed!
USB 2.0 ports work with USB 1.1. speeds and USB 1.1 ports work with speeds, any floppy would be ashamed of!

Tested on older Intel, VIA and nForce boards.

Did just for laughs a test on a computer, which offers really great USB boot speed. Though not what the stick would be able to deliver.
Plop slowed also this computer down to a crawl.


Maybe a question for our g4d experts. Can G4D make use of BIOS modules?