20 replies to this topic

[jezza333]

G'day everyone

I have been browsing these forums for quite a while and reading over posts by members, which have helped me get a better enderstanding of VistaPE and other various 'booting methods'. You definitely seem to know a substantial amount about this topic , and I've wasted days and had a ton of frustration with inconsistencies and lack of information into what I've been researching. I'm hoping some of you chaps could help me out if at all possible.

Now to the juice...

I need to have the following functionality on a bootable USB thumb drive:

- USB 2.0 speeds
- Ability to execute command line utilities and commands [DOS shell]
- NTFS read/write
- Partitionable drive

It's just a project I've set out to do, and I need a little help. It may seem rediculous, but I use it to backup files off systems which have no boot capability [ruined FS, registry, BSOD's, corrupt MBR/bootector]. Transferring many GB's over USB 1.x takes a VERY LONG time.

After I am able to achieve this, I am also looking at customizing a VistaPE build, while using USB 2.0 during bootup. This will come later though.

USB 2.0 speeds

This is the main issue; everything else can be done, although a little slowly. I've used two drivers under DOS known as usbaspi.sys [detects USB media] and di1000dd.sys [maps media to drive letters, acoompanies first driver]

Info on these can be found here: http://www.computing...orum/13288.html

When these are loaded from RAM after a copy from the bootable USB, NTSDOS will not detect and NTS drives on the system. When I load these via a bootable floppy with the USB still connected, two out of three are detected. Seems a little weird, and always booting via floppy to install drivers is just not going to happen in my opinion; it's too impractical, very slow, prone to corruption and most new systems don't come with a floppy drive.

I found an alternative, which I extracted from the XPe SP2 FP 2007 setup, called ufdprep.exe. I tried it under VMWare and it booted without problems, although I'm yet to verify if it runs at USB 2.0 speeds. One hiccup though, it will boot if formatted as NTFS, but won't if formatted as FAT32 [even though I've been able to boot from a FAT32 USB using other bootsectors, and others have had success: http://www.911cd.net...l=petousb fat32]

Now, if I have a NTFS formatted drive, I either need to have a second partition as FAT16/FAT32 or somehow use DOS commands on a NTFS drive [which isn't possible, is it? :thumbsup:]. I tried adding a DOS floppy bootsector to NTLDR [which I copied onto the root of the partition I had formatted with ufdprep], following the steps in the following two pages as best as I could:

http://www.bcpl.net/...-dual-boot.html
http://www.tburke.ne...cking_guide.htm

The bootsector thinks the floppy drive is where it should be heading to, which in VMWare works, but in reality does not ["Disk I/O error..."]. None of these, either real of virtual, are doing what I want it to do. I want the bootsector to look for the DOS bootdisk files that I have copied onto the second partition on the USB [formatted as a FAT variant].

So I try to install DOS onto the USB's FAT partition [which I had set as active] and what do you know, it formats my OS hard drive [even though it was NTFS, which I thought it wouldn't recognise as a valid drive]. So, after all my work on this was stored on that HDD, I'm essentially back to square one after an OS reinstall. I really don't want to try to find out what I had been working with again and start re-fiddling.

Ability to execute command line utilities [DOS shell]

No worries here, I've been using HP's USB Disk Storage Format Tool to write a bootsector/MBR to the drive. After that I just copy over whatever files I want to use and they work fine. The drawback here is no USB 2.0 support, which I'm after.

NTFS read/write

I have Sysinternals' NTFSDOS Professional which works under DOS for the most part. The part where it doesn't work is when I load the USB 2.0 drivers via both methods described above. NTFS partitions are mounted and they are easily accessible if those USB drivers [usbaspi.sys and di1000dd.sys] aren't installed beforehand. The system locks up if I try to load those drivers after I have started NTFSDOS Professional.

Partitionable drive

I have a driver that does this fine, but I have to change drivers back to default consistently to be able to use some tools [drive is recognised as 'non removable' when the custom driver is installed]. Is there anything out there that will allow me to hotswap drivers or get the drive recognised as a removable disk, yet remain partitionable? [The first is the preferrred, as I'm sure Windows makes it completely impossible to partition and access removable drives

---------------------------------------------------------------------------------------------------------

So, after scanning all that mess ^, any ideas? :tabletalk:

If there's any more info you need, please, just say the word. I should be able to recall most of what I have used. My sincere thanks goes to people who have read this opst, and even moreso to those who are able to throw some ideas out there. :D


Oh, and while I think of it, what's the difference between a Master Boot Record and Boot Sector? Im under the *impression* that the MBR resides in the bootsector.... if so, what's the point of being able to isolate them?

EDIT: If you have read through the rest of the posts in this thread, you will see that I have solved the problem and created a tutorial on how to achieve the same results yourself. I have attached the tutorial to this post for easy access :D

Attached Files

•  howto_updated.zip   29.73KB   2523 downloads

[was_jaclaz]

Ok, i've read your post and completey failed to understand it.

ufdprep.exe is AFAIK NOT a DOS driver, rather a USB stick formatting utility:
http://ubcd4win.com/...?showtopic=6242

In order to boot DOS, the partition on the USB stick needs to be formatted as either FAT16 or FAT32.

The best DOS disk containing USB dos drivers can be found here:
http://johnson.tmfc....dos/usbdrv.html

Whether they support USB 2.0 speed, I cannot say.

You can experiment using grub4dos to load a floppy image containing those drivers and the Avira NTFS4DOS ones, INSTEAD of the Sysinternals one:
http://free-av.com/e...s_personal.html

Oh, and while I think of it, what's the difference between a Master Boot Record and Boot Sector? Im under the *impression* that the MBR resides in the bootsector.... if so, what's the point of being able to isolate them?

You won't get very far until you do not understand fully the difference between the MBR (Master Boot Record) and Boot Sector.

The MBR is the first sector of a (partitioned) device.
The boot sector is first sector of any partition (or volume) inside the partitioned device.

The MBR is generally speaking filesystem and operating system independent, it's business is to load the boot sector of the active partition.

The boot sector is filesystem AND operating system dependent.

Read these pages:
http://thestarman.pc.../mbr/index.html
http://www.ranish.com/part/primer.htm

The limit within 2K/XP is that a device seen as "removable" can be a partitioned one but ONLY one partition (the active one) wil be visible/accessible from within the 2K/XP OS.
You can use a filter driver, either the Hitachi Microdrive Filter cfadisk.sys or Anton Bassov's dummydisk.sys.
When booted with DOS or Linux, all partitions will be visible/accessible anyway.

jaclaz

[jezza333]

Ok, i've read your post and completey failed to understand it.

Haha, I thought it would be impossible to understand

ufdprep.exe is AFAIK NOT a DOS driver, rather a USB stick formatting utility:
http://ubcd4win.com/...?showtopic=6242

I was under the impression it wrote a USB 2.0 component to the disk somewhere, not creating DOS files. After a little re-reading, it appears that I was mistaken. It's just a bootsector write utility, which isn't much of a help when I look at it now because I already use the HP utility to write them when I boot to DOS. I think I am understanding things a little better than I was previously; in hindsight some of my previous ideas about methods of booting don't make sense.

Here's a PDF which addresses the subject, although I think I need to dig up XPe to follow it through: http://pheatt.empori...SB_Bootable.pdf

In order to boot DOS, the partition on the USB stick needs to be formatted as either FAT16 or FAT32.

The best DOS disk containing USB dos drivers can be found here:
http://johnson.tmfc....dos/usbdrv.html

Whether they support USB 2.0 speed, I cannot say.

Well, I'm aware that DOS cannot be loaded unless it's on a FAT16 or FAT32 partition, so we're on the same level there. I can virtually guarantee that I've tried all DOS based USB 2.0 drivers [USB 1.x is not needed as I'm already able to boot from the drive]

You can experiment using grub4dos to load a floppy image containing those drivers and the Avira NTFS4DOS ones, INSTEAD of the Sysinternals one:
http://free-av.com/e...s_personal.html

Yep, tried that one too. It can't find NTFS partitions either...

@Active NTFS reader [http://www.ntfs.com/products.htm] CAN read NTFS partitions when that USB 2.0 driver had been installed, but the interface is so extremely slow [it scans every folder opened, and takes the same length of time on each one regardless of contents] that it's not really a help in this case.

You won't get very far until you do not understand fully the difference between the MBR (Master Boot Record) and Boot Sector.

The MBR is the first sector of a (partitioned) device.
The boot sector is first sector of any partition (or volume) inside the partitioned device.

The MBR is generally speaking filesystem and operating system independent, it's business is to load the boot sector of the active partition.

The boot sector is filesystem AND operating system dependent.

Read these pages:
http://thestarman.pc.../mbr/index.html
http://www.ranish.com/part/primer.htm

Ah excellent, gotcha!

The limit within 2K/XP is that a device seen as "removable" can be a partitioned one but ONLY one partition (the active one) wil be visible/accessible from within the 2K/XP OS.
You can use a filter driver, either the Hitachi Microdrive Filter cfadisk.sys or Anton Bassov's dummydisk.sys.
When booted with DOS or Linux, all partitions will be visible/accessible anyway.

Yep, I'm already clued in there. I'll take a look at dummydisk and see if there's any difference [I've already used the Microdrive driver which I edited].


In summary of my above post, all I want to do is add USB 2.0 functionality and boot to DOS on a partition I have on the flash drive. I have read though quite a few posts on the 911 and Boot-land forums and don't seem to have any definitive answer, only confusing the matter.

Thanks for your fast reply mate, you come highly recommended and seem to know your business. Cheers


I think I'm off to take a deeper lok at that PDF.

[was_jaclaz]

Yep, tried that one too. It can't find NTFS partitions either...

Hmmmm....

The missing point is most probably a limit/problem in DI1000DD, similar to the one with FAT16/32 reported on the given link:
http://johnson.tmfc....dos/usbdrv.html

If you are going to use XP Embedded, you might as well have a shot at a "more affordable" PE based system, like any of the small sized projects created with winbuilder, like Native_ex based ones.

A smallish, command line only (CMD.EXE as shell) PE build should be in the 30÷45 Mbytes range.

If you like to experiment, you can have a try with the (still VERY experimental) XPcli:
http://www.boot-land...?showtopic=3717
there is not (yet ) "official" USB booting support, but it should be just a matter of adding USB drivers and the entries by Dietmar:
http://www.911cd.net...showtopic=21242

jaclaz

[jezza333]

Hmmmm....

The missing point is most probably a limit/problem in DI1000DD, similar to the one with FAT16/32 reported on the given link:
http://johnson.tmfc....dos/usbdrv.html

At first that's what I thought the problem could be until I eliminated it completely and the only variable was the usbaspi.sys driver > I copied all the needed files to a ramdisk, then called ctload to install the drivers [adding them in config.sys wasn't working because the USB became unmapped and it couldn't load the rest of the files] > so no command interpreter or DI1000DD.sys, so the drive couldn't be re-mapped. I loaded usbaspi.sys when on the ramdisk, then immediately after ran NTFSDOS Professional and also tried NTFS4DOS. Same problem, so that eliminates the drive letter mapping driver as the cause.

I strongly suspect the driver is somehow writing something to memory which affects the other storage drives on the system, not just the USB flash drive. There are a number of switches that usbaspi accept, one supposedly being the /L switch. This is representative of LUN, which I read to be the number of devices the driver should install the diver for. I tried adding a number after the switch, according to the documentation found on the net, but it doesn't accept it. This is the best resource I have found on it: http://jedlik.vein.h...susb/readme.txt

/l# Luns, to specify highest number of LUN assigned, default /L0

Weird... any ideas?

If you are going to use XP Embedded, you might as well have a shot at a "more affordable" PE based system, like any of the small sized projects created with winbuilder, like Native_ex based ones.

A smallish, command line only (CMD.EXE as shell) PE build should be in the 30÷45 Mbytes range.

If you like to experiment, you can have a try with the (still VERY experimental) XPcli:
http://www.boot-land...?showtopic=3717
there is not (yet ) "official" USB booting support, but it should be just a matter of adding USB drivers and the entries by Dietmar:
http://www.911cd.net...showtopic=21242

Ah nice, I might take a look. Thanks

EDIT: The second link seems to be broken, but the first definitely looks interesting. Cheers

EDIT 2: Well, I took a look and tried 3 methods of building one. The first attemt was the batch file [911 thread here] which fetched the necessary files from the setup CD and copied the other files from the system. I tried to boot it up, but it was missing biosinfo.inf [/windows/inf folder wasn't there at all]. I found a few on my system [all the same size, some were in the Winbuilder directory] and it just gave me a bluescreen and got into a reboot loop.

The second one got me nowhere, built from the winbuilder script posted at the end of the main thread: [Boot-land post]

The third was your post [http://www.boot-land...ndpost&p=27822], but I'm still not able to get a working build running. I've copied the files onto my flash drive and started VMWare using a physical disk [as I have done in the past, so I know it works :D]. I first got a DLL not found error, and subsequently I have blue screens. No errors were found in the log created by your command/batch file, so I have no clue what I'm doing wrong...

I apologise for my lack of understanding with these matters, I guess I just haven't been shown before .

[was_jaclaz]

I loaded usbaspi.sys when on the ramdisk, then immediately after ran NTFSDOS Professional and also tried NTFS4DOS. Same problem, so that eliminates the drive letter mapping driver as the cause.

No, sorry, but that proves nothing, USBASPI is a USB controller driver, you need to give it a USB DEVICE driver, DI1000DD.sys, USBCD.sys, njusbcda.sys, NJ32DISK.SYS or whatever, otherwise NO drive can be found.

I'll rephrase:
1) Instead of Xp embedded, try with any of the Native_ex based projects, they are all tested and working.
2) If you want to experiment, have a look at this UNTESTED on USB idea, XPcli
3) If you are going down path #2, you will need to add to it USB support, by ADDING yourself the USB drivers AND trying the changes on USB drivers booting order section, detailed on Dietmar's Tutorials, which you can find in his signature

In other words:
1) Forget using DOS for USB 2.0 speed
2) Forget about XP embedded
3) Forget about the experimental builds I linked you to
4) Try using Winbuilder and the Native_ex project(s) to build a minimal USB booting PE:
http://www.boot-land...s/?showforum=52
http://www.boot-land...?showtopic=2376
once, and only once, you have succeeded building the latter, you may want to try the smallest possible build:
http://www.boot-land...?showtopic=3910
customizing it to your needs.

jaclaz

[jezza333]

No, sorry, but that proves nothing, USBASPI is a USB controller driver, you need to give it a USB DEVICE driver, DI1000DD.sys, USBCD.sys, njusbcda.sys, NJ32DISK.SYS or whatever, otherwise NO drive can be found.

Ah, how would a USB device driver map a HDD formatted with NTFS? The device driver in this case only maps FAT and FAT32 partitions...

I have tested a USB 1.x driver, just to see if they work at all, and they did. I ran two tests:

1) I ran NTFSPRO just after the controller driver was installed, and before I installed the device driver. It picked up NTFS drives fine
2) I installed the controller driver and device driver, and then ran NTSPRO. It detected them fine also.

I'm 101% convinced it's a problem with that controller driver. There are a few others, but they're only USB 1.x; the Panasonic driver [usbaspi - USB 2.0] is reported to be the best and support the broadest range of drives, not to mention support the fastest speeds, so that's what my tests have been using

I'll rephrase:
1) Instead of Xp embedded, try with any of the Native_ex based projects, they are all tested and working.

Yep, I've been looking into Native_ex for a few hours. Cheers

2) If you want to experiment, have a look at this UNTESTED on USB idea, XPcli

Well, you tested it and had success, along with almost every other member who posted in that thread. I wouldn't exactly call that 'untested', but I still failed horribly at an attempt anyway, which proves your point

3) If you are going down path #2, you will need to add to it USB support, by ADDING yourself the USB drivers AND trying the changes on USB drivers booting order section, detailed on Dietmar's Tutorials, which you can find in his signature

Sure thing, I didn't notice them. Thanks for the pointer

In other words:
1) Forget using DOS for USB 2.0 speed
2) Forget about XP embedded

I guess I knew that was coming, not a worry though; as you have shown me some very valid alternatives that should do the trick just fine

3) Forget about the experimental builds I linked you to

OK, thanks for linking them in the first place though...

4) Try using Winbuilder and the Native_ex project(s) to build a minimal USB booting PE:
http://www.boot-land...s/?showforum=52
http://www.boot-land...?showtopic=2376
once, and only once, you have succeeded building the latter, you may want to try the smallest possible build:
http://www.boot-land...?showtopic=3910
customizing it to your needs.

Gotcha. I'll fiddle around and see what I can come up with. Thanks for your replies mate, much appreciated !

[powaking]

USB 2.0 speeds at bootup is not really dependent on the actual USB device but the speed the BIOS gives the port at bootup.
Case in point I can bootup VistaPE on my Cruzer Micro USB thumb drive on an HP NC6000 and boots up as fast if not SLOWER as booting from a CD.
On a Dell D620 machine I get to the desktop in about 26 seconds. In this area your mileage will vary for sure.

[jezza333]

USB 2.0 speeds at bootup is not really dependent on the actual USB device but the speed the BIOS gives the port at bootup.

Well yeah, in my case the BIOS selects USB 1.x, when the system is definitely capable of 2.0...


@jaclaz:EDIT: I built NativeEx_barebone and have a working ISO, 20MB, all I need are a few drivers to remove the BSODs using USB boot... Is there a way to access the HDDs on the system? The ramdrive is the only accessible drive...

I know this question is OT, but how on earth do I install EWF? Searches turn up relatively little, and Dietmar mentions it as being important to preserve the life of flash media... From the looks of it I need some XPe files, correct? I don't want to kill my drives if I can help it

EDIT 2: Well, that's it I guess. Dietmar's posts are packed with information, but I have NO CLUE as to how it all ties in. No offense to him, but his English grammatical structure is impossible for me to make sense of. A ton of info, just no idea how to implement it.

It just seems like the original idea of using a single driver, which doesn't happen to function as expected, will cause me to go extravagant measures just to get USB 2.0 speeds. I could spend countless days trying to make head and tail of everything, like I already have; unless it's just an easy step then for the moment I'm just going to drop it for a few days. I don't have the motivation presently to try to extract 100's of pieces of information from everywhere it could possibly be spread to.... Sorry.

Thanks for the help mate.

[was_jaclaz]

For the record:
EWF is NEEDED if you format the stick as NTFS (due to the semi-journaling features of this filesystem), if you have FAT16 or FAT32 you have NOTHING to worry if using a PE based build, that won't right anything to it.

I am not expert on Native_ex, but surely there is already a way to put it on a USB stick, a tested and working one, I mean.

Do post a request for help in the sub-forum:
http://www.boot-land...s/?showforum=52
Peter (psc) or someone else will definitely help you.

jaclaz

[jezza333]

Well, things are looking fairly sweet at the moment. Last night I managed to find a different version of Iomega's aspiehci.sys, which, when installed after ntfspro is launched, both still work correctly. I am then able to map drives using di1000dd.sys without issues. [if I try to load ntfsdos AFTER the driver is installed, I get the same problem as before. Panasonic's usbaspi.sys [universal, supposedly the best] locks up/reboots my machine if I try this method]

So I have achieved what my initial objective was, and that's a very cool thing in itself, especially when things didn't want to work together [and using DOS after all :D]. The two things now working under DOS, together are:

- USB 2.0 speeds
- NTFS read/write

I'm still very interested in getting into building PEs, but a little busy at the moment. I'll take a look and call on advice if [and when :thumbsup:] I need it. Cheers mate, you have been a massive help !

[was_jaclaz]

Care to share some info in this "particular" version of Iomega drivers?

Like version, where did you find it and so on?

Can you also confirm that aspiehci.sys is USB 2.0?

jaclaz

[jezza333]

Care to share some info in this "particular" version of Iomega drivers?

Like version, where did you find it and so on?

Well I grabbed it off a torrent with a ton of DOS boot materials in it [all legitimate though, no illegal business], and started testing all 8 or so drivers that were included in it. Most were USB 1.x, so I excluded them. The only one that works with NTFS software/drivers under DOS is aspiehci.sys [only from my testing]. For those interested, I don't know if it's the latest version, but it works and the other version I tested earlier took a long time to detect the drive and I couldn't map the FAT partitions on my USB drive [errors with the subsequent driver]

Here's its info: Iomega ASPI USB-EHCI 1.0 V.13 09 May 2003
Switches I used: /all /int

the /all switch is used to detect all USB drives possible, not just Iomega drives. Not sure what the /int switch is for, but it's used in the official config.sys in Iomega's bootdisk, so I've used it. The second driver [ASPI Storage block device driver/mapping drives to letters] is made by Motto Hairu [most peeps have never heard of them]. I can attach/upload them if you'd like me to

Can you also confirm that aspiehci.sys is USB 2.0?

Absolutely, the results between standard initialisation and using that driver are immense. I transferred 30MB in about 5-7 seconds from a NTFS partition as opposed to 4MB in about 25 seconds. Massive difference, and that's under DOS and using NTFS [which isn't native] with the default cache [which can be adjusted]

There are three standards compliant controllers in PC's. In Windows, Go to ControlPanel - System and (look under Hardware or DeviceManager to) examine the listed devices under Universal Serial Bus Controllers to determine the type of USB ports detected:

USB 1.x UHCI (Universal Host Controller Interface) Intel, VIA
USB 1.x OHCI (Open Host Controller Interface) Compaq, Microsoft and National Semiconductor
USB 2.x EHCI (Enhanced Host Controller Interface)

Source: http://jedlik.vein.h...susb/readme.txt

[was_jaclaz]

Hmmmm, the 0.13 version is the one that ships (shipped) with SYMANTEC GHOST 2003, cannot say if it's available freely elsewhere.

It seems to be however part of this "multi-device" DOS CD:
http://vivil.free.fr/usbdosboot/
(Warning 56k users, the .iso is 42 Mb)


Good to know that it gives USB 2.0 speed.

jaclaz

[jezza333]

Hmmmm, the 0.13 version is the one that ships (shipped) with SYMANTEC GHOST 2003, cannot say if it's available freely elsewhere.

Why wouldn't it be? It's also available in the bootdisk.com USB boot collection according to their website; not to mention a ton of other sites on the internet - just varying versions

Good to know that it gives USB 2.0 speed.

Yep, excellent stuff. USBASPI.SYS also provides USB 2.0 speeds and is supposedly compatible with a larger number of drives, but isn't compatible with any of the DOS based NTFS drivers I've tried.

[Brito]

Would you make available this image?

These features sound awesome, I really like DOS and can't conform that people call it "old"..

Great work!

[jezza333]

Would you make available this image?

Would you like me to upload the drivers I have used and the order of execution? A ramdisk needs to be loaded and the drivers copied to the ramdisk [how can they be installed when the drive/files being referenced temporarily doesn't exist], then installed using ctload. I can upload or supply the URL of all the files needed if you wish

Not a problem either way.

[Brito]

I can upload or supply the URL of all the files needed if you wish

This would be great along with a small tutorial!

[jezza333]

Sure thing, I'll see what I can do.

EDIT: sorry, a little late, but here it is. I've made it easy enough to understand, even for the not-so-proficient PC user. Any 'baby talk' isn't meant to be offensive; I know you know a ton about this sort of field.

Let me know if I'm missing anything
Cheers.

EDIT: Updated the tutorial slightly... you can find it on the first post in this thread:
http://www.boot-land...?...ost&p=31969

[Brito]

Cool instructions, really liked the intro:

Gday there reader

Here are the [de/in]structions you need to create a bootable USB thumb drive..

(only found one typo: archiover)

Would you edit your first post on this topic to ease other people to find this good guide?

It was a very easy to follow tutorial.

Very good work!

[jezza333]

(only found one typo: archiover)

FiXeD

It was a very easy to follow tutorial

Ah good stuff, I hoped it was but was unsure

Thanks for the feedback mate, appreciated.