46 replies to this topic

[ktp]

@jaclaz

Thank you for your rectification, I did understand as you said but my writing did not reflect this (grub4dos bypassing the bootsector).
Now normally the good news is that no need for real stick, I can recreate the problem with tools like:
- fsz, vdk, qemu (with the associated 2 GB .pln file done few months ago with your help to simulate a 2GB key).

I will post precise details of the operations as you said.
Roughly the operations will be the following (chronological order):

fsz i:\outputiso\ldlc2gb.img 2097152000

ldlc2gb.pln file:
DRIVETYPE ide
CYLINDERS 254
HEADS 255
SECTORS 63
ACCESS "i:\outputiso\ldlc2gb.img" 0 4095937

vdk open 0 ldlc2gb.pln /rw
Use ptedit32 to set the MBR
(initially all 4 partition entries are 0)
partition 0B-80-0-1-1-254-245-55-63-4095937
vdk close 0
vdk open 0 ldlc2gb.pln /rw
Under XP: do format f: /fs:fat32
Save MBR of the disk with hdhacker
Save boot sector of the disk with hdhacker

Copy ntldr, ntdetect.com, boot.ini to f:\
vdk close 0

Run QEMU:
qemu -M pc -hda i:\outputiso\ldlc2gb.img -m 256 -localtime -net nic -L .
Check bootability problem.

[ktp]

@jaclaz and all
Following are the required data. Hopefully you can find out why it does not boot.

1)
fsz i:\outputiso\ldlc2gb.img 2097152000


2)
vdk open 0 ldlc2gb.pln /rw

with ldlc2gb.pln file:
DRIVETYPE ide
CYLINDERS 254
HEADS 255
SECTORS 63
ACCESS "i:\outputiso\ldlc2gb.img" 0 4095937


Virtual Disk Driver for Windows version 3.1
http://chitchat.at.i...k.co.jp/vmware/

Virtual Disk 0
Access Type : Writable
Disk Capacity : 4096000 sectors (2000 MB)
Geometry : © 254 * (H) 255 * (S) 63
Number Of Files : 1

Type Size Path
------- ------- ----
FLAT 4096000 i:\outputiso\ldlc2gb.img

Partitions :
# Start Sector Length in sectors Type
-- ------------ --------------------- ----
0 0 4096000 ( 2000 MB) <disk>

3)
Use ptedit32 to set the MBR
(initially all 4 partition entries are 0)
partition 0B-80-0-1-1-254-245-55-63-4095937
4)
vdk close 0
5)
vdk open 0 ldlc2gb.pln /rw
Virtual Disk Driver for Windows version 3.1
http://chitchat.at.i...k.co.jp/vmware/

Virtual Disk 0
Access Type : Writable
Disk Capacity : 4096000 sectors (2000 MB)
Geometry : © 254 * (H) 255 * (S) 63
Number Of Files : 1

Type Size Path
------- ------- ----
FLAT 4096000 i:\outputiso\ldlc2gb.img

Partitions :
# Start Sector Length in sectors Type
-- ------------ --------------------- ----
0 0 4096000 ( 2000 MB) <disk>
F: 1 63 4095937 ( 1999 MB) 0bh:FAT32

6)
Under XP: do format f: /fs:fat32

format f: /fs:fat32
Le type du système de fichiers est RAW.
Le nouveau type de système de fichiers est FAT32.

Attention : toutes les données sur le lecteur de disque
non amovible F: seront perdues !
Continuer le formatage (O/N) ? o
Vérification de 1999 Mo
Initialisation de table d'allocation des fichiers (FAT) en cours...
Nom du volume (11 caractères, Entrée pour ne rien mettre) ? ldlc2gb
Formatage terminé.

2 093 010 944 octets d'espace disque au total.
2 093 006 848 octets disponibles sur le disque.

4 096 octets dans chaque unité d'allocation.
510 988 unités d'allocation disponibles sur le disque.

32 bits dans chaque entrée FAT.

Le numéro de série du volume est 38B6-8DA9


7)
Save MBR of the disk with hdhacker
Save boot sector of the disk with hdhacker

zip of the mbr, boot sector and qemu screen capture:
http://www.mediafire.com/?9mnzxgzdmmb

8)
Copy ntldr, ntdetect.com, boot.ini to f:\

dir f:\
Le volume dans le lecteur F s'appelle LDLC2GB
Le numéro de série du volume est 38B6-8DA9

Répertoire de f:\

13/03/2007 20:23 1 399 boot.ini
04/02/2007 22:57 47 596 ntdetect.com
05/08/2004 05:00 251 712 ntldr
3 fichier(s) 300 707 octets
0 Rép(s) 2 092 699 648 octets libres

9)
vdk close 0

10)
Run QEMU:
qemu -M pc -hda i:\outputiso\ldlc2gb.img -m 256 -localtime -net nic -L .
Check bootability problem. Result: just hang (blinkking cursor, refer to capture image in the zip above)!

[ktp]

I found an error: no MBR code. So I need to do a MBRFix first then post result.

[ktp]

11)
vdk open 0 ldlc2gb.pln /rw
mbrfix /drive 1 fixmbr
vdk close 0
Run QEMU
qemu -M pc -hda i:\outputiso\ldlc2gb.img -m 256 -localtime -net nic -L .
Got: Error disk, press a key to reboot

MBR/Bootsector and screen capture:
http://www.mediafire.com/?fmamq2tvoc2


So the problem is well with the boot sector created with XP format.

[was_jaclaz]

So the problem is well with the boot sector created with XP format.

Yes, OR in the way QEMU reads the geometry of the drive.

As soon as I have time I'll check the procedure.

BTW, whilst VDK for this kind of operations is easily managed from command line, QEMU is much more easy to use if you "adopt" QEMU manager:
http://www.davereyn.co.uk/

Also, not to use much disk space, if you have a NTFS partition, it is a great commodity to create disk images as "sparse", using Bo Branten's mksparse:
http://www.acc.umu.se/~bosse/
http://www.acc.umu.s...se/mksparse.zip

jaclaz

[ktp]

Yes, OR in the way QEMU reads the geometry of the drive.

Yes, but the problem appeared also on real hardware, real USB key, real USB HDD.
So the simulation here with vdk/qemu is only for convenience of the experimentation.

Also I continue few more steps.

12)
vdk open 0 ldlc2gb.pln /rw
bootsect /nt60 f:
Run QEMU
qemu -M pc -hda i:\outputiso\ldlc2gb.img -m 256 -localtime -net nic -L .
Got: Bootmgr is missing, press any key to restart
=> This is excellent, since I do not copy bootmgr yet to the root of the disk.

13)
vdk open 0 ldlc2gb.pln /rw
copy bootmgr f:\
copy \boot f:\
Run QEMU
qemu -M pc -hda i:\outputiso\ldlc2gb.img -m 256 -localtime -net nic -L .
OK, since does not copy \sources and hence no boot.wim, but the system boots ok until error message
due to missing files (which is normal).


So for me the boot sector for Vista (from bootsect /nt60) is OK,
but not the one provided by my built-in XP format (or HP-USB format, FAT32).

[ktp]

Note: if at step 6) I format the disk with FAT (=FAT16) format (using XP format)),
then the disk boots OK. So for me there is a problem with the FAT32 boot sector
in XP format (at least with my localized version). This is also coherent that all my HDD or keys
boots ok if I used PetoUSB to format (since FAT16/FAT16X were used by PEtoUSB formatter).

[ktp]

Other confirmation of FAT32 boot sector. I just try mbldr-1.43 (Master bootloader) as MBR (it contains just one first sector).
Same error message: Disk error, press any key to restart, althout ntldr/ntdetect.com/boot.ini are there (the only change was
the MBR). So grub4dos MBR has really magic code.

[was_jaclaz]

I really don't know what to reply.

Apart:
1) having a wrong size for the image in the .pln file
2) having a non-standard address for End Sectors CHS, with fractional end heads and sectors and one cylinder too much
3) Qemu reading the image (as hinted before) with a different geometry of nx64x63 (and probably this happens on some hardware too)
4) Using 0B kind of partition (CHS mapped) instead of 0C (LBA mapped)
5) Did I forgot anything?

The image you made is fully "standard" and should boot everywhere.

By the way it boots on VMWare...

Try this geometry:
0C-80-0-1-1-1014-63-63-63-4092417
and this .pln file:

DRIVETYPE ide

CYLINDERS 1015

HEADS 64

SECTORS 63

ACCESS "ldlc2gb.img" 0 4096000

and see if it boots on Qemu.

jaclaz

[ktp]

@jaclaz

Congratulations! I follow the new pln settings and geometry and now it boots perfectly in FAT32 (0x0b) under QEMU.
Note that when I use XP to format it in FAT32, the 0x0c set by ptedit32 first is reset back to 0x0b in the MBR.
But I think that the new geometry is the real trick.
How do you change from 254/255/63 to 1015/64/63 ?

Note: on real hardware probably I cannot change as is the geometry.

By the way it boots on VMWare...

How do you test/boot under VMware? You mount directly the ldlc2gb.img file? If yes, how?

[was_jaclaz]

It seems like the FAT32 bootsector is more "sensible" to correct values than the FAT16 one.

Basically it seems like it checks, the CHS structure even if the partition type is LBA.

Qemu's "bios", which actually should be the "Bochs" one, appears to be VERY accurate in following "standards", it tends to determine geometry of the RAW images based on their size.

So, what we need is to "force" it to the nx255x63 geometry.

Use mksparse to make an image any size bigger than the CHS barrier, say roughly 8 Gb:

mksparse 8gb.img  8233281k

Create a .pln file for it:

DRIVETYPE ide

CYLINDERS 1024

HEADS 255

SECTORS 63

ACCESS "8gb.img" 0 186466562

Write to it a MBR and this geometry:
0C-80-0-1-1-253-254-63-63-4080447

Format it, and try it in Qemu.

Experiment with different CHS values and partition types (0B and 0C) until you get back to your previous geometry:

0B-80-0-1-1-254-245-55-63-4095937

You should be able to find WHICH setting is the one that creates the non-bootability.

The alternative is to experiment with the Qemu settings:
http://fabrice.bella...-doc.html#SEC10

`-hdachs c,h,s,[,t]'
Force hard disk 0 physical geometry (1 <= c <= 16383, 1 <= h <= 16, 1 <= s <= 63) and optionally force the BIOS translation mode (t=none, lba or auto). Usually QEMU can guess all thoses parameters. This option is useful for old MS-DOS disk images.

(I never tested it, so I cannot say how and if it will work for this)

How do you change from 254/255/63 to 1015/64/63 ?

I boot Qemu from a DOS floppy with good ol' Ranish Partition Manager and check how the HD image is seen in it.

How do you test/boot under VMware? You mount directly the ldlc2gb.img file? If yes, how?

I have a very old version of VMWare, 3.x.something, that can use .pln files directly (please note that the path to the image in .pln file NEEDS to be a full path, unlike VDK which will work with just the file name, as long as .img and .pln are in the same directory). Cannot say if newer versions maintained this feature.

Note: on real hardware probably I cannot change as is the geometry.

Yep, that is the problem.
Probably the stick is read with different geometries by different BIOSes, I am afraid that, apart the grub4dos workaround, there is nothing you can do about it.

To resume, everything appears to be related to the various BIOSes involved, and, if I am allowed to use some stereotypes :
the FAT32 bootsector code is German or Swiss, very exact and accurate, may fail is some data is "non-standard"
the FAT16 bootsector code is Russian, simple, effective, it works almost always, with almost any data you "feed" it
the grub4dos MBR chainloading of the loader by-passing the bootsector is the Latin (or I should say Chinese) approach: use some fantasy to workaround the problem, who cares what data you feed it, the NTLDR will re-scan everything anyway


jaclaz

[ktp]

@jaclaz

I am afraid that, apart the grub4dos workaround, there is nothing you can do about it.

I agree.

To resume, everything appears to be related to the various BIOSes involved, and, if I am allowed to use some stereotypes :
the FAT32 bootsector code is German or Swiss, very exact and accurate, may fail is some data is "non-standard"
the FAT16 bootsector code is Russian, simple, effective, it works almost always, with almost any data you "feed" it
the grub4dos MBR chainloading of the loader by-passing the bootsector is the Latin (or I should say Chinese) approach: use some fantasy to workaround the problem, who cares what data you feed it, the NTLDR will re-scan everything anyway

I would add:
The explanations are Italian, very volubile and informative, with lot of references (links)!

[ktp]

Maybe some of you already know this, but I just learn it today.

Previously to test my USB keys boot, I need to use real hardware, which is very inconvenient (need second laptop, change BIOS order etc..., unplug/plug on changes/modification cycle, reboot...).

Then apart from real USB key bootability (BIOS), you can test all the other (OS, boot, applications, grub4dos/syslinux menu/cfg) by using simulated disk using fsz/vdk/qemu.

You can also test with real USB key by using VMware, adding new hard disk (USB key must be pluggged before VMWare start so it can see the corresponding USB physical drive, the disk will be seen as SCSI disk). Then start the virtual machine, press F2 to change hard disk boot order. With this method be careful though if concurrent access to the USB key from both vmware and the host. It is better to stop the virtual machine first before any modification on the USB key on the host (cache). With vdk/qemu there is no such problem since the disk cannot be mounted simultaneously by vdk and qemu (exclusive use).

So with either method (probably second method is better) you need much less access to real hardware for test, with excellent results.

[MedEvil]

This is a great idea. I'm also suffering from not being able to test USB-Boot.
Does qEmu support booting an OS from a real USB-Stick?

[ktp]

Note: if at step 6) I format the disk with FAT (=FAT16) format (using XP format)),
then the disk boots OK. So for me there is a problem with the FAT32 boot sector
in XP format (at least with my localized version). This is also coherent that all my HDD or keys
boots ok if I used PetoUSB to format (since FAT16/FAT16X were used by PEtoUSB formatter).

For cross-reference sake:
The FAT32 boot code problem (Disk error message) has been explained in various topics. The bypass ("CHS knock-out") is to patch the FAT32 boot sector to fix buggy code:
http://www.boot-land...?showtopic=8528

[maanu]

This is a great idea. I'm also suffering from not being able to test USB-Boot.
Does qEmu support booting an OS from a real USB-Stick?

mobalive usb can be used for this purpose .

[wimb]

The most important patch for the FAT32 Bootsector to make USB-stick bootable is:
- drive-id patch 0x80 at offset 0x40 , after XP format it is 00 and Not bootable

Disconnect Or touchdrive.exe are needed to preserve BootSector 0x80 byte at offset 0x40 for FAT32 and offset 0x24 for FAT FileSystem
Otherwise Windows OS will make the drive-id byte 00 again so that USB-stick is Not Bootable

More Info on BootSector patches in Format Stick Section of Tutorial
http://www.boot-land...?showtopic=5306

[was_jaclaz]

The most important patch for the FAT32 Bootsector to make USB-stick bootable is:
- drive-id patch 0x80 at offset 0x40 , after XP format it is 00 and Not bootable

Since this is the second or third time you post this, could you please describe under which conditions the byte at offset 0x40 was NOT made "80"?

I mean, as is, you are (all by yourself) claiming that the byte at offset 0x40 does not get the right value of 80, but you fail to describe when/what/which.

If I make an image with MBRBATCH/MKIMG the byte at offset 0x40 is set by the 2K/XP FORMAT command to 80 allright.

jaclaz

[wimb]

When the USB-stick is Removable disk then XP Format from My Computer will give 00 at offset 0x40 in FAT32 BootSector.

Moreover, the FAT32 BootSectors given by ktp for his mp3-player all have 00 at offset 0x40
http://www.boot-land...?...=8528&st=10

[ktp]

For information, I just retry with both boot sectors (from grub4dos and XP format) and byte offset 0x40 set to 0x80. Same problem: "disk error" message.

[was_jaclaz]

When the USB-stick is Removable disk then XP Format from My Computer will give 00 at offset 0x40 in FAT32 BootSector.

Moreover, the FAT32 BootSectors given by ktp for his mp3-player all have 00 at offset 0x40
http://www.boot-land...?...=8528&st=10

I see.

So the problem is ONLY when the device is Removable.

About the ktp's MP3 thingy, the bootability of it was the least of the problems, the main point, left UNSOLVED due to ktp giving up , was to create a "double effect" MBR/bootsector, in such a way that the "stupid" self-check would not re-self-format the thingy.

Since the test bootsectors (not the original ones from ktp) were made through MBRBATCH/MKIMG, they should have the 80 allright, next step, if and when ktp will be willing to resume the game will be that of correcting a number of other parameters, as hinted in my last post there.



jaclaz

[wimb]

@ktp
Well your problem with the mp3-player is more complicated then,
but anyway the 0x80 at offset 0x40 is required in FAT32 BootSector for bootability of USB-stick.

May be jaclaz has more ideas to solve your problem.

@jaclaz
Yes, this is an interesting difference in FAT32 Format of Fixed and Removable USB-stick.