29 replies to this topic

[Ninho]

For accessing USB mass storage devices, does Grub4DOS have
its own USB software "stack", or does it rely on BIOS support (if any) ?

I guess it uses own stack entirely - and I hope it's the case.

If so, I would suggest a much needed IMO fix/enhancement :
at the moment, Grub4DOS appears not to be able to deal with USB disks that
present 4 K-byte sectors to the host. Could such support be programmed in,
up to and including "BIOS int 13" extension to allow bootstrapping from such
disks' MBR and their legacy OS PBRs ?

[Added:] If OTOH G4D does not include an USB stack at all /yet/, please consider
doing it, including large sectors from the inception !

Are there any alternatives ? (don't say "plop", that does NOT have 4k sector support, but non-working 'experimental')

Edited by Ninho, 27 December 2014 - 03:09 PM.

[Wonko the Sane]

There is a line drawn between the 0.4.5 "series" (that are "evolving but "stable" or "good enough for production use") and the 0.4.6 series (that is the more experimental versions), the latter does have a "own" USB stack, whilst the former has it not.

 

BUT, at least in theory, there is no *need* for specific 4 Kb sector support for "normal" booting, a couple (or more) "tricks" may be needed in the partitioning/formatting but it greatly depends on the specific device that is connected via USB.

 

If you post a "real life" example, including specific hardware involved, we may help you finding a specific solution.

 

Wonko

[Ninho]

There is a line drawn between the 0.4.5 "series" (that are "evolving but "stable" or "good enough for production use") and the 0.4.6 series (that is the more experimental versions), the latter does have a "own" USB stack, whilst the former has it not.

Great ! I'll have to look at the newer 0.4.6 then. Where is the current download hosted, please ?

BUT, at least in theory, there is no *need* for specific 4 Kb sector support for "normal" booting, a couple (or more) "tricks" may be needed in the partitioning/formatting but it greatly depends on the specific device that is connected via USB.

I'm referring to fairly standard USB hard disks 'enclosures'. The one I'm using presents 4K byte sectors to the host, as is the standard - or used to be when I bought it, in part driven by MS for better compatibility to Windows XP. Such disks are not bootable by most (any?) BIOSes, nor usable in DOS nor any version of Windows prior to Win XP-SP2 (or was it SP1?)

This being so, i'm surprised you think there is "no need" for 4K sector support
for booting, at least for booting-from-USB ! Ideally it should be provided by a BIOS (update), however it seems the industry never got it.

If you post a "real life" example, including specific hardware involved, we may help you finding a specific solution.

I don't have the HW here nor its exact references, it's an IOMEGA brand storage system, the actual disk inside being 1 TByte Hitachi and the USB "bridge" component vendor was Oxbridge IIRC. Safe for getting the bridge's programming guide and/or special utilities, which nobody will provide to a consumer, there does not seem like there's a way to reprogram the bridge itself even though it certainly /is/ possible, in theory.

I'm incredibly anxious to hearing what solutions you have in mind for this problem.

Edited by Ninho, 27 December 2014 - 04:11 PM.

[pscEx]

I'm incredibly anxious to hearing what solutions you have in mind for this problem.

Please have in mind, that there are never problems here, there are only issues!

 

Peter

[Wonko the Sane]

Great ! I'll have to look at the newer 0.4.6 then. Where is the current download hosted, please ?
 

http://reboot.pro/to...b4dos/?p=186085

 

In theory the idea is that the BIOS reads first sector of a device (no matter which size is the sector, 512 bytes or greater) then checks if at offset 510 there are the "Magic Number" 55AA bytes, and if they are there, execute the code starting at offset 0. 

Of course each and every BIOS vendor may have followed the above or have introduced any kind of stupid check/limitation/whatever that makes USB booting this way impossible.

Then there is the issue about the nature of the media (partitioned or non-partitioned).

Again in theory the idea is that (in the case of non-partitioned media) there is a field in the BPB (BIOS Parameter Block) stating the size of the sector (and again a given OS may use or totally ignore that field or simply be incapable of using anything but the "default till recently" 512 bytes size for the sector.

In the case of partitioned media, the data in the MBR (which provides no hints about sector size as both CHS and LBA are expressed in "sectors") may be read by the BIOS (or by the OS or by both or by one but not the other) as "sectors sized 512 bytes" or "sectors sized the size that the device is reporting".

Most (but not all) hard disks with 4K sectors are not "pure" 4K sectors as they simulate anyway a 512 bytes geometry, i.e. while being "Advanced Format" (which should be really called "Retarded Sectored" ) are actually so-called 512e:

http://en.wikipedia....Advanced_Format

It is extremely rare (though  possible) that a 1 Tb hard disk (which has no real reason to use 4096 Kb sectors as the limit of 512 bytes sectored media addressable space on MBR disks is around 2.2 Tb) is "pure" 4K, generally only models bigger then 2 Tb (and intended/produced for the stupid EFI/UEFI platform, i.e. intended to be GPT disks) are not 512e.

On the other hand it is entirely possible that the actual disk drive is 512e BUT the "bridge" inside the case "emulates" 4K sectors only.

 

So, if the limit is in the BIOS (that is simply hardcoded to read from the device a single sector, sector 0, sized 512 bytes and gets instead a 4096 one and panics ) you are perfectly right  and a "third party" USB stack, 4K compatible is needed (and of course it needs to reside on an internal hard disk or on a "helper" media), but sometimes this is also not enough as the OS itself cannot deal properly with these 4K sectors (at least in the early phases of booting), see -as an example - the mess that the good MS guys put together:

http://msdn.microsof...5(v=vs.85).aspx

 

But if the issue is only in the boot code (of the MBR) it may be possible to workaround the issue (still provided that the OS actually can use for booting a 4K sectored device, conversely, if the latter is not true, it will be really tough (though maybe still possible) to actually boot.

 

As you can see there are a lot of variables involved, surely there is not a "one-size-fits-all" solution, maybe there are specific workarounds, that will need EXACT data/specific hardware and OS involved info and a lot of testing.

 

Wonko

[Ninho]

Please have in mind, that there are never problems here, there are only issues!

Yup, my bad - English is not my native language !

As for why Wonko asserted that this is not an issue,
I can only guess it's because he may think that consumer
disks have "advanced format with emulation of 512k-sectors"(AF/e), thus
allowing them to be seen and booted from legacy BIOSes and OSes.
But it is not the case with USB-connected tera-byte sized drives, these do NOT 'emulate' legacy sectors :-(

[Ninho]

So, if the limit is in the BIOS (that is simply hardcoded to read from the device a single sector, sector 0, sized 512 bytes and gets instead a 4096 one and panics ) you are perfectly right  and a "third party" USB stack, 4K compatible is needed

This is indeed the case. More properly, the BIOS issues an int 13h asking to read ONE sector's worth of data (BIOS int 13 interface does NOT know how large one sector is, although for ISA/IDE hard disks it had always been exactly 512 bytes until recently).

My (limited number of, but several) USB_boot capable BIOSes just lock up when trying to access the disk, and I assume it is the rule rather than the exception, is it not so in your experience ?

(and of course it needs to reside on an internal hard disk or on a "helper" media)...

Therefore, I reiterate my pledge for Grub4DOS and/or other (pre-)boot environments
which wish to support booting from external USB-drives to implement proper support for 4k-sectores disks !

[Wonko the Sane]

Therefore, I reiterate my pledge for Grub4DOS and/or other (pre-)boot environments
which wish to support booting from external USB-drives to implement proper support for 4k-sectores disks !

Before pledging, what happens right now in grub4dos 0.4.6 (booted from internal or "helper" media)?

Does it see the disk device?

Does it sees it not?

Etc.

Then, provided that the device is seen, HOW is it seen? (i.e. through the geometry command?)

 

Wonko

[Ninho]

What happens right now in grub4dos 0.4.6 (booted from internal or "helper" media)?

Wonko

Again : please tell/remind me of the proper place, i.e., URL, where to get the Grub4DOS version you're having in mind ! Is it Chenall's ?

[Ninho]

OK... I already was using 0.4.6a,which I have now updated to Chenall's latest 01/17 2014 build. No joy ! With the offending USB drive connected, > usb --init -> Found 1 USB device Drive num : 0x0; Obviously, should be 0x82 (this machine has 2 physical fixed IDE drives attached). With, instead, a standard USB-flash attached (reporting 512k sectors), Grub4DOS correctly identifies and reports the flash disk as BIOS 0x82. I'm not sure whether this G4D branch has its own USB stack, or whether it uses the BIOS, but in either case, it is conspicuously not prepared to manage 4K-sectors yet :-( Obviously, G4D is not prepared for 4k-sectors.

[Wonko the Sane]

Again : please tell/remind me of the proper place, i.e., URL, where to get the Grub4DOS version you're having in mind ! Is it Chenall's ?

Why do you think I posted this?

 

http://reboot.pro/to...b4dos/?p=186085

 

Obviously the idea is that you go here:

http://grub4dos.chen...egories/0-4-6a/

and get the very latest version.

 

And obviously latest is:

http://grub4dos.chen....6a-2014-12-27/

 

You see, I can fit two "obviously" in a post as well.

 

Wonko

[Ninho]

Obviously the idea is that you go here:
http://grub4dos.chen...egories/0-4-6a/
and get the very latest version.
Wonko

That very latest... does not not detect the problematic device any better
than the January build tried earlier. Although the diagnostic issued is different
(I wouldn't trust its report to be accurate. Obviously :=) )

> usb --init
-> Error 82. No USB device found. USB device not ready.
(sic!)

Edited by Ninho, 28 December 2014 - 12:13 AM.

[Agent47]

 

 

Ideally it should be provided by a BIOS (update), however it seems the industry never got it.

 

 

I disagree. Industry got it right and that is why we have UEFI to replace legacy BIOS. One of the feature of UEFI firmware is to boot from native 4K sector sized disks. 

[Wonko the Sane]

I disagree. Industry got it right and that is why we have UEFI to replace legacy BIOS. One of the feature of UEFI firmware is to boot from native 4K sector sized disks. 

Sure  and this happens EXACTLY at the same time when any decent PC will use for booting a 128/256/512 Gb SSD (that of course has no reason to be "native 4K") and *any* rotational hard disk actually bigger than 2 Tb (which may need to be 4K sectored) is used for storage only and not for booting, and of course coming with EFI/UEFI is GPT style of partitioning, also completely UNneeded for anything below 2.2Tb in size but however well capable of addressing a fantazillion  of 512 bytes sectors.

Maybe the industry got it right, but for them only, what was pushed (and is pushed) on us is a sort of forced obsolescence (if it was planned obsolescence the EFI/UEFI and GPT would not be the mess they are).

 

@Ninho

At this point you may really need a EFI/UEFI based solution, possibly (but cannot say for sure) Clover will do.

 

However I would be interested to know what happens (with the version of grub4dos 0.4.6a that actually detected the USB device), in this scenario:

find (

[TAB]

usb --init

[ENTER]

find (

[TAB]

is a device added to the list that find outputs?

 

I mean the usb device is seemingly found, but it is not found as hard disk device, drive num 0x0 may mean the (fd0)

 

 

Wonko

[Ninho]

...

Maybe the industry got it right, but for them only, what was pushed (and is pushed) on us is a sort of forced obsolescence (if it was planned obsolescence the EFI/UEFI and GPT would not be the mess they are).

 

@Ninho

At this point you may really need a EFI/UEFI based solution, possibly (but cannot say for sure) Clover will do.

 

However I would be interested to know what happens (with the version of grub4dos 0.4.6a that actually detected the USB device), in this scenario:

find (

[TAB]

usb --init

[ENTER]

find (

[TAB]

is a device added to the list that find outputs?

 

I mean the usb device is seemingly found, but it is not found as hard disk device, drive num 0x0 may mean the (fd0)

 

 

Wonko

I didn't mention it previously, but actually - based on Adaptec's (abandoned) ASPI SDK, a few months ago I wrote a MSDOS device driver

that accesses FAT (12/16/32) partitions on the type of disks in question. Works OK - not finalised for distribution, mainly because "long filenames" DOS drivers for FAT/DOS in existence are compatible with 512-byte sectors ONLY, their authors/maintainers lost interest long ago,

and I feel asking a regular user to use only 8.3 names on a portable storage device is not realist nowadays. 

Anyway, my experiment suggests Grub4DOS could rather easily process the 4K sectors if they wanted to.

 

 

I'll try the Grub4DOS commands per your above suggestion. But note I have a real legacy floppy controller, not USB, couldn't have triggered the (false) detection.

[Wonko the Sane]

I'll try the Grub4DOS commands per your above suggestion. But note I have a real legacy floppy controller, not USB, couldn't have triggered the (false) detection.

Sure, it is well possible, but it is a strange result, if *somehow* the device is found and mapped to *something* in grub4dos, then there is a chance of reading it.

 

I don' t think that it is such a great issue to add the 4K sector compatibility to the Grub4dos USB stack, AFAICU the issue is more or less something about a "conversation" going *like*:

g4d: hey, disk, give me your first 512 bytes

disk: sorry I cannot, the minimum amount of bytes I can deliver is 4096

g4d: Ah, ok, then please send me 4096 bytes, please, I will use first 512 and trash the rest

 

It is what happens later that might make things difficult (or impossible) without some major re-writing of the code.

 

I now see that the issue is probably originated  still by the same PLX controller of a few years ago:

http://reboot.pro/to...or-4kb-sectors/

and I found a reference to that same device here:

http://forum.hddguru...?t=21099&start=

 

Personally (with all due respect for you and with all the sympathy for your troubles with that specific device) I would like to see reports of more devices behaving like that one before making a case out of it.

 

Random thought, but your device driver (I presume together with some DOS USB drivers) can access that disk from DOS, right? (let's put aside the 8.3 names limitation)

And what happens when you run GRUB.EXE on that drive from DOS?

 

Wonko

[Ninho]

A few answers (to Wonko upthread) :

 

1. You think it's only this (or a few) specific device ? I was thinking it is a common issue and by (unintelligent) design. but I don't know one way or another, failing a sample of relevant testimonies. Nobody seemed to complain only proves people are fatalist and accepting whatever dogfood they are forced to swallow...

I too would like to read reports by (knowledgeable) users...

 

2. 

BIOS int 13, used for booting, deals with sector numbers - but USB mass storage commands, being a thin wrapper around SCSI-like commands, address disks in bytes rather than sectors. Therefore a BIOS extension, which G4D installs for USB disks has to do a conversion, bytes <->sectors and in order to deal properly with different sector sizes, it should start by sending an "identify" to the disk to get its sector size, then use the reported sector size in converting sectors to absolute bytes. Else the driver will try to read/write bytes at a wrong offset !

There is another aspect to this  : although internally specified as number of bytes, individual tranfers had better be an exact multiple of the physical sector size (as presented at the bridge-host interface). Trying to read 512 bytes might fail when the sector size is actually 4096 bytes. 

 

3. My driver is a pure DOS "block device driver" (leveraging the USB stack provided by USBASPI.SYS) which DOS in turn leverages to assign drive "letters", and provide the DOS disk API - including int 21, int 25, int 26 - for each FAT partition on the disk. But I have not bothered to implement the BIOS (int 13) API (that is not required or used by DOS anyway). As such, you cannot expect Grub4DOS to in turn leverage it to recognise the USB. Grub4DOS needs the BIOS not DOS to access UMSDs, and failing that it needs to mount its own USB BIOS replacement. Even if I had provided an int 13 extension (possible in principle) I am not convinced it would be usable by G4D, running in protected mode, to bootstrap the disk.

Edited by Ninho, 29 December 2014 - 12:20 AM.

[Wonko the Sane]

#1 I am only saying that if it was "common" we would have most probably seen more requests/protests/error reports/etc. dealing with this specific matter, with the exception of your specific devices, I have seen not any other case with disk drives smaller than 2 Tb

 

#2 Exactly , that is exactly the kind of issue I tried to "simplify" in the (completely fake) example conversation between disk and grub4dos.

 

#3 Obviously I don' t know what you have in your driver (or what you don't have), and quite frankly I doubt that DOS (unless it has been patched specifically for 4Kb sectors) will actually be able to work (or to work fully) on a 4 Kb sectored device, see also:

http://www.msfn.org/...s-not-detected/

http://www.msfn.org/...-now-available/

I personally see this as the most challenging part, i.e., provided that grub4dos supports booting from/accessing the 4Kb device WHICH OS are we going to boot from it?

 

But (as an example) what happens when we map a grub4dos --mem disk from an image on the stupid external disk?

Or maybe we can dd a stream of bytes to a new (virtual) device in it?

Review the good ol' hmload.com approach:

http://sysdocs.stu.q...ent/GrubForDOS/

Or *something* else....

 

Coincidentally (though I still doubt it would change anything) tinybit has just released an experimental version of grub.exe that has some interesting new capabilities, JFYI:

http://reboot.pro/to...os-and-dos-now/

 

Wonko

[Ninho]

Couple comments, #1 yes of course it's "only" above 2TB that 4k sectors became compulsory.
At a point in time, MS recommended and manufacturers found agreeable,
that "all" external USB MSD would have the 4k sectors.
For instance the Iomega device I bought sold in 3 sizes, 512 G/1T/2T : of course
they used the same hardware/firmware, safe for the actual disks. I would think
this became the industrywide norm rather than the exception.
The fact we don't read many complaints, I think, is because most people - other
than us geeks - would want to install bootable OSes on external, secondary drives.

#3 Vanilla, unpatched, MS-DOS (not FreeDOS, unfortunately) works flawlessly with 4k-sectors !
Even larger sectors can be accomodated by DOS "block" drivers - the *official* limit per Microsoft
is 8k, although experimentally (and unreasonably) the *actual* sector size limit seems to be 32 k !

Edited by Ninho, 29 December 2014 - 01:22 PM.

[Wonko the Sane]

The fact we don't read many complaints, I think, is because most people - other
than us geeks - would want to install bootable OSes on external, secondary drives.

Well, though I do consider myself a geek I would not even think of buying a non-512e disk and more generally *any* disk larger than 512 Gb, particularly in a 2.5" size, maybe we will need to draw a line between "old school" (or if you prefer "antiquated") geeks and  "new, modern" geeks.

And, given that no OS (except maybe the stupid Windows 8.x or later, possibly only on UEFI and some Linuxes) seems like being fully compatible with these devices, as hinted before, even once a way is found to boot *something* from them, it won't run or not run properly.

This said, it would be of course fun to find a way to bypass this senseless limitation.

 

#3 Vanilla, unpatched, MS-DOS (not FreeDOS, unfortunately) works flawlessly with 4k-sectors !
Even larger sectors can be accomodated by DOS "block" drivers - the *official* limit per Microsoft
is 8k, although experimentally (and unreasonably) the *actual* sector size limit seems to be 32 k !

Well, this is completely UNLIKE what RLoew reported :

http://www.msfn.org/...lable/?p=903095

as according to him a few patches to IO.SYS are needed.

 

Which version of MS-DOS did you try/test?

And how exactly did you test it? (I mean any chance that your tests were not exhaustive or involved some hardware that *somehow* provided nonetheless 512 bytes sectors? One thing is accessing a filesystem residing on 4k "native" device and another thing is to actually boot DOS from it.)

 

There were some talks in 2012 about supporting 4k sectored devices in FreeDOS, but I seem to remember that in the end the good guys set this to low-low-low priority.

 

Wonko

[Ninho]

Wonko wrote :
...
Well, this is completely UNLIKE what RLoew reported :
http://www.msfn.org/...lable/?p=903095
as according to him a few patches to IO.SYS are needed.

Reply: I'm aware and appreciative of R Loew's extensive work. That said, keep in mind he's SELLING his patches for MS-DOS and Win 9x, which explains why you must take his claims with a grain of salt and put them in perspective. The truth is the MS-DOS kernel IS prepared to accomodate 4k-sectors on *installable* (in config.sys) drives. My tests (and reading leaked sources) have confirmed that.

What R Loew's patches would address are 1. support of 4-sectors by the disk driver BUILT-IN IO.SYS, and for booting from a 4k-sectored device, and 2. bugs with utilities that are not part of the DOS core itself (e.g. "format.exe"). 3. Fixes on the Windows 9x side, are however irrelvant in the context of this thread. - Disclaimer : I have not bought or tried R. Loew's patches. Am speculating from claims on his website and his and others' forum posts.

&gt; Which version of MS-DOS did you try/test?
And how exactly did you test it? (I mean any chance that your tests were not exhaustive or involved some hardware that *somehow* provided nonetheless 512 bytes sectors? One thing is accessing a filesystem residing on 4k "native" device and another thing is to actually boot DOS from it.)

Dos 6 and 7. Even checked DOS 3.21 (that had minor bugs).
First tests used a modified ramdisk with 4k sectors, later I wrote the support needed to mount the external USB drive and confirmed it works satisfactorily. Creating/deleting/accessing files on several partitions, cross-checking the files in Linux and Windows XP. If someone wants me to stage more "exhaustive" tests, please contact me for a quote :-) I stopped developing the concept further for two main reasons - lack of working support for 4k sectors in the FreeDOS kernel ("Perdition" has done some work towards that goal), and lack of support for LFNs.

IF we ever get Grub4DOS or another boot environment to install proper BIOS extensions as I ask in this thread, then I'll get R. Loews patched IO.SYS - or I'll try to do my own.


&gt; There were some talks in 2012 about supporting 4k sectored devices in FreeDOS, but I seem to remember that in the end the good guys set this to low-low-low priority.

Yep. FreeDOS core development is mostly stalled, unfortunately.

Edited by Ninho, 29 December 2014 - 03:43 PM.

[Wonko the Sane]

Well, you are just confirming that disk/volume access needs no patching whilst booting from the 4K sectored thingie NEEDS a patch.
 

One thing is accessing a filesystem residing on 4k "native" device and another thing is to actually boot DOS from it.)

 
And we are back to the basic issue:
 

I personally see this as the most challenging part, i.e., provided that grub4dos supports booting from/accessing the 4Kb device WHICH OS are we going to boot from it? 

 

DOS (as is) does NOT boot from 4k sectored devices.

Windows 9x/Me (as is) won't.
As well (if we do trust MS info on this) none of the NT based OS's (exception made for the stupid 8.x and later) will boot from them, and possibly even those ones on UEFI only.
The various Linuxes very likely can be tricked into booting from these devices, but since there is no BIOS support for those, a "helper" is anyway needed, and then this helper may consist of the GRUB2 (provided that it does support the 4K sectors ) or go all the way down to a kernel+intird.
 
It would make then still make more sense to try Clover or a similar "fake UEFI" approach.
 

Wonko

[Ninho]

Indeed this topic has been 'derailed' somehow into the specifics of DOS mods,
which although interesting per se was not the original question ! I'm primarily
interested in booting Linux from the removable; DOS is only present in the equation
as a base for running Grub4DOS, from a standard, fixed disk.

For that is needed fixed support for 4k-sectors, at least on USB-attached disks
- and possibly directly attached disks too, though I have no idea if customer oriented SATA disks with 4K native - no emulation - sectors are in existence or in production yet.

Does Tinybit still visit reboot'pro ? Does someone know if fixing G4DOS's detection and support for 4K sectors on USB is on the TODO-list ?

And...lest i forget... Happy new year to all !

[Wonko the Sane]

NInho, this, JFYI, is an interesting new kind of issue (first time I come across this one):

http://www.msfn.org/...ent-interfaces/

 

Wonko

[Ninho]

Re: the issue reported at MSFN. Not too surprisingly, their device presents 4K sectors
when attached thru USB, but (emulated) 512 byte sectors are passed-thru its eSATA interface.

There is no way the disk can be formatted to satisfy both, unless they write a "filter" driver
in the OS to convert sector numbers on the fly during read/write operations.


Re: Grub4DOS. Instead of relying on its built-in USB stack, I've tried to provide
my own (experimental) int 13 support. G4D does find the disk but all its in/out fail with a grub4dos "Error 25". Since my DOS driver albeit uncompletely tested is well able to read/write
4k sectors, I have to hypothesize that G4D is not taking the sector size, as reported by
int 13/48, in account, and defaults instead to 512 bytes sector for BIOS hard disks
At least, it does not crash nor locks up ;=)

Chenall, Tinybit, can you hear me cry ?

Edited by Ninho, 03 January 2015 - 07:52 PM.