Jump to content











Photo
- - - - -

[RELEASE] idd.g4b interactive dd command composer


  • Please log in to reply
59 replies to this topic

#26 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 16 October 2021 - 03:04 PM

Well, adding the --heads=0 --sectors-per-track=0 costs nothing, if it helps, why not adding it :).

 

The (hd0)63+2 issue is a bummer :(, but here it seems to be working just fine. :w00t:

 

It must be *something else* :dubbio: in my case it is a "normal" (smallish) hard disk with a FAT 12 partition, with bootsector on (hd0)63, i.e.

geometry (hd0)

drive 0x80 (LBA): C/H/S=20/16/63, sector count/size 20160/512

Partition num: 0, Filesystem type is FAT12, partition type 0x01

Partition num: 1, ....

 

:duff:

Wonko

Attached Thumbnails

  • hd063+1.jpg


#27 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 16 October 2021 - 07:29 PM

Congratulations with your (hd0)63+2-success, but I think (hd0)63+1 will work too if partition fits into memory?

Screenshot_20211016-211836.png

BTW: My memory on Limbo x86 is only 128MB.
BTW2: After mapping (fd1)+2 rd_size was 2 sectors.

Addition to previous post:
I said your script exited silently while mapping the 512 bytes of-file. This is not true, I forgot to mention the -unsuccesfull- map command can be seen on screen.
Screenshot_20211016-132015.png
After adding --heads=0 --sectors-per-track=0 no problem with the file anymore.
Screenshot_20211016-132500.png

#28 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 17 October 2021 - 08:26 AM

I see.
It makes sense, while not making any :w00t:.

While we are clearly abusing the use of the map command, it is a sort of bug, or rather a "wrong" philosophy.

The grub4dos checks by reading the contents of the sector(s) what size the image (that not necessarily is an image) BEFORE seeing that what is asked is to map only 1 or 2 sectors, and if the whole image fits in memory, maps what it was asked to (1 or 2 sectors only) otherwise throw that error 28.

Question is if it can be disabled with errorcheck off or with one of the other tricks/flags/whatever.

I need to make a stupidly large (fake) MBR and EPB to do more tests.

:duff:
Wonko

#29 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 18 October 2021 - 07:41 AM

I did a few tests.

 

It seems that the automatic/automagic detection of a MBR or of a bootsector depends on the presence of the Magic Bytes 55AA signature.

 

BUT the stupid grub4dos (at least the one I am testing with) auto-protects *anything* with these Magic Bytes, preventing changing them :frusty:, when on floppy device (I mean a file like (fd0)/dev_mbr.bin or a block such as (fd0)313+1).

 

This creates a whole bunch of unexpected errors in batches making use of  dd, cat --hex --replace and write, on floppy sectors that have the 55AA magic bytes, (error 29 disk write error)..

 

Most probably :dubbio: there is somewhere a documented way on how to be able to overwrite the 55AA with (say) 44AA of a sector on floppy  :dubbio: (on hd it seems to be working :)). but right now I am failing to find this info :unsure:.

 

The "right" way to operate to be able to map to (rd) *whatever* we like would be to:

1) check if the file has 55AA at offset 510 of its first sector

2) if it has it not, proceed, this is likely something like 99% of cases

3) if it has it, replace in the source file/sector with - say - 44AA

4) map to (rd)

5) replace back on both the source and (rd) the 44AA with 55AA

6) proceed with the comparison/crc32

 

The 1+1,(actual source) besides being "ugly" would cover like 99.999% of cases, but not a few corner cases, like a copy of a "normal" hard disk first track, where the second sector is actually a copy of the original MBR after grldr.mbr installation.

One could replace the 1+1 with the first sector found that has NOT the 55AA, but it would remain IMHO "ugly".

 

If anyone knows how to workaround this floppy limitation, please let me know, it is very possible I am missing something obvious, either in the floppy write options or in the map to (rd) ones.

 

:duff:

Wonko



#30 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 18 October 2021 - 05:18 PM

Very good research, interesting but 'weird' results. :blink:

 

You're right that 'borrowing' a sector isn't always a solution. See for instance next print-screen:

 

Screenshot_map_FAT32_partition.png

 

I found another really 'weird' case. In next print-screen. The involved file TEST.BIN is only 512 bytes!

 

Screenshot_map _512_bytes_file_with PBR.png

 

As can be seen on the print-screen, can be still mapped as a blocklist with a 'borrowed' sector outside.

 

About 'ugly' solutions: writing to files/ blocklists \x44\xAA to replace \x55\xAA in order to verify their content, doesn't seem desirable to me. There are always risks involved in such methods.

 

I have tested errorcheck, but only result is the error is not send to screen.

 

But 'borrowing' a sector without the 55 AA-signature seems not so bad to me. Can be done with a simple loop by checking output of the map-command with %@retval%. See next print-screen.

 

Screenshot_map_blocklist_with _borrowed_good_sector.png

 

I do not really understand following part of your post:

 

This creates a whole bunch of unexpected errors in batches making use of  dd, cat --hex --replace and write, on floppy sectors that have the 55AA magic bytes, (error 29 disk write error).

 

Can your give more information/ examples in which cases this would be a problem?

 



#31 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 18 October 2021 - 06:39 PM

Example:

you boot from floppy

on the floppy there is a copy of (say) the MBR of (hd1), i.e. (hd1)0+1 in  a file called mbrhd1.bin

you load it in (say) sectedit.g4b and change (say) the disk signature

you now want to save it to (hd1)0+1 (which you can do just fine)

but you also want to save it to the file on floppy 

the sectedit.g4b crashes (badly, but the way it crashes can be bettered)

 

I can understand why the bootsector of the boot floppy might be protected, but a file?

And a file accessed as blocklist?

 

I don't understand why I am allowed to make huge holes in a hard disk freely and I cannot change a few bytes on a floppy file (or sector).

I want a flag "YESIWANTTOWRITETOFD0".

BTW this is not limited to sectors with the 55AA signature, the whole floppy cannot be written to, but - in common use - making a backup of some hard disk sectors and/or modifying them is - or it seems to me - a quite natural use of grub4dos.

 

As well I would like a flag for map ;):

map --do-not-parse-first-sector-I-already-set-rd-size-and-anyway-I-only-want-to-map-a-small-file (fd0)425+2 (rd) 

 

In verifydd there is already a :chblocks sub-routine that can "separate" the device from the blocklist, since it uses Fn.11 it is (should be) "universal", it is just a matter of adapting it.

 

The 1+1, or - better n+1, remains "ugly" (at least in my eyes), but it seems like it is the only way out.

 

I have to think at how this can (should) be applied to the various cases.

 

What happens if (say infile - but outfile would be the same) is a fragmented file that happens to have 55AA at offset 510?

You cannot use the n+1, trick because the file cannot be represented as a (second) blocklist. :dubbio:

If there is not a way files 00xxx, 10xxx and 01xxx won't work. :unsure:

And we will have to go back to your dd approach (but again using it makes verifydd pretty much useless).

So at the end of the day it would be better to use the "plainer" (md) device.

Since as you noticed when fresh booted the rd_base is 0, one could "initialize" the (rd) with *any* sector, than map the file to (md) at the (rd) address and then change (rd) size accordingly. I'll have to do some more tests  :dubbio:

.

 

:duff:

Wonko



#32 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 18 October 2021 - 11:14 PM

I will try to reproduce such a crash.

About mapping a fragmented file containing 55 AA at offset 510. I do not see the problem (or do not understand).
In my previous post I showed that even a one-sector file containing a FAT12/16 PBR can be mapped as a blocklist while 'borrowing' some n+1 'good' sector.

In print-screen below I used a fragmented file after copying a PBR to first sector and a few other sectors. I borrowed 1+1 as a 'good' sector and mapped the fragmented blocklist successfully to (rd).

You can see all 55 AA-signatures are shifted up 512 bytes. So your meta CRC32 should be possible if based on the file-size with an extra 512 bytes-skip (not on the blocksize - which will in many occasions be 1-511 bytes 'bigger').
Screenshot_20211019-011222.png

#33 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 19 October 2021 - 08:38 AM

Yep, but one of the advantages of the mapping to (rd) is about NOT having to deal with a blocklist.

 

As long as it is a handful of blocks it may work, but I am afraid that in the "real world" it will be common to exceed this (or that) limit of the mechanism.

 

The mapping to (rd) was (IMHO) a good idea because it was grub4dos that managed automagically address and size, but - essentially - the (rd) is just another (convenient in this case) way to access an (md) extent as a "monolithic drive".

 

If it was possible to (hypothetical) a map (hd0,0)1+1,/myfile.ext (rd) it would be another thing, but having the /myfile.ext represented by a potentially endless list of blocks is a no-go.

 

Back to the drawing board.

 

:duff:

Wonko



#34 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 19 October 2021 - 09:51 PM

It's all up to you of course.

Luckily the user can still force the n+1 trick if used before in the (raw) dd-commandline.

 

Today I tried to make a backup of the 'whole' FAT32 from a 8GB VHD to a file (in VBox). To avoid problems with VERIFYDD.G4B I borrowed 'friendly' sector 2+1 (although third sector of FAT32!).

 

Although everything was fine, afterwards I was a bit surprised by your 'crazy flag' in this case. Everything was a whole multiple of sectors, skip & seek exactly one sector. VERIFYDD.G4B should have taken the 'real' crc32 in my opinion. Instead I had to wait a very, VERY long time until your 'flags:110' finished. While waiting I had time to think: in fact the whole mapping-problem shouldn't be there in this case.

 

VERIFYDD.G4B raw dd if=(hd0,0)2+1,0+16790 of=block (of 2+1,8395-file in 3 fragments) bs=1 count=8395k skip=512 seek=512 + crc32 blocklists III.png

 

BTW: For comparison the 'real' crc32's in the picture too.

BTW2: First I forgot to specify bs, only skip=1 and seek=1 (sector). VERIFYDD.G4B took this as one byte??



#35 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 20 October 2021 - 08:19 AM

Well, if you add the sector - as is - the batch (correctly? :unsure: ) considers it a fragmented file, as there is a comma in the output of the blocklist command for your infile and outfile in the dd command.

 

If we go that way we will need to "carve an exception" of sorts.

 

The 110, actually 00110 means:

0 infile is fragmented[1]

0 outfile is fragmented[1]

1 skeep is 512 or multiple

1 seek is 512 or multiple

0 bs*count non multiple <- this is WRONG, should be 2 (but this doesn't actually change the choice of algorithm used)

while doing other checks, I found that *somehow* the flag mechanism didn't work correctly :frusty: in a few (other) cases for the bs*count check, it will hopefully be fixed in the next version, but it won't change the first two 0's,. and the two 1's, right now the batch covers only a part of 11xxx and the xx222 cases "differently", all the rest go to the meta-crc calc.

 

If you prefer, it seems to me like you "forced" the batch to use the meta-crc by adding the sector, if you had it not it would have used direct crc32 and you would have not needed to load it to (rd), which is what would have caused the too large to fit in memory error :unsure:

 

No idea about the bs/skip/seek=1 issue, the batch includes a check/calculation for those cases:

 

 

#special cases for omitted values
if not exist count set count=0
if not exist bs set bs=512
if not exist skip set skip=0
if not exist seek set seek=0

 

and later:

 

 


set /a amount=%bs% * %count%
#special case of count=0
if %count%.==0. set /a amount=%ifsize%

 

 

#we need to normalize parameters to bytes, i.e. as if with bs=1, and check both skip and seek
set /a skip=%skip% * %bs%
set /a seek=%seek% * %bs%

set /a bs=1
set /a count=%amount% / %bs%

 

I am failing to see where the issue could have arised. :unsure:

 

:duff:

Wonko

 

[1] because it is not a single blocklist, if we adopt the 1+1, approach it must be implemented in the batch, not added in the dd command, flags in this cases will become 11110 or 11111 or 11110 and will use the "direct" crc32



#36 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 20 October 2021 - 06:06 PM

Thanks for information about your 'crazy flags', I didn't studied that part of your script.

Sorry about the bs/skip/seek=1 issue, can't reproduce it anymore. Probably I set somehow bs=1 before.

If i do this (setting bs=1 before running dd and VERIFYDD.G4B), I see the skip/ seek of 1 bytes again, so not a direct error in the script (although cleaning used variables is always a good idea).

Also i see another problem. Because I wanted to do a 'quicky' with only 10 sectors 'infile', copied to 'outfile' I specified before (1GB), now I am punished with endless 'flags: 10001'. Shouldn't be related to bs=1, also size of 'infile' looks good.

VERIFYDD.G4B bad with existing var bs=1 also bad if ifsize smaller than ofsize I.png

About the n+1 approach, I thought I understood you didn't want to go along this way anymore?

I think I found another approach for cases skip/ seek are not whole multiples of one sector, but still using map FILE (rd). In fact it is a variation of my earlier suggestion of copying with dd to (rd)+1 with needed skip/ seek.

I found after mapping to (rd), rd_base can be set higher and rd_size can be set lower, without the (remaining) content of (rd)+1 is changed.

I invented a somehow practical example: copying the first 1GB of a disk to file, without the MBR boot-code.
In the next print-screen combined with the n+1 trick.

1GB met skip=seek=446 CRC32 (rd)+1 na map to (rd) na raw dd if=(hd0)0+sectors of=f2 bs=1 count=1GB-446 skip=446 seek=446  buf=64m buflen=1g GOOD III.png

To check everything is right, I compared crc32 of the file with the crc32 of resized (rd) after mapping the blocklist with n+1 added (in this case no skip/ seek of course). Looks correct to me.

1GB met skip=seek=446 CRC32 1GB-file en na map bloklist 1_1,file1gb en rebase&resize (rd) GOOD IV.png

If I am right, this will make your Fn.42-approach superfluous :( , which is far to slow anyway. :ph34r:

#37 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 21 October 2021 - 10:22 AM

Hmmm.

 

You seemingly  tried to extract an amount of 1073741824 bytes out of a file that only contains 5120 of them :w00t: :ph34r:

 

dd as I noted earlier is a "strange" tool, on one side it is very exact (and potentially "dangerous"), on the other it allows - autocorrecting them - some "user errors".

 

If PC's had a robotic arm :w00t:, I would personally implement an "aggressive" interface, which would slap the user in the face shouting "are you §@ç#ing crazy?" ;)

 

If you had attempted to write that dd command through the idd.g4b you should have gotten a warning that you were trying to transfer more bytes than the infile contains.

 

Of course it is not a problem to check that "amount" does not exceed infile size (and "correct" the data in verifydd.g4b), will do :).

 

About the 1+1, approach, the issues are twofold:

1) it is "ugly" <- but once said that, who cares, if it works, it works :) in war and love and grub4dos scripting everything is fair

2) it won't work with fragmented files and as such it is only good for contiguous blocklists (in practice it is a good thing, as it will cover all the 11xxx cases different from the two already covered by :begins11 , i.e. 11210:11221 and 11110:11122

 

 

#test to see if quick verify is possible
if %flags:~-3,3%==222 call :ends222 &; goto :eof
checkrange 11210:11221 calc %flags% && call :begins11 &; goto :eof
checkrange 11110:11122 calc %flags% && call :begins11 &; goto :eof
#only first three cases above actually working in a different way
#the other ones are all together in the (rd) approach

 

Some comments from the version of verifydd.g4b I am currently working on, this is the subroutine that is called if there is a 55AA at offest 510 in the first sector of the file/blocklist:

 

 

:ends55AA
#if we are here there are two possibilities:
#1) either both infile and outfile are a (single) blocklist
#2) at least one of the two is a file (fragmented)
#let's make the easier case #1 above, flags must be 11xxx
checkrange 11 calc %flags% / 1000 && echo flags is 11xxx
#now if flags is 11xxx we can use the +1, trick by deomsh
#we need to find a sector - any sector - on the same device where the file/blocklist resides
#that has NOT 55AA at the end, we try first following sectors (if any) and then we look on following sectors on the device
#we re-use the chblocks subroutine to get the device, blocks start and blocks length from current blocklists
## it is a rare-rare case, but what happens if infile or outfile are the last sector of a device?

 

 

 

I found after mapping to (rd), rd_base can be set higher and rd_size can be set lower, without the (remaining) content of (rd)+1 is changed.
 

You are under-selling yourself.

 

You are telling me that::

the (rd) mapping can be considered a resizable sliding window on the contents of the (md) device with single byte resolution.

Now it sounds a lot more professional. :thumbsup:

 

And yes, it should work just fine and be much faster than the Fn.42 piping into crc32 :).

 

Still the problem with fragmented files remain, I know that the list of blocklist works, but it works - more or less - as long as the amount of bytes in the blocklist command does not exceed this (or that) limit, since we are now using a "blocklist %infile% | set ifblocks=" the limit is currently grub4dos variable size, this could be worked around writing to a (temporary) memory area *like* "blocklist %infile% > (md)0x60000+1" and then parse it with cat, but is there a limit in the size of parameters of the map command?

I suspect there is one, so then we would need to "partialize" the output of blocklist in several map commands, it seems to me like it is a no-go. :(

 

On the other hand, my idea of invalidating the magic bytes and restore them after the mapping to (rd) is risky as if there is an error after the writing of the 44AA and before the restoring to the previous 55AA the source and or destination file may become invalid (it is a perfectly fixable issue, still it is "ugly", possibly even uglier than the 1+1, trick ).

 

:duff:

Wonko

 

P.S. attached version 0.05, still Fn.42 meta-crc, but added the "n+1," trick, (the batch is a mess, will need some re-ordering/cleaning, but it seems like being working), I am particularly preoccupied by the loop to find first non-ending-with-55AA sector that might be invalid in some corner cases.

Attached Files



#38 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 21 October 2021 - 10:31 PM

Nothing wrong with dd. Stopped after the 10 sectors were copied. IDD.G4B is not so handy for my experiments: doesn't take variables (in between I made some changes: call var=%var% after each set /p line - a direct 'call set /p' is not working).

About the fragmented blocklist: Can (normally) only be a file. And file is only not mappable with 55AA on offset 510 in the first sector (together with other -unknown MBR/ PBR 'parts') if I am right?

In this case it would be easiest to copy the whole file to (rd)+1 with dd, and check both with crc32. Then (rd) can be resized etc. Can even making the n+1-trick superfluous, if used for infile/ outfile both.

Thanks for VERIFYDD.G4B v005. I will test later.

#39 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 22 October 2021 - 11:13 AM

Well, idd.g4b is intended as a manual helper for dd-impaired people :w00t:, so it is a good thing (in my perverted mind) that it doesn't take variables, (it should not by design).

 

Yes, the "worst" case is a fragmented file (multiple blocklist) with 55AA at offset 510 as either (or both) infile and/or outfile.

 

For all it matters we could dd to (rd) also the non-fragmented files, but (though I suspect that under the hood the map to (rd) command uses the same "base" routines as dd) the whole point of a verification is that it must be made through a different mechanism.

If you

a. dd "A" to "B"

b. then dd "A" to (rd) and crc (rd) and crc matches with below.

c. then dd "B" to (rd) and crc (rd) and crc matches with above.

 

If we use some flags, and we use 1 for correct and 0 for not correct, and call the operations abc as per above, we have (should be 8 or 2^3) possibilities:

abc

1. 100 <- this creates NOT an error, the actual "main" operation "a" succeeded, but since both the b and c ones failed (in the same manner) it seems OK

2. 101<- this creates an error BUT the "main" operation "a" succeeded nonetheless, case indistinguishable from below one

3. 110<- this creates an error BUT the "main" operation "a" succeeded nonetheless, case indistinguishable from above one

4. 111 <- this is what should happen, if we always get this we can say that dd is perfect and just stop verifying its behaviour

5. 011 <- this is the most "dangerous" case, for *some reasons* the "main" operation failed but the two to (rd) ones gave same result

6. 010 <- this creates an error AND the "main" operation "a" failed, good, case indistinguishable from below one

7. 001 <- this creates an error AND the "main" operation "a" failed, good, case indistinguishable from above one

8. 000 <- this is the other most "dangerous" case, for *some reasons* the "main" operation failed but the two to (rd) ones gave same result, this should be a rarer case than #5 above, still it is possible.

 

Out of 8 cases, the procedure is "right" in cases:

#1 (out of sheer luck)

#4 (correctly)

#6 (correctly)

#7 (correctly)

 

And "wrong" in cases:

#2

#3

#5

#8

 

So - statistically - we could use flipism instead of verifydd.g4b :w00t: :ph34r:

https://en.wikipedia.org/wiki/Flipism

with roughly the same overall accuracy ;).

 

:duff:

Wonko



#40 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 22 October 2021 - 05:51 PM

Hmm.

I suggested to use dd a/b=>(rd) in special cases only: a highly fragmented file AND skip/ seek AND failure of map to (rd).

BTW not all 55AA-cases will fail, if *some* MBR/ PBR-bytes are missing --heads=0 --sectors-per-track=0 is enough for map.
In case of a FAT12/16-PBR, experiments suggests missing first 62 bytes is already 'good'.

About your case #5: if dd a=>b failed, and copying a=>(rd) and b=>(rd) succeeds, their respective (rd)-crc32's will NOT match [1] (after resizing with value of skip/ seek to compare 'real' copied count).

About #8: I don't see the problem, result of verify=0

About #2 and #3: in this case dd a=>(rd)/ dd b=>(rd) can be repeated. Also %@retval%-return value of dd can be used (cannot be used in current IDD.G4B!).

In my more serious scripts I use a dd-loop for return-value. Can be a double loop for (first!) crc32-match of dd a=>(rd)/ dd b=>(rd).

On my quite unreliable Limbo x86 total of three 'tries' seems to be enough (but: with 'disk errors' grub4dos usually crashes).

Further I did a critical test with VERIFYDD.G4B v005: something is wrong. Copied were first four sectors of a FAT32-partition.

Screenshot_20211022-131739.png

BTW: interesting 55AA crc32-value.

[1] Assuming crc32 is always correct.



#41 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 23 October 2021 - 07:31 AM

Sure, we are talking only of the special-special cases. :)

 

About the error you got, it must be something in the loop to find the sector to add before :(, but there is *something else*, flags cannot (shouldn't) be 0. :dubbio:

 

And sure, if you take a bootsector and remove the BPB, you have a non-bootsector, the data about the size of a volume are within the first 62 bytes on bootsectors (and past 446 on MBR's).

 

BTW, even if a number of sectors (ending with 55AA) can be mapped they shouldn't. :w00t:

I mean, imagine a volume image just short of 4 GB (the default max value for map to (rd), if I get it right), you want to dd only the first - say - three sectors of it, if you try that, it will work, but the whole "slightly less than 4 GB" will be mapped to (rd) taking several seconds.

 

#5 is entirely possible, imagine that disk-to-disk operation in dd is correct BUT for *some reasons* the map or dd operation disk-to-(rd) is not correct, since a and b are the same the map or dd operation to (rd) might well be deterministic and thus give the same result, i.e. crc32(rda)=crc32(rdb), this is "dangerous", the verifydd would cry wolf when there is nothing to be seen.

 

#2 and #3 are false negatives.

 

#8 is, as said a rare-rare case, but imagine that the dd (hypothetical) error consists in writing all 00's (instead of the source content).

a (all 1's :w00t:) -> b (all 0's)

a -> (rd)=all 0's

b -> (rd) =all 0's

crc32(rda)=crc32(rdb)

 

:duff:

Wonko



#42 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 23 October 2021 - 01:48 PM

About #5: if a=b, while rd's are all zero's, crc32-a and crc32-(rda) will not match. same for b. In my proposal there must be THREE crc32 matches. First two crc32's of whole a/ b and their respective rd's. But a false negative cannot be ruled out of course.

 

So case #8 doesn't seem possible, and a false positive is most dangerous. Am I right?

 

I did some experiments with a FAT12/16-PBR that gives 'Error 28: Selected item cannot fit into memory'. 

 

This is what I found (jump is not needed, only for PBRVIEW.G4B (still unfinished)

Screenshot_20211023-142245.png

BTW: If Heads is set to 00 00 00 00, map (without --heads=0 and --sectors-per-track=0) stops, reboot needed.

BTW2: Mediabyte set to 0x0 make no difference

 

It's interesting that Error 28 is not there if UUID is set to 00 00 00 00.

Screenshot_20211023-142412.png

BTW: Also can be seen that 'jump' is not needed to identify a PBR.



#43 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 23 October 2021 - 02:35 PM

Well, the "right" way to identify a PBR is actually (due to a number of reasons) rather complex, JFYI:

https://web.archive....ystem-type.html

 

Cannot really say if the grub4dos actually uses the above (which is the right one) or any of the n possible subsets of variations, including the check for 55AA that is not-definitive (under NT based OS's, it may mean that a PBR is of a non-BIOS-bootable volume).

 

The EB XX 90 or E9 XX XX (so called "jump bytes") are not definitive:

https://msfn.org/boa...&comment=987482

but it is the same old issue, they cover 99% of possible PBRs/VBRs, but FreeDOS - say - happily ignores them.

 

:duff:

Wonko



#44 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 24 October 2021 - 12:54 PM

Interesting stuff, thanks.

 

I mapped all 11 images to (fd)'s with --mem. Only if 55AA was present, map identified a PBR. But vol identified all images as FAT12!

Screenshot_20211024-114553.png Screenshot_20211024-114912.png Screenshot_20211024-115446.png Screenshot_20211024-120242.png Screenshot_20211024-120531.png



#45 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 24 October 2021 - 02:05 PM

Yep :), this confirms that everyone detects a FATxx BPB/bootsector in a different way, the E9 jump is perfectly "legal".

 

Vol might well be "less strict" than map, after all, all it is asked for it is to read the volume name and (possibly) the filesystem type field, which brings us right here:

https://web.archive....eter-block.html

and here:

https://web.archive....fat-widths.html

and right when you think that - after all - it is not such a mess, and that with a little attention you can understand all possible issues in BPB's, you might look for the OEM field: 

https://web.archive....name-field.html

and loose any hope ;).

 

:duff:

Wonko



#46 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 24 October 2021 - 08:54 PM

Interesting stories. Maybe I can use it in behalf of my PBRVIEW-script.

 

Something else: I think found a variation of the n+1 trick, a solution to map 'problem-files' to (rd). I hope you will like it...

 

I copied first sectors to a fragmented 6mb-file (12288 sectors) from (hd0), so including the MBR. I mapped this time to (rd) with debug 3.

I had no size-error, only the unknown-head etc error. With --heads=0 etc. everything was fine.

 

But if I copied the sectors from (hd0,0): again the size-error, as before. So the PBR is the bad guy?

 

Now the n+1 extra-ugly-trick: first map the file to (hd3) - with --heads=0 etc. Then map --hook, and at last map (hd3)1+1,0+12288 to (rd). After resizing crc32's of file and (rd)+1 were equal.

Screenshot_20211024-221725.png Screenshot_20211024-222450.png Screenshot_20211024-222859.png Screenshot_20211024-223607.png Screenshot_20211024-223732.png

 

BTW: I tested a file with a size unequal to a whole multiple of sector-size, no problem too



#47 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 25 October 2021 - 03:06 PM

So, unlike what I thought (my bad) the 

--heads=0 --sectors-per-track=0

doesn't mean "ignore geometry", but rather "autodetect geometry"? :unsure:

 

If this is the case maybe one can (intentionally) force some crazy numbers for them? :dubbio:

It seems to me like (but I may well be wrong) you got away with the map to (rd) only because the hard disk image is (according to the grub4dos) invalid, if we could somehow "force" a situation similar to "Sorry, no solution ..." but without generating the error:63 it could solve at least a part of the problem (for ending 55AA but not PBR's).

 

More experiments to do ...

 

The (hd3) trick is nice :) (though complex) in a batch subjected to be used in an "unknown" environment, one must be careful with existing (or non-existing devices), i.e. probably one should use a higher (hdn), or the "next one" (which in modern grub4dos is (hd) if I recall correctly, i.e.

map *whatever* (hd)

map --hook

But then, to "exit clean" one would need to:

map --unhook

map --umnap=xxxx <- I'll have to understand/study how to convert the last disk (that should be the one that was above the "next" ) from the "normal" (hd-1) form to (hdn) and then to hex number

map -rehook

with the above three commands, the map --status should be the same as before the initial mapping. :unsure:

 

Also, in your test the infile (mapped to (hd3) ) was a blocklist (please read as non-fragmented), what will happen with a fragmented file? :unsure:

 

:duff:

Wonko



#48 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 25 October 2021 - 06:53 PM

About fragmented files: as can be seen on the first picture the file '6mbtail.000' WAS fragmented, although TWO fragments only.

In next print-screen I did same tests with a more fragmented file, about 16 fragments.

Bad news: too fragmented. So a dead end at last?
Screenshot_20211025-190415.png Screenshot_20211025-204829.png

BTW: map --mem is not a solution, because of the stupid Error 28.
Screenshot_20211025-211342.png

#49 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 26 October 2021 - 08:43 AM

Yep.

 

If a file is fragmented (I believe that the two fragments only are a - BTW useful/nice workaround to a known issue with large images on small NTFS formatted USB sticks[1]) otherwise the map --mem is needed, and this is subject at the "size verification check" (that may lead to the error 28).

 

BTW, I forgot the details, but while doing some experiments with 

map *something* (rd)

 

I managed to get an error from grub4dos *like*

you cannot use map --mem ....

 

So most probably (under the hood) when you map something to (rd) the --mem is implied (and it is logical that it is so).

 

:duff:

Wonko

 

[1] JFYI:

http://reboot.pro/in...showtopic=18022



#50 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 26 October 2021 - 08:08 PM

I have tested mapping some fragmented blocklists to (hd), up to four succeeded.

Screenshot_20211026-151314.png

 

But in the post you linked to, it seems two fragments can be too much already. So not all fragments are equal in this respect.

 

About using map --mem in case of mapping to (rd), I believe you are right that --mem is in fact used always. But I never saw errors if --mem is specified.

Screenshot_20211026-213854.png

 

I followed another possibility to map blocklists longer than size of a variable.

My idea was to redirect output of the blocklist command to some (md)-sectors. Then pipe chuncks to seperate variables and unite them while running the map command.

Although it's possible to combine some variables to a full blocklist that can be mapped to (rd), there is a limitation. 

It is the a strange number of total 1200 chars INSIDE the used variables only. Much smaller than the max command-line size of 1535-1536 chars. Ever heard of a 1200 chars limit?

Screenshot_20211026-203157.png

 

BTW: same with cat, and even with the echo command.

 

Screenshot_20211026-133357.png Screenshot_20211026-133622.png Screenshot_20211026-150715.png






1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users