Jump to content











Photo
- - - - -

[RELEASE] SectEdit.g4b hex sector editor for grub4dos


  • Please log in to reply
72 replies to this topic

#26 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15871 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 12 September 2021 - 11:48 AM

New version uploaded on first post.

 

Hopefully some bugs fixed.

 

Added (pseudo) Copy Paste functions, (yes I am lying about the clipboard).

 

@deomsh

If the source for the Load is a device (as it should normally be) just specify a higher beginning.

If the source for the Load is a file I don't like to skip/seek, because when (if) the user will save (to file) it is likely to create issues, just press N a few (many) times (until I add a goto sector or internal skip function).

Added Goto sector in version 0.3, but it is only useful for "small" sectors offset difference, If the idea is that of using large image files, then a "skip" function of sorts is needed. 

 

@steve6375

That could be an idea, but all in all, currently it amounts to:

sectedit.g4b

[Enter]

W

L

(hd0)0+20

[Enter]

[Enter]

[Enter]

Adding a number of sanity checks to save the user 3 or 4 key presses (I mean it is not like this batch can be used called from another batch, and the user has to provide anyway the right object to load into memdrive) ? :dubbio:

 

:duff:

Wonko



#27 steve6375

steve6375

    Platinum Member

  • Developer
  • 7453 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars, www.easy2boot.com
  •  
    United Kingdom

Posted 12 September 2021 - 12:23 PM

Under grub4efi:

 

1. If start with illegal parameter on command line, e.g.  fred or 

/SectEdit.g4b (hd0)0+2 

  then it seems to hang with no message, then pressing Enter returns to shell with invalid argument message

 

2. Backspace does not work in grub4efi (as I mentioned earlier).

 

Copy/Paste looks useful

 

3. error message myfile.exe) has  ) at end

 

4. If specify invalid Load device, get error message  - then if press ESC key it says 'Length in sectors (ENTER=)  

If press ESC again it aborts to shell with error

Attached Thumbnails

  • err2.JPG


#28 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15871 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 12 September 2021 - 12:59 PM

Under grub4efi:

 

I don't care.

 

When/If it will actually work properly under grub4dos, then we will. talk about the quirks in grub4efi, otherwise we will never have anything working on this,nor on that.

 

 

 

1. If start with illegal parameter on command line, e.g.  fred or 

 

Well, in real life when you perform an illegal action you risk going to jail, so it is not that bad.

 

 

 

2. Backspace does not work in grub4efi (as I mentioned earlier).

 

Backspace is NOT an accepted key that I remember. :unsure:

AND you never mentioned it before that I can see. :w00t:

 

 

 

3. error message myfile.exe) has  ) at end

 

I don't understand.

 

 

 

4. If specify invalid Load device, get error message  - then if press ESC key it says 'Length in sectors (ENTER=)  

If press ESC again it aborts to shell with error

then do not specify invalid Load device :whistling:

 

Seriously, 2/3 to 3/4 of an interactive batch problems is preventing the user from providing crazy or wrong values, there are 101 ways a user can break an input, and usually, when ALL 101 has been found the user manages to find a 102th one.

 

Let's see first if things work with correct parameters :dubbio: then we will talk of preventing the user from breaking it intentionally.

 

:duff:

Wonko



#29 steve6375

steve6375

    Platinum Member

  • Developer
  • 7453 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars, www.easy2boot.com
  •  
    United Kingdom

Posted 12 September 2021 - 01:11 PM

if not exist %dd_dev% call :status "File or device not found!" && pause && call :status "(need full device or file, i.e. (hd0)0+2 or (hd0)/myfile.ext)" && pause && goto :load

no need for (  ) parentheses ??  confusing to see message ending in )



#30 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15871 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 12 September 2021 - 03:52 PM

I don't know. :unsure:

 

The:

File or device not found!

(need full device or file, i.e. (hd0)0+2 or (hd0)/myfile.ext)

 

shown in sequence come from a politically correct translation from the original :whistling::

 

WRONG! (you dumb@§§, go study dd syntax)

 

but since the translation didn't fit in one line I tried to keep unchanged the parenthetical structure of the original sentence :innocent: . 

 

Having it on two lines is on one hand better, as the first message tells what happened and the second tells the why (or if you prefer suggests a possible remedy) <- heck I did it again, but it is more prone to confusing the user..

 

Can we have everything on a single line?

 

Would something like:

File or device not found! Valid examples: (hd0)/myfile.ext or (hd0)0+2

 

be suitable and more understandable?

 

:duff:

Wonko



#31 steve6375

steve6375

    Platinum Member

  • Developer
  • 7453 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars, www.easy2boot.com
  •  
    United Kingdom

Posted 12 September 2021 - 03:57 PM

Maybe you could display all error messages in red on status lines at bottom of screen and just return to user input prompt immediately?

#32 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15871 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 12 September 2021 - 05:50 PM

Maybe you could display all error messages in red on status lines at bottom of screen and just return to user input prompt immediately?

Maybe. I'll think about it, the last four lines are the footer (now part of :header :w00t:) and I could separate and re-use them like I now do with :status (line 19)

 

@deomsh

Uploaded version 0.4 with skip and seek support for "large" files (i.e. larger than the limit I arbitrarily set at 2048 sectors).

 

The "Load with skip" is not an issue, in the worst case a stupidly large amount of sectors transferred will result in a long waiting time and in the grub4dos crashing having filled all memory.

 

But I am not at all convinced of the "Save with seek" part, an even small mistake will hose the file (or device) for good :ph34r:, and while when using device it is easier (for me) to visualize the size/offsets, with a file a mistake between - say - a disk or a partition/volume image is very likely to happen. :dubbio:

 

I wonder how it would be possible to make the seek choice "safer" :dubbio:

 

:duff:

Wonko



#33 steve6375

steve6375

    Platinum Member

  • Developer
  • 7453 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars, www.easy2boot.com
  •  
    United Kingdom

Posted 13 September 2021 - 07:43 AM

Backspace is in source file

#special provisions for dot, Del and Backspace
if %thiskey%==0x5300 call :comms "DR"
if %thiskey%==0xE08 call :comms "CE"

and if you looked at the code I modified and provided for you to compare, you would have noticed that I added a fix so that it would also work for grub4efi.

if %thiskey%==0x08 call :comms "CE"


#34 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15871 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 13 September 2021 - 12:22 PM

You see, if you had looked at the code before modifying it, the point is that there is no "CE" nor "DR" commands implemented so your "fix" will do nothing in the program run in grub4efi exactly as before, as it did (and still does) nothing in the program run in grub4dos.

 

We can agree that they are two different levels of doing nothing, stll the end result is the same.

 

Anyway the experience is good :) since it seems that we can draw a line *somewhere*, the point is that I don't know exactly where. :dubbio:

 

In grub4dos the pause --test-key always returns a two (or possibly more on "queer" keyboards) bytes scancode, no matter if the key has an ascii code defined or not

 

According to Atari800XL report (admittedly on a "queer" laptop keyboard) in grub4efi the pause --test-key  returns a two (or as seen more) bytes scancode, of which the 2 rightmost bytes are the same as the "normal" scancodes in grub4dos

 

According to your report (two examples, 0x0d [ENTER] and 0x08 [BACKSPACE] in grub4efi the pause --test-key  returns - when the key corresponds to a valid ASCII code - a single  bytes scancode, which is the same as the rightmost byte of the "normal" scancode in grub4dos.

 

But I presume that [Esc] escape ASCII 0x1B is actually "caught" by the current check in the program for 0x011B? Or did you need to change it?

 

So the general rule might be that if the rightmost byte is 00 then we need to use two bytes (&0xFFFF), otherwise we need to use only the rightmost byte (&0xFF) to have (hopefully) full compatibility, BUT what happens with 0x01 to 0x07 and the other non-printable characters?

 

Why does Escape seemingly behaves differently from Enter and Backspace?

 

:duff:

Wonko



#35 steve6375

steve6375

    Platinum Member

  • Developer
  • 7453 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars, www.easy2boot.com
  •  
    United Kingdom

Posted 13 September 2021 - 01:26 PM

I think it's because ESC is trapped by pause command and returns different status.
Efi kbd codes seem to be just 'different'
so best just to test for both.
Grub4efi has a nasty habit of returning 0xffffffff instead of 0xffffffffffffffff which is same as -1 in grub4dos.
I have raised issue about that but yaya ds not seem to realise the importance of compatibility between grub4dos and grub4efi.
Ps. Backspace was working under grub4efi on the code I offered

#36 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15871 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 13 September 2021 - 04:15 PM

I think it's because ESC is trapped by pause command and returns different status.
Efi kbd codes seem to be just 'different'
so best just to test for both.

I was trying to make a "rule" so that key presses can be detected in the same way on both.

 

Esc key (for *some reasons*) is 0x011b on both, but - according to the tentative line I drew - since it is also the same as 0x1b and 0x1b is greater than 0 testing it for ascii code only would work.

 

So Backspace Esc and Enter can be treated as printable characters and detectable by just the Ascii (scancode&0xFF).

For the 0-9 a-z A-Z (and other printable characters) they seem to be also detectable by just the Ascii (scancode&0xFF) on both, which reinforces the rule.

Arrow keys, Page Up/Down, Home/End, i.e. those that in grub4dos have the rightmost byte 00 can be detected on both by scancode&0xFFFF 

 

The question remains for the other possible key presses that have scancodes that do not end in 00 and are not among the above listed. 

 

:duff:

Wonko



#37 deomsh

deomsh

    Member

  • Advanced user
  • 98 posts
  •  
    Netherlands

Posted 13 September 2021 - 06:45 PM

@deomsh

Uploaded version 0.4 with skip and seek support for "large" files (i.e. larger than the limit I arbitrarily set at 2048 sectors).

 

The "Load with skip" is not an issue, in the worst case a stupidly large amount of sectors transferred will result in a long waiting time and in the grub4dos crashing having filled all memory.

 

Thanks a lot.

Goto-function is very nice! :rolleyes:

 

But if I skip for instance first 63 sectors and specify length of 10 sectors from my image-file, I get only ONE sector.

In cyan I see following just before Enter to confirm:

 

dd if=(hd0,0)/H33MF32.IMG of=(md)0x19000+1 bs=512 count=0x1 skip=0x3F

 

So somehow something is not good.

 

I will wait a bit until testing seek... :unsure:



#38 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15871 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 4 weeks ago

Yep, it is half-baked (and half-@§§ed), will be fixed in a next release.

In the new version I am putting together I am also limiting the [P] and the [G]oto to NOT go before the initial (md) address and using (flip on/flip off) the lines where the command hits are with a "main commands history".

I am thinking to limit the [S]ave to only the exact same address and extents from which the "file" has been [L]oaded, if it was loaded with skip, i.e. preventing the user from:
1) start saving extents from current sector (as opposed to the initial sector it was loaded to)
2) save with a seek that is different from the [L]oaded skip
and anyway add a warning if an attempt to [S]ave is done starting from an address that is not the initial one to which the file was loaded.

And - on second thought - a large file (if - as it should be - is contiguous in the filesystem) can be accessed (by the more advanced users) via blocklist, so we don't really-really need to use skip, a smaller "focused" number of sectors can be loaded as block once determined the blocklist of the file.

I.e. say you have a large disk image mydisk.img, and you want to view/edit its bootsector (at offset 2048 in the image).

You can:
blocklist (hd0,0)/mydisk.img
and get:
(hd0,1)456897+123589758
456897+2048=458945
and then you can [L]oad only:
(hd0,1)458945+1

It is a bit cumbersome, but I think it is safer, I will see if I can add a blocklist provision so that - for files that are contiguous - this part is automated.

Furthermore, but I don't like this approach as it will open any kind of possible problems (though greatly simplifying the program) there is nothing (in theory) preventing from directly editing disk sectors if we use cat --hex --length=256 (md)0x300+1 as opposed to echo --mem=0x60000=256, we can also have (say) cat --hex --length=256 (hd0,0)2048+1the only difference is in the offset on the left that is relative instead of absolute, not such an issue in itself.

As a side-side note, for memory there is also Fn.42 but in this version of grub4dos (and presumably on later ones) it behaves "strangely", some of my notes:

Quote

#42 hexdump – e.g. call Fn.42 0x8000 0 3 – list 3 hex bytes at 0x8000
#The above is NOT VALID anymore
#Syntax is now seemingly
#call Fn.42 <number1> <number2> <absolute start address> <length>
#where <number1> seems to be the 32 bit part (in hex) of the address to be shown in the output (or however the rightmost 8 characters, zero padded to 8 byte)
#and <number2> seems to be the rest to 64 bit part (in hex) of the address to be shown in the output (or however the rightmost 8 characters, zero padded to 1 byte)
#so, to actually get the same result of
#cat --hex --skip=0 --length=16 (md)0x300+1
#you need now
#call Fn.42 0 0 0x60000 16
#I cannot see any actual reason for this change, but we have an interesting side effect:
##1 we have an easy way to convert from decimal to hex AND to zero pad (leftside) hex numbers
#call Fn.42 10 0 0x60000 1 | set hex=
#set hex=%hex:~0,8%
#but outside of this, for normal use now we have also:
#echo --mem=0x60000=16
#which seems much more user friendly
##2 we can abuse the call parameters using hexadecimal labels (but its use doesn't seem to have any practical utility)

So
call Fn.42 0x60000 0 0x60000 256
has same output as:
map --mem=0x60000=256

:duff:
Wonko



#39 deomsh

deomsh

    Member

  • Advanced user
  • 98 posts
  •  
    Netherlands

Posted 4 weeks ago

Using blocklist is a nice idea, maybe you can use it too with cmp to verify the dd-written sectors. I believe dd doesn't do such a check.

I hope 'normal' seek will be still possible in next version, not all files will be contiguous (in case of images: can be still loaded with map --mem)

#40 steve6375

steve6375

    Platinum Member

  • Developer
  • 7453 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars, www.easy2boot.com
  •  
    United Kingdom

Posted 4 weeks ago

It seems Fn.42 changed in grub4dos 0.4.6a at some point.

grub4efi is based on old source files of grub4dos so it currently uses old parameter syntax

I have requested change to grub4efi

https://github.com/c...4dos/issues/334



#41 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15871 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 4 weeks ago

Using blocklist is a nice idea, maybe you can use it too with cmp to verify the dd-written sectors. I believe dd doesn't do such a check.

I tend to trust dd, but that could be an useful addition as an option, as always with this kind of stuff (including dd) it is always a matter of size of data, using cmp on a bunch of sectors takes no time, but when the number of sectors grows, it may become a time hog.

 

 

 

I hope 'normal' seek will be still possible in next version, not all files will be contiguous (in case of images: can be still loaded with map --mem)

So, you have an image that can be loaded with map --mem, which should be small enough to fit in memory and to be transferred in a reasonable time.

It would make more sense to actually map it in memory, edit it in memory and then dd it back integrally to device after the modifications. :dubbio:

 

While attempting to make the batch  besides (hopefully) working also actually useful, I found another (little or big) issue.

 

The virtual (or fake) clipboard as it is now (which actually is "the first three quarters of a dd command") can lead to a mess.

When you Cop[Y] a dd command with infile the chosen area of memory is created, then when you Pas[T]e the dd command is completed with the outfile and executed, so it is possible to:

1) [L]oad a file in memory

2) Cop[Y] a part of it

3) [L]oad a completely different file

4) Pas[T]e

BUT what will be pasted won't be what was originally selected, but rather the contents of the second loaded file at the address selected on the first file. :ph34r:

 

So I will have to add a "safety feature" that prevents the use of [L]oad if the (virtual) clipboard is in use, or, if you prefer limit the possibility of Cop[Y] within the same file.

 

I tried (and it actually seemingly works) to add a "real" clipboard by means of using the memory sector before the first one in use (now that I have prevented the possibility of the program to go before the first address initially loaded this is "safe"). 

 

But this clipboard is obviously limited in size to 512 bytes, this allows to Cop[Y] up to 512 bytes (or 1 sector) and being able to [L]oad another file and Pas[T]e it there.

 

As an example one could quickly load a MBR, Cop[Y] it, then load the initial part of a disk image or the first sectors of a (hdn) and Pas[T]e it.

 

Question is, what would be a "logical" and "useful" "max size" for this "persistent" clipboard?

Maybe 512 bytes or 1 sector is too little a size, but I don't want to make it too large, as it will write to "memory", I could :unsure: crc32 a number of sectors before the loaded address and make sure that the checksum is for 00 sectors, and zero out them back on (hopefully) clean exit, but it is not like the program can try a size, find that the memory is not 00, try a smaller size, etc., etc. and without this check there is the risk (if the user selects certain addresses at launch) to "touch" areas that should be left alone.

BTW the same possible issues may happen "forward", i.e. (md)0x300 or 0x60000 may be a too low address to start fiddling with, when loading a largish file/number of sectors.

To be useful on "old" hard disk, the max size could be 63 sectors (or first head), on GPT the canonical 34 sectors could be enough, all in all putting the limit to 100 sectors before (or 51200 bytes) could be enough, about the "forward" what is the lowest address that could be used with no grub4dos data after it? :dubbio:

 

I initially thought that (on the version I am testing) that the first address with all 00's following it was (md)0x800, but while testing I noticed that this area is already used by grub4dos when dd is used (it is its buffer):

"ADDR and SIZE are used for user-defined buffer. ADDR defaults at 1M, and size default at 64K"  

 

So one could use as default load address 0x800+0x80+0x<max clipboard size>, let's say (md)0x1000=0x200000 for a nice, round number and max_clipboard_size=0x780 sectors or 1920*512=983040 bytes (not nice nor round), or even go a bit higher, like - still say - (md)0x1200=0x240000 and max_clipboard_size=2432*512=1245184, which we could define as "something more than 1 M".

 

If there was *something* like a (md)nul device for dd one could re-use its buffer as "clipboard", but I think there is no way to dd to a nul device :unsure: 

 

Ideas?

Suggestions?

 

:duff:

Wonko



#42 steve6375

steve6375

    Platinum Member

  • Developer
  • 7453 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars, www.easy2boot.com
  •  
    United Kingdom

Posted 4 weeks ago

maybe just call it 'Copy' and ask for destination address too?

 

User presses 'Y'

'Source address (press H for current cursor address or P for clipboard area):' (user enters Y or P or address)
'Number of bytes : '  (user enters a number)
'Destination address (H=current cursor address or P for clipboard area):' (user enters H or address)

'Command = dd if=xxx of=yyy blah blah..   Press ENTER to execute command or ESC to abort :' 

 

 

Also Z could ask for a length parameter?

 

https://rmprepusb.co...dos-internals/ has some info on memory locations.

Suggested:

(md)0x200 - 0x227 (20K) this seems to be used by chenall, et al. and so is probably safe!

(md)0x300 - 0x3DF (114k) 384-600KB approx. (0x3E0 may be used for chainloading by grub4dos)

(md)0x1F40 - 0x3FFF(4M)I have used (md)0x3000-(md)0x3300 without problems

(md)0x9800 - 0xBFFF(5M) Safe if only using graphics mode 1024x768 or lower resolutions E2B used 0xA000+0xA0

(md)0xD800 - 0xFFFFF (can start from 0x9800 as long as not using higher Res than 1024x768)

(md)0x19000+ (50MB+)

As most systems will have over 64MB of memory, use memory after (md)0x20000 to be ultra safe!

 

 



#43 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15871 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 4 weeks ago

Naah, copy is only from current cursor address, and only goes to clipboard, otherwise it is dd by another name.

As well paste is only from clipboard and only goes to current cursor address, otherwise it is again dd.

 

Now, adding a (separate) dd-like option would probably be a nice addition.

 

Z can be as well made more complicated (BTW, I already did so in the unreleased version using Z to zero (asking the user) either current sector or clipboard(s)), but asking (optionally) the length (in sectors) is one thing (that seems like nice/useful), but - to make it "symmetrical" to the above, i.e. asking for both offset and length becomes Fn.24 by another name. :dubbio:

 

Yep, I have seen that info, thanks :) , but the questions  were:

1) what is a "reasonable" amount of maximum size for the (real.sector backed) clipboard?

2) what is a lowish start address that at this point I would hardcode? Or is the (md)0x20000 at the end of the day a "good enough for all" address?

This - since the (md)0x19000 is already usable would give us up to 0x20000-0x19000=0x7000 sectors for the clipboard, i.e. 28672*512=14680064 bytes which sounds way too much (but of course can be less than that).

 

I like 0x20000 as it is as well a nice, round number.

 

:duff:

Wonko



#44 deomsh

deomsh

    Member

  • Advanced user
  • 98 posts
  •  
    Netherlands

Posted 4 weeks ago

@Wonko,
About the dd-buffer: you can control buffer's start-address with 'buf=' and size with 'buflen='. With cmp I found errors after copying with dd in case of bs=1 and if count>buffer ('real' cut and copy/paste operations with skip/seek). I am not sure anymore about the default buffer in this case (I believe 64kb). BTW I was testing bigger files of about 140kb - with small files never problems

I believe your Paste-function consists of overwriting 'existing' bytes with bytes from your clipboard, so maybe less/ not dd-critical?

About size of the clipboard: 1MB should be enough for everyone, was said once...

#45 steve6375

steve6375

    Platinum Member

  • Developer
  • 7453 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars, www.easy2boot.com
  •  
    United Kingdom

Posted 4 weeks ago

This - since the (md)0x19000 is already usable would give us up to 0x20000-0x19000=0x7000 sectors for the clipboard, i.e. 28672*512=14680064 bytes which sounds way too much (but of course can be less than that).

 

I like 0x20000 as it is as well a nice, round number.

 

Seems reasonable

 

@deomsh - are you saying there is a bug in grub4dos? If so can you give an example.

 

P.S. When using commands like cmp, dd, etc, you need to precede it with raw so that grub4dos does not try to expand the bytes if it thinks they are in a compressed format.

 

Attached Thumbnails

  • dd_raw.JPG


#46 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15871 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 4 weeks ago

@deomsh - are you saying there is a bug in grub4dos? If so can you give an example.

 

P.S. When using commands like cmp, dd, etc, you need to precede it with raw so that grub4dos does not try to expand the bytes if it thinks they are in a compressed format.

 

Then probably everything is cool, I completely forgot :blush: to prepend "raw" to the dd commands inside the batch. (done now in the unreleased version I am working on).

 

If the issue was while testing the batch, then it can happen that part of a "normal" file is seen as gzipped (and thus expanded) by dd. 

 

:duff:

Wonko



#47 deomsh

deomsh

    Member

  • Advanced user
  • 98 posts
  •  
    Netherlands

Posted 4 weeks ago

@deomsh - are you saying there is a bug in grub4dos? If so can you give an example.

P.S. When using commands like cmp, dd, etc, you need to precede it with raw so that grub4dos does not try to expand the bytes if it thinks they are in a compressed format.

I use raw a lot (Jaclaz teached me on MSFN), but I don't believe it's related to the issue I mentioned. I don't have any opinion about 'bugs' in Grub4Dos. I only report my experimental findings.

Although I already gave an example, of case you want an example that can be reproduced.
I played a bit on my smartphone, with following results, can be seen as converted print-screens below.

First print-screen is build from 2 print-screens. This is the example where things are going 'bad'.

But remember: Text-file (so no header) is BIG (0x24FD3), and bs=1 (must be, because of using dd-read/ write on byte-level). In this experiment dd-buffer is default, I believe indeed 64KB (0x10000 isn't it?).

This is an example of a copy/ paste operation I use in my line-editor script (in a nutshell and simplified to make it possible to reproduce - in fact I don't use write for this operation).

So after comparing two identical text-files (only extension is different, I cleaned some memory (511 sector is my experimental-found limit of echo -n > FILE). After copying file to memory-region (bs is default - so one sector, no problems), I make some 'room' to copy the line 'echo' including MS-DOS EOL (bs=1). Everything LOOKS fine, see cat --hex lines.

Second I cut 'echo+EOL', everything SEEMS to be fine, first line restored as in the beginning. But after copying memory back to file, cmp says things are NOT fine anymore.

This behavior started while I was testing BIG text-files (in fact I was editing FATTEXT.G4B on another disk with the FATTEXT.G4B-script). Before I was testing SMALL files, never (such) problems.

Limbo x86 PC Emulator

grub> cmp /fattext.g4b /fattext.txt
grub> echo -n > (md)0x3000+511
grub> raw dd if=(hd0,0)/fattext.txt of =(md)0x3000+511
buf_size=0x10000, loops=0x3. in_pos=0x0, out_pos=0x0
0000000 00000001 00000002
Bytes read / written = 0x24FD3 / 0x24FD3
grub> raw dd if=(md)0x3000+511 of=(md)0x3000+511 bs=1 skip=6 seek=12
buf_size=0x10000, loops=0x4. in_pos=0x6, out_pos=0xC
00000000 00000001 00000002 00000003
Bytes read / written = 0x3FDFA / 0x3FDF4
grub> cat --hex --length=16 (md)0x3000+511
00000000: 21 42 41 54  0D 0A 23 23  46 41 54 54  23 23 46 41 ; !BAT..##FATT##FA
grub> raw write --offset=6 (md)0x3000+511 echo\x0D\x0A
0x6 bytes written at offset 0x6.
grub> cat --hex --length=16 (md)0x3000+511
00000000: 21 42 41 54  0D 0A 65 63  68 6F 0D 0A  23 23 46 41 ; !BAT..echo..##FA
grub> raw dd if=(md)0x3000+511 of=(md)0x3000+511 bs=1 skip=12 seek=6
buf_size=0x10000, loops=0x4. in_pos=0xC, out_pos=0x6
00000000 00000001 00000002 00000003
Bytes read / written = 0x3FDF4 / 0x3FDF4
grub> cat --hex --length=16 (md) 0x3000+511
00000000: 21 42 41 54  0D 0A 23 23  46 41 54 54  45 58 54 2E ; !BAT..##FATTEXT.
grub> raw dd if=(md)0x3000+511 of=(hd0,0)/fattext.txt
buf_size=0x10000, loops=0x3. in_pos=0x0, out_pos=0x0
00000000 00000001 00000002
Bytes read / written = 0x30000 / 0x24FD3
grub> cmp /fattext.g4b /fattext.txt
Differ at the offset 65542: 0x69 [/fattext.g4b], 0x22 [/fattext.txt ]
grub> cat --hex --length=16 /fattext.g4b
00000000: 21 42 41 54  0D 0A 23 23  46 41 54 54  45 58 54 2E ; !BAT..##FATTEXT.
grub> cat --hex --length=16 /fattext.txt
00000000: 21 42 41 54  0D 0A 23 23  46 41 54 54  45 58 54 2E ; !BAT..##FATTEXT.
grub> cat --hex --skip=65530 --length=32 /fattext.txt
0000FFFA: 65 6E 67 74  68 25 22 20  22 25 73 6B  22 20 22 25 ; ength%." "%sk" "/
0001000A: 73 6B 69 66  20 25 65 64  3D 43 20 26  69 74 25 3D: skif %edit%==C &
grub> cat --hex --skip=65530 --length=32 /fattext.g4b
0000FFFA: 65 6E 67 74  68 25 22 20  22 25 73 6B  69 70 25 22 ; ength%" "%skip%"
0001000A: 0D 0A 69 66  20 25 65 64  69 74 25 3D  3D 43 20 26 ; ..if %edit%==C &
grub>

Maybe someone can explain to me the output of cmp: Differ at the offset 65542: 0x69 [/fattext.g4b], 0x22 [/fattext.txt ] ??
First I thought my script was 'bad' (true) - but found no explanation. Later I saw the region involved, around 64KB, so on the boundary of the default dd-buffersize.


Next experiment uses dd's buf and buflen - see second print-screen.

After producing an identical copy with only different extension (see first check below with cmp)
For sake of length of one screen-line I introduced variable %mem%.
Further same proceedings, WITH EXCEPTION of buf (0x3200, so just above 0x31FF) AND buflen set to a four times bigger buffer (AND bs=1 for sake of experiment).
I skipped the cat part to stay on one screen, and cmp said PASS this time.

 

EDIT: OOPS, forget the line with write, so experiment repeated and new second  'print-screen' below  (without line with write shouldn't make any difference anyway, didn't indeed... sorry without raw write - but all raw dd as before)

Limbo x86 PC Emulator

grub> fat copy /o /fattext.g4b /fattext.txt
grub> cmp /fattext.g4b /fattext.txt
grub> raw dd if=(hd0,0)/fattext.txt of=%mem% bs=1 buf=0x640000 buflen=0x40000
buf_size=0x40000, loops=0x1. in_pos=0x0, out_pos=0x0
00000000
Bytes read / written = 0x24FD3 / 0x24FD3
grub> raw dd if=%mem% of=%mem% bs=1 skip=6 seek=12 buf=0x640000 buflen=0x40000
buf_size=0x40000, loops=0x1. in_pos=0x6, out_pos=0xC
00000000
Bytes read / written = 0x3FDFA / 0x3FDF4
grub> write --offset=6 %mem% echo\x0D\x0A
0x6 bytes written at offset 0x6.
grub> raw dd if=%mem% of=%mem% bs=1 skip=12 seek=6 buf=0x640000 buflen=0x40000
buf_size=0x40000, loops=0x1. in_pos=0xC, out_pos=0x6
00000000
Bytes read / written = 0x3FDF4 / 0x3FDF4
grub> raw dd if=(hd0,0)/fattext.txt of=%mem% bs=1 buf=0x640000 buflen=0x4000
buf_size=0x40000, loops=0x1. in_pos=0x0, out_pos=0x0
00000000
Bytes read / written = 0x24FD3 / 0x24FD3
grub> cmp /fattext.g4b /fattext.txt
grub>

@steve6375 is this what you meant with an example?
In my humble opinion anyone can try to reproduce these findings, so have fun.
Maybe you have an explanation?

Personal note: for me these misery was a bad period, I almost gave up my whole project. Luckily everything seems fine now (although I have also scratch-memory in use to write too, if seek is used - and coping everything 'back' in one run, which seems to be SUPER-safe - to me). But in simple cases like inserting 0D in case of an unix text-file, or deleting 0D to convert back, I auto-choose buflen > filesize (in memory).

BTW I used Grub4Dos version 20210516.
BTW2 count is not used in examples above, filesize of fattext.txt limits final result.

I still do not understand if my problems as described will originate during skip < seek (so to make 'room' for copying a line, so inserting), or the reverse if skip > seek (so cutting bytes out) Or both?



#48 steve6375

steve6375

    Platinum Member

  • Developer
  • 7453 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars, www.easy2boot.com
  •  
    United Kingdom

Posted 4 weeks ago

seems like a bug - I have reported it

https://github.com/c...4dos/issues/336



#49 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15871 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 4 weeks ago

Maybe the issue is because you have if and of on the same area and the buffer *somehow* gets into play? :unsure:

 

I mean, what happens if - while still within a sensible size - you "make a new copy" by dd-ing to a higher memory address and then dd it  back?

 

Using Steve's bug report code:

 

!BAT
raw cmp /f1 /f2 && echo F1 is SAME as F2
raw dd if=(hd0,0)/f2 of=(md)0x3000+511
raw dd if=(md)0x3000+511 of=(md)0x3000+511 bs=1 skip=6 seek=12 raw dd if=(md)0x3000+511 of=(md)0x20000+511 bs=1 skip=6 seek=12
raw dd if=(md)0x3000+511 of=(md)0x3000+511 bs=1 skip=12 seek=6 raw dd if=(md)0x20000+511 of=(md)0x3000+511 bs=1 skip=12 seek=6
raw dd if=(md)0x3000+511 of=(hd0,0)/f2
raw cmp /f1 /f2 && echo F1 is SAME as F2

 

does the same happen?

 

:duff:

Wonko



#50 steve6375

steve6375

    Platinum Member

  • Developer
  • 7453 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars, www.easy2boot.com
  •  
    United Kingdom

Posted 4 weeks ago

!BAT
raw cmp /f1 /f2 && echo F1 is SAME as F2
raw dd if=(hd0,0)/f2 of=(md)0x3000+511
raw dd if=(md)0x3000+511 of=(md)0x20000+511 bs=1 skip=6 seek=12
raw dd if=(md)0x20000+511 of=(md)0x3000+511 bs=1 skip=12 seek=6
raw dd if=(md)0x3000+511 of=(hd0,0)/f2
raw cmp /f1 /f2 && echo F1 is SAME as F2

That works

So must be way that dd is copying memory?

Attached Thumbnails

  • test2.JPG





4 user(s) are reading this topic

0 members, 4 guests, 0 anonymous users