Jump to content











Photo
- - - - -

[RELEASE] idd.g4b interactive dd command composer


  • Please log in to reply
59 replies to this topic

#51 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 26 October 2021 - 09:29 PM

 

But in the post you linked to, it seems two fragments can be too much already. So not all fragments are equal in this respect.

Maybe there is a misunderstanding.

At the time of the mentioned post even 2 fragments meant that you had to map with --mem, and, for the specific case of large img on small NTFS we tested successfully the workaround of making a very small NTFS volume and then enlarging its filesystem.[1]

 

Some time later someone (I believe karyonix) made a modification to the grub4dos (that became official) to allow mapping fragmented files.

 

I never tested it (as I try to have actual images contiguous), but suspected that the number of fragments allowed by this modification were a handful (which you just confirmed experimentally amount to 4).

 

How many fragments (on average) fit in the 1200 characters limit you just found?

If we say that - even on a largish disk - a fragment (including the separating comma) is within the 12 characters, this makes "up to 100 fragments roughly" possible.

All in all, a file with more than 100 fragments (or so) should (hopefully) be very, very, very rare.

 

BUT (stupid idea maybe).

 

Is it possible to write to (md) the list of blocks then prepend to it the map command, append to it the (rd) and then copy/dd the whole stuff to cmd buffer?

This way no variables would be needed (for the fragments).

 

Still, I suspect that the gain would be minimal, because of the "other" 1535/6 limit on command line.

 

The "proper" way is most probably mapping each fragment (or - say - sets of 10 fragments) to a (rd) moving each time the rd_addr, then finally move back the rd_addr to first block and set rd_size to the whole file size.

 

:duff:

Wonko

 

[1] which remains a nice trick for a few other things, like using truncated images.



#52 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 27 October 2021 - 04:25 PM

Thanks for clarification of fragments with older grub4dos versions.

 

My blocklist (3x %v% !) contains 128 fragments, one 1+1 fragment added.

 

Nice idea to omit the pipe-to-variables-part, but I think messing around with the cmd buffer is not needed (did you actually ment the History Buffer?). Just making a batch-file in memory and call it.

Procedure is in the print-screen

Screenshot_20211027-133049.png Screenshot_20211027-154756.png

 

BTW crc32-line for presentation only.

BTW2 blocklist happily listed my vars (don't have a very fragmented file), same limit of 1200 chars

BTW3 I added redirect to file only to save the result.



#53 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 27 October 2021 - 06:16 PM

Yep, I meant the cmd history buffer, but it was only to about having not the need to choose an arbitrary address in memory (such as the (md)0+3000+6 you used in your - nice BTW :) - test).

 

Otherwise, starting from a "clean" rd_base=0x0 and rd_size=0x100000000 one could map to (rd) *any* 6 sectors, and thus have a "grub4dos managed address", then write to the area the batch with the blocklist command adding to it a commadn to shift the rd_base of 6 sectors  Has to be tested but the batch should be loaded fully in memory for execution, so you can actually in grub4dos "pull the carpet under its feet" :unsure:

 

I thought that the 1200 characters was a limit *somewhere else* not on the actual output of the blocklist command. :(

 

The other idea of the incremental mapping, on second thought, would probably be slower (due to the loops in the batch) and provide no advantage if the limit is in the output of blocklist.

 

:duff:

Wonko



#54 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 27 October 2021 - 08:02 PM

BTW I didn't mean OUTPUT of the blocklist command. INPUT in case of  %device%%%path%%%file% will never exceed 1200 chars I presume.

But mapping a blocklist at once will be  still limited to 1200 chars.

 

BTW2 using some (md)-address was part of you're own idea. Nevermind, be my guest if you want to parse al fragments one by one (if I did understand you well).

 

I tried already mapping to (rd) with pre-defined base and size, but it seems map ignores this, and choose --rd-base freely, *somehow* dependent on block-file-size.

 

If I pre-define a rd as number with map --ram-drive=0 (254 seems to be max) and map to (0), this behavior of map doesn't change.

BTW in my version of steve6375's grub4dos-book he gave the number 0xFF7F - doesn't work for me).

 

If map cannot be tamed, 'merging' of rd's looks problematic. Or did I something wrong? Hopefully...

Screenshot_20211027-215307.png

 

If I map a second file after defining rd (1) with another base-address, I cannot go back to (0), memory from 0x4000000 is still all zero's and 0x5000000 too (this is not in the print-screen below!).

Screenshot_20211027-223031.png

BTW watch the crc32 of (0)+1

 

With dd no problems, although I can only access one rd at the time.

Screenshot_20211027-225451.png



#55 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 28 October 2021 - 09:11 AM

The 0XFF7F must be a typo.

 

The way drives are numbered is complex, every single time I have to re-read the (scarce) docs, and hope I get it right.

 Drive (0) is AFAICT Drive (0x00) aka (fd0).

This should be the relevant snippet (on Steve6375's page https://rmprepusb.co...grub4dos/021b/ 

 

 

Named in digital equipment:

fd0 – fd126: 0x0 – 0x7E (virtual floppy diskettes and exclude 0x21 and 0x23)

pxe: 0x21 (network boot device)

ud: 0x23 (Fbinst hidden partition)

rd: 0x7F (random memory storage drive)

hd0 – hd30: 0x80 – 0x9E (hard disk and virtual hard disks)

cd: 0x9F (captured bootable CD, this is not necessarily correct, depending on the bios may be)

hd32 – hd127: 0xA0 – 0xFF (CD and virtual CD-ROM)

BUT (I never managed to understand this) the ram_drive can not be 0xff because 0xFF is used for mapping --mem drives (or *whatever*), cannot find right now when/where I read this, but this should be the reason why you cannot use 0xFF for (rd), i.e. the limit is 0xFE or 254.

If you do (say):

map --mem (fd0) (fd1)

map --status

in the drive table (that I still do not understand)  you will find From=01 and To=FF

 

Anyway, it seems that this is another show-stopper, as you found out, it seems that the rd-base and rd-size are overwritten by a following map command, while I can understand how rd-size is auto-adjusted, why is the rd-base changed?. :(

 

:duff:

Wonko



#56 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 29 October 2021 - 02:31 AM

The map-table is indeed strange. I would expect 'From=FF' 'To=01'

 

About the 'shifting' rd_base: my theory is because of the 'implied' --mem. Will shift the mapping to the highest possible memory address, given rd_size. And because (rd) isn't write-protected, an earlier (rd) will be ignored.

 

I did some experiments which illustrate this. As soon (rd)+1 is mapped to an int13-device memory is protected. Next (rd) will have lower rd_base.

Screenshot_20211029-014506.png

 

BTW map --hook is not needed to protect memory. Of cause there is no access to drive 0x2 in this case.

 

So it should be possible to map individual, or a couple of fragments in revererse order, last one with the n+1 trick and then resize (rd).

Screenshot_20211028-205008.png Screenshot_20211028-205422.png

 

BTW strange is that FIRST MAPPED one-sector fragment needs 8 sectors. In this case not disturbing of course, only first sector is inside rd_size. But I created my 'blocklist' by adding the blocks of my 'most' fragmented file twice and 'round' at last to 25+1 to get 400 chars. On my 2GB FAT16 drive minimum is 64 sectors because of 32KB cluster-size. So with minimum of 4KB clusters there should be no issue. Only n+1 could be better n+8?



#57 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 29 October 2021 - 12:48 PM

The way the map table works is AFAICU is a reversed lookup, i.e. you want to find drive (fd2) (or (0x2) or (2)) and you lookup for it, in the "From", when you find it in the table, you discover that it is mapped to (0xFF).

 

What I don't understand (more experiments to do :() is how the rest of the table works, i.e. what most of the other columns mean remain a mistery to me.

 

The start sectors of subsequent device mappings seem like being in reverse order, going down.

 

In your first attached image, you have rd_base=0xBA60000 and rd_size=0x2D0000  (as the (fd2) later mapped, 0x1680 sectors or 0x1680*0x200=0x2D0000, sounds good).

0xBA60000+0x2D0000=0xBD30000

0xBD30000/0x200=0x5E980

The (fd2) mapping has start sector 0x5E980, good, the second mapping, the (fd2) one is appended at the end of the first (rd) mapping.

 

In the other two images you map in sequence to (rd) and then to (2), (3) and (4), but the start sectors are decreasing. :dubbio:

 

If this is "normal" (i.e. you have to map from the bottom up) at a certain point the remaining area might be too small, what will happen then, another error 28 ? :unsure:

 

The only thing tha<t is clear to me is that there is *something* that is working "behind the hood" to manage these mappings, that *whatever it is* it works fine in normal use, but totally escapes me how it actually works.

 

Try this batch:

 

 

!BAT
#selfmap.g4b
debug msg=0

blocklist %~dpnx0
cat --length=0 %~dpnx0
set /A filesize=*0x8290 > nul
set /A filesect=%filesize% / 512 > nul
set /A filerest=%filesize% % 512 > nul
checkrange 0 calc %filerest% > nul || set /A filesect=%filesect%+1 > nul
set filesect
echo

map %~dpnx0 (rd) > nul
call :check_rd

map (hd0)1+1000 (rd)
call :check_rd

map --mem %~dpnx0 (fd2) > nul
call :check_rd
map --hook
call :check_rd

map --mem %~dpnx0 (fd3) > nul
call :check_rd
map --hook
call :check_rd

map --mem %~dpnx0 (fd2) > nul
call :check_rd
map --hook
call :check_rd

map --mem %~dpnx0 (fd4) > nul
call :check_rd
map --hook
call :check_rd

map --unhook
map --unmap=0x02:0xFF
call :check_rd
debug msg=3
goto :eof

:check_rd
#we use sector 0x20000+2 aka 0x4000000 as storage for the map --status output
call Fn.24 0x4000000 0x00 0x200
map --status > (md)0x20000+2
cat --skip=0x75 (md)0x20000+2
#0000:82CC 4 (DWORD) rdnum - rd device number
#0000:82D0 8 (QWORD) rdbase - sector start address of rd <- NO, it is address in bytes
#0000:82D8 8 (QWORD) rdsize - max. size in bytes of rd
#set /A rd_dnum=*0x82CC&0xFF > nul
set /A rd_base=*0x82D0 > nul
set /A rd_bsect=*0x82D0/0x200 > nul
set /A rd_size=*0x82D8 > nul
set /A rd_esect=*0x82D0+*0x82D8/0x200 > nul

echo $[0x0F]rd_base=%rd_base%, rd_size=%rd_size% rd_bsect=%rd_bsect% rd_esect=%rd_esect%
pause
goto :eof

 

 

:duff:

Wonko 



#58 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 30 October 2021 - 12:51 AM

Hmm. 'To' and 'From': I mapped (rd) == FF I think to a floppie, so do I read the reverse order.

 

In the last collomn is following clear to me: if mapped only, in the middle is '<', after map --hook becomes '=' and if 'overmapped' a '>' and after map --hook gone.

 

I have run your script, same as my theory. Also for instance map --mem %~dpnx0 (fd2) tries to find highest available memory area, next rd_base AFTER mapping to (rd) shifts down. Don't know what happens if there is no free memory anymore DOWNWARDS. Error 28 sounds logical. 

 

But as soon drives are unmapped next mapping to (rd) will search for highest available memory (addresses). This is not tested in your script.

Screenshot_20211029-222039.png

 

BTW it seems that rd_base is always a multiple of 0x1000, so 8 sectors?

 

BTW2 I do not understand what you mean with 'map from the bottom up'. If you map --mem twice to some floppie and unmap the first one, their is a some free memory above free. I think if mapping to (rd) 'fits', this area will be 'taken', otherwise the area below. In my theory you can only map --mem from the top down. So far all experiments confirm this. Or am I wrong?



#59 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 30 October 2021 - 08:52 AM

In the last collomn is following clear to me: if mapped only, in the middle is '<', after map --hook becomes '=' and if 'overmapped' a '>' and after map --hook gone.

 

Sure :), you decoded the "H" in DHR.

What is D?

What is R?

What is Hm?

What is Sm?

etc., etc.

I know nothing about the actual meaning of the columns, the only things that I understand (somehow and not fully) are the Fr,To, Start sector, Sector_count, and now I have your observation (similar to mine) on the behaviour of <>= in the H in DHR.

 

Why the FF drive seen is the mapping table represent (maybe) the (md) (and not the 0xFF drive, which is a cd emulation device)?

geometry (md)

returns drive 0xFFFF

and you can actually map to (0xFFFF) and in the map table it comes out as FF.

so (md) is drive 0xFFFF, the reference to 0xF7FF might then not be after all a simple typo, but *something else*.

 

The more time I read the (only) documentation available (on Steve6375's pages  :worship: ) the less I understand.

 

Anyway, though I still do not understand most of what  I am doing :w00t: the attached is semi-functional to map to drives in the "normal" order. (it seems to work, buit is not as clean as it should be on exit).

 

Whether it will be useful for anything, when (and if) it will work fully, is another thing.

 

 

:duff:

Wonko

Attached Files



#60 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 30 October 2021 - 05:38 PM

Maybe following fragment from source code is helpfull (buitins.c).

BTW I searched for 'DHR'

//struct drive_map_slot
//{
	/* Remember to update DRIVE_MAP_SLOT_SIZE once this is modified.
	 * The struct size must be a multiple of 4.
	 */

	  /* X=max_sector bit 7: read only or fake write */
	  /* Y=to_sector  bit 6: safe boot or fake write */
	  /* ------------------------------------------- */
	  /* X Y: meaning of restrictions imposed on map */
	  /* ------------------------------------------- */
	  /* 1 1: read only=0, fake write=1, safe boot=0 */
	  /* 1 0: read only=1, fake write=0, safe boot=0 */
	  /* 0 1: read only=0, fake write=0, safe boot=1 */
	  /* 0 0: read only=0, fake write=0, safe boot=0 */

//	unsigned char from_drive;
//	unsigned char to_drive;		/* 0xFF indicates a memdrive */
//	unsigned char max_head;
//	unsigned char max_sector;	/* bit 7: read only */
					/* bit 6: disable lba */

//	unsigned short to_cylinder;	/* max cylinder of the TO drive */
					/* bit 15:  TO  drive support LBA */
					/* bit 14:  TO  drive is CDROM(with big 2048-byte sector) */
					/* bit 13: FROM drive is CDROM(with big 2048-byte sector) */
					/* bit 12:  TO  drive is BIFURCATE */
					/* bit 11:  TO  drive has a known boot sector type */
					/* bit 10:  TO  drive has Fragment */
					
//	unsigned char to_head;		/* max head of the TO drive */
//	unsigned char to_sector;	/* max sector of the TO drive */
					/* bit 7: in-situ */
					/* bit 6: fake-write or safe-boot */

//	unsigned long start_sector;
//	unsigned long start_sector_hi;	/* hi dword of the 64-bit value */
//	unsigned long sector_count;
//	unsigned long sector_count_hi;	/* hi dword of the 64-bit value */
//};

	/* From To MaxHead MaxSector ToMaxCylinder ToMaxHead ToMaxSector StartLBA_lo StartLBA_hi Sector_count_lo Sector_count_hi Hook Type */
//	if (debug > 0)
	  grub_printf ("\nFr To Hm Sm To_C _H _S   Start_Sector     Sector_Count   DHR"
		       "\n-- -- -- -- ---- -- -- ---------------- ---------------- ---\n");
	if (! unset_int13_handler (1))
	  for (i = 0; i < DRIVE_MAP_SIZE; i++)
	    {
		if (drive_map_slot_empty (hooked_drive_map[i]))
			break;
		for (j = 0; j < DRIVE_MAP_SIZE; j++)
		  {
		    if (drive_map_slot_equal(bios_drive_map[j], hooked_drive_map[i]))
			break;
		  }



#61 deomsh

deomsh

    Frequent Member

  • Advanced user
  • 196 posts
  •  
    Netherlands

Posted 22 July 2023 - 12:04 PM

I think I found a more universal solution for the 'verify' problem.
Recently I (re?)discovered grub4dos command cmp takes --skip=bytes too. But is the same for FILE1 and FILE2.

With help of a rd it's seems to be possible verify dd with different skip and seek and not aligned to blocks.
The idea is to create (rd) and shift the base/ resize, if needed use cmp --skip=n FILE (rd)+1
File sizes plus extra size needed should be less than available memory.

Steps:
1) Get size of FILE1 and FILE2
2) Make rd with a suitable base, and size of FILE2
3) dd FILE2 to (rd)+1 and compare
4) Calculate DIFFERENCE of skip and seek, change rd-base accordingly and set rd-size to size of FILE1
5) dd FILE1 to (rd)+1 with bs=1 skip=skip seek=skip and compare with --skip=skip
6) Reset rd-base and set rd-size to size of FILE2 again
7) dd FILE2 to (rd)+1 with bs=1 skip=seek+count seek=seek+count and compare with --skip=seek+count
8) dd (rd)+1 to FILE2 and compare

Example on print-screen:

dd if=S1 of=S2 bs=1 count=21 skip=36 seek=86 & cmp (rd) trick II.png

BTW if FILE2 is fully overwritten after seek step 7 can be left out.






1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users