Jump to content











Photo
- - - - -

[Release] GPTcrc32.g4b to (re-)calculate CRC32 hashes in GPT header


  • Please log in to reply
36 replies to this topic

#1 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15888 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 01 September 2021 - 12:54 PM

For no apparent reason [1] a small grub4dos batch to (re-)calculate the two CRC32 hashes in the Header of GPT style disks.

 

The first (please read as second) CRC32 hash is of the partition entires array (normally 128*128 bytes each=16.384 bytes, i.e.  32 sectors, LBA 2-33).

 

The second (please read as first) CRC32 hash is that of the GPT header (normally 92 bytes long) and includes the previous (please read as later) partition entries array hash.

 

Contrary to my expectations making a working CRC32 algorithm in grub4dos batch wasn't too difficult, the creation of the CRC32 table is fast, but the parsing of the 32 sectors is slowish.

 

The inner CRC32 routine could be easily re-used if CRC32 calculation is needed somewhere else.

 

Briefly tested on grub4dos (NOT grub4EFI) 0.4.6a 2021-05-16, while it should work on grub4EFI also, it needs to be tested there.

 

:duff:

Wonko

 

 

 

[1] if not this one here:

http://reboot.pro/in...=22544&p=219264

Attached Files


Edited by Wonko the Sane, 04 September 2021 - 01:34 PM.
Updated script to use internal CRC32 function

  • Atari800XL likes this

#2 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 191 posts
  •  
    Netherlands

Posted 02 September 2021 - 05:14 PM

Tested on DELL Inspiron 3531 (same as "the other" thread):

 

- First part working OK, displays info until "Everything seems cool" and "Press a key to continue ...

- After a few (5?) seconds, clears screen and says "Please wait ... calculating CRC32"

- After quite a few seconds (400) the second screen is displayed (with the original and calculated hashes).

 

I must admit, I almost gave up, but then I remembered you calling it "slowish" :-)

 

Nice proof of concept, amazing that you whipped up that code so quickly!

 

.



#3 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15888 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 02 September 2021 - 06:19 PM

I have no idea why it takes that much time.

 

I can only test in a Virtualbox, and while admittedly slowish, here up to "Everything seems cool ..." has a (minimal) lag, but then it takes only like 3-4 seconds to get the CRC32.

 

It must be the cat (actually echo --mem but catting from memory should be extremely fast anyway). :unsure:

 

You could try to insmod the script, maybe the problem are the loops with the script on a physical device.  :dubbio:.

 

I don't think (but I did not test it) if using read or the direct set from memory would be faster than cat.

 

I could cheat and ignore the sectors that are empty (after all who actually has 128 partitions?) in most cases only 1, 2 or 3 sectors (i.e. 4/8/12 partitons) are actually used, using a pre-computed 512 00's bytes for them.

 

I'll have a look at this, when you have time, try with insmod.

 

:duff:

Wonko



#4 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 191 posts
  •  
    Netherlands

Posted 02 September 2021 - 07:00 PM

I inserted "insmod" into the script.

Part 1 is almost instant, then after 17 seconds:

Please wait ... calculating CRC32
echo --mem=%b_offset%=1 | call :line_to_crc
call :crc_loop %crc_max%

 



#5 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15888 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 03 September 2021 - 08:41 AM

 

I inserted "insmod" into the script.

I am not sure to understand, I meant "try loading the script with insmod as module", i.e. on command line:

 

insmod (hd1,2)/grubfiles/GPTcrc32.g4b

 

then run the script without its path:

GPTcrc32.g4b (hd1)0+34

 

To unload the module use:

delmod GPTcrc32.g4b

 

There was a similar issue that Steve6375 reported about another old batch:

http://reboot.pro/in...=17728&p=162722

 

the issue at the time was about WENV (that in the original batch was used for most of the loops), I suspect that the issue might have the same origin, i.e. the script is continuously re-parsed and being on slow (relatively) physical media it wastes a lot of time, alternatively, use a floppy image and map it to memory, as seen on the mentioned thread. .

 

:duff:

Wonko



#6 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 191 posts
  •  
    Netherlands

Posted 03 September 2021 - 11:39 AM

I tested with a floppy image, but unfortunately this didn't fix the long delay.



#7 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 191 posts
  •  
    Netherlands

Posted 03 September 2021 - 11:51 AM

I'm not actively following Grub4Dos development, but what do you think would be the chances of something like this being included natively in the code?



#8 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15888 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 03 September 2021 - 12:49 PM

I tested with a floppy image, but unfortunately this didn't fix the long delay.

Why not the insmod? :dubbio:

 

And HOW EXACTLY did you test the floppy image? :unsure:

 

 

I'm not actively following Grub4Dos development, but what do you think would be the chances of something like this being included natively in the code?

Let's go step by step.

 

IF (even if now it takes a long time to compute) the approach works, then we will find a way (one way or the other) to make it faster, it is entirely possible that the way I implemented the algorithms sucks (and sucks BIG) or simply that the batch mechanism is simply too slow, the same thing done as a native grub4dos program will surely be much faster (it should be practically instantaneous).

 

Looking around for info on the matter it seems, see here:

https://github.com/p...ufus/issues/954

 

BTW and OT kudos to Rufus' user JackTrapper, he made a wonderfully detailed bug report, something that is extremely rare. 

 

that as expected each and every programmer looked at the (stupid) UEFI/GPT specs and read them differently.

 

I.e. it seems that Windows tools (but it needs to be confirmed) write a "default" number of 128 "possible" partitions (of 128 bytes each) in the EFI PART sector, no matter how many partitions are actually used (this actually makes sense as they reserve space for more partition entries), see also:

 

whilst other OS's (in the given link OpenBSD) seem like "closing" the image with the actual number of partition entries used.

 

Moreover seemingly (again from the given OpenBSD example) the partition entry array checksum is either that of the exact number of valid partitions[1]:

(Number of partition entries in array)*(Size of a single partition entry)

or that of the same BUT rounded to a whole sector:

roundup((Number of partition entries in array)*(Size of a single partition entry), 512)

 

Now, the general idea (for the future) is to be able to manipulate this field:

(Number of partition entries in array)

 

lowering it to the actual number of partitions in use (let's say first 4 partitions for the sake of the example), so that anything from the 5th partition onwards is effectively out of the partition table array (and as such really-really hidden).

 

The limitation will be that that partitions to be hidden needs to be in the higher # of partition slots, but that should be possible as I don't see anywhere that the partitions need to be in the partition table in the same order they are on disk (just like they can be so on MBR, notwithstanding a numebr of tools - more royalist than the king himself - that might tag this as an issue).

 

Also it is not clear (to me) how different OS's validate (or ignore) the relationship between 

(Number of partition entries in array)*(Size of a single partition entry)

and

(First usable LBA for partitions (primary partition table last LBA + 1))

 

and how the above exactly affects the GPT backup at the end of the disk.

 

WHEN (IF) we can succeed in understanding the above, THEN we will talk about convincing (or bribing ;) ) a "real" programmer into writing a "proper" grub4dos/grub4EFI module.

 

:duff:

Wonko

 

 

 

[1] for the sake of clarity I am using the definitions of fields in the Wikipedia page:

https://en.wikipedia..._header_(LBA_1)



#9 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 191 posts
  •  
    Netherlands

Posted 03 September 2021 - 01:08 PM

Sorry, Wonko, I **did** test the insmod method as well!!!

 

....more later...

 

EDIT: Further tests not necessary after Steve's hint about the built-in "crc" command...



#10 steve6375

steve6375

    Platinum Member

  • Developer
  • 7453 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars, www.easy2boot.com
  •  
    United Kingdom

Posted 04 September 2021 - 09:34 AM

Maybe you could make use of the crc32 command that is already in grub4dos and grub4efi?


Edited by steve6375, 04 September 2021 - 09:41 AM.

  • Atari800XL likes this

#11 steve6375

steve6375

    Platinum Member

  • Developer
  • 7453 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars, www.easy2boot.com
  •  
    United Kingdom

Posted 04 September 2021 - 10:19 AM

P.S. crc32 (hd0)0+1,20 will calculate crc32 for first 20 bytes in LBA0. crc32 (hd1)2+32 returns same result as your calc

 

or you can use dd to copy bytes to a memory device (md) and then calculate crc32 of (md) device.

 

P.S. crc32 (md)0x303+1,92  also works in your batch file for GPT Header where 92 = crc_max+1.

Attached Thumbnails

  • crc32.JPG

Edited by steve6375, 04 September 2021 - 11:29 AM.

  • Atari800XL likes this

#12 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15888 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 04 September 2021 - 01:33 PM

Ah, very good to know, thanks.  :worship:

 

I didn't realize that you could add a length parameter after a comma  :frusty:  and of course completely forgot :angry7: how a (md) drive is actually a file.  :blush: .

 

Updated file, removing the half-@§§ed crc32 home-made routine (which was however fun to write)  and polishing it a little, and removing the pause  and clear as it fits in a single screen.

 

Now (here) it is nearly instantaneous, adding an animated .gif.

 

:duff:

Wonko

 

 

 

Attached Thumbnails

  • GPTCRC32.gif

  • Atari800XL likes this

#13 steve6375

steve6375

    Platinum Member

  • Developer
  • 7453 posts
  • Location:UK
  • Interests:computers, programming (masm,vb6,C,vbs), photography,TV,films,guitars, www.easy2boot.com
  •  
    United Kingdom

Posted 04 September 2021 - 02:51 PM

Nice, and I relearned about the echo --mem command which I forgot about!
  • Atari800XL likes this

#14 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 191 posts
  •  
    Netherlands

Posted 04 September 2021 - 03:36 PM

Excellent work, Wonko and Steve!

The output of the batch is instant now, even on the old DELL, even without the "floppy route".

 

There's no telling now what beautiful things will happen when two brilliant minds work together :-)

 

Me and the old DELL are ready to test anything you can come up with.

(Preferrably relating to hiding GPT partitions, of course.)



#15 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15888 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 04 September 2021 - 05:17 PM

Me and the old DELL are ready to test anything you can come up with.

(Preferrably relating to hiding GPT partitions, of course.)

Good :), but I don't know if I can trust you :w00t:

 

No, seriously, the batch right now is essentially read-only (though in latest incarnation it leaves the 34 sectors loaded into (md)0x300+34), so it is basically "safe" (though as said leaves memory drive "dirty").

 

At this point what I would do (had I available a suitable testing platform) would be to make a modified batch, let's say GPTcrc32nl.g4b without the initial loading of the sectors from disk and re-using those copied to memdrive, and start experimenting, finally reversing the modifications by issuing manually the "opposite" of the initial loading, i.e. if you ran (say) the:

 

GPTcrc32.g4b (hd0)0+34

 

the batch will execute initially:

dd if=%source% of=(md)%sect_md%+%sect_num% bs=512 count=%sect_num% 

i.e.

dd if=(hd0)0+34 of=(md)0x300+34 bs=512 count=34

which you can invert by issuing:

dd of=(hd0)0+34 if=(md)0x300+34 bs=512 count=34

with this command the modified sectors in (md) will be written back to disk.

 

Of course if there is a mistake in either the theory (the idea of limiting the partitions number) or in the practice (the modifications made) the result will be an unbootable disk (and no way back).

 

So, if you have:

1) an "expendable" USB stick
2) a dd image of it stored on some other media
3) the possibility of booting to grub4UEFI from the stick on your Dell computer

4) the possibilty of continuing booting to a PE or a Windows installation (residing on the internal disk)

then you can proceed.

 

The theory is as follows:

1) it seems that Windows based tools (but it has to be checked) default to make a EFI PART setup with 128 entries (128 bytes each) for partitions, these leads to the famous 32 sectors (LBA 2-33)

2) the specifications use a field (at offset 0x50 on sector LBA 1) that defines the amount of partiton entries

3) as well the specifications have a field (at offset 0x28 on sector LBA 1) that defines the first available sector for actual partitions (this is normally set to 34 for the reasons above BUT *any* GPT partitioning tool would normally have first partition at LBA 2048) most probably - but has to be verified - this is only precautionary (i.e. you can reduce the number of available partitions from 128 to - say - 4, leaving this field set to 34 or 0x22).

4) I have no idea if modifications to the EFI PART sector and/or to the partition array need to be reflected on their backup copy (at the end of the disk) i.e. whether a booting OS will "trust" the modified first few sectors (as long as the CRC32 checksums are valid) or if it will override them with the backup sectors (or viceversa it will automatically fix the backup sectors) or *whatever else*, so in order to make valid tests it might be needed to change not only the "main" but also the "backup"

 

The practice would be more or less *like*:

1) make a disk with 4 partitions

2) verify that it works and all drive letters are assigned/all 4 partitions are visible, etc.

3) change the EFI PART at 0x50 from 128 (0x80) to 4 (0x4)

4) change the first crc32 to reflect this change 

5) repeat tests in #2, nothing should have changed

6) copy last (fourth) partition entry (last 128 bytes of LBA 3) to the fifth position  (first 128 bytes of LBA 4)

7) 00 out fourth partition entry (write 128 00's to last 128 bytes of LBA 3)

8) change both the crc32's to reflect the change[1]

9) repeat tests in #2 this time the fourth partition should have "vanished"

10) change the EFI PART at 0x50 from 4 (0x4) to 5 (0x5)

11)change the first crc32 to reflect this change 

12) repeat tests in #2, situation should be back as in #2 and #5

13) copy third partition entry (128 bytes at offset 256 on LBA 3) to the sixth position (offset 128 on LBA 4)

14) 00 out third partition entry (write 128 00's at offset 256 of LBA 3)

15) change the EFI PART at 0x50 from 5 (0x5) to 6 (0x6)

16) change both the crc32's to reflect the change

17) repeat tests in #2, situation should be back as in #2 and #5 and #12

18) change the EFI PART at 0x50 from 6 (0x6) to 4 (0x4)

19) change the first crc32 to reflect this change 

20) repeat tests in #2, now you should only be able to see the first 2 partitions.

 

We have to understand if all (or only part) of these is possible:

1) have "holes" in the partition table (it should be possible because it is what should happen when you delete a partition)

2) have partition entries in an order different from actual physical order/position on disk

3)  whether the second crc32 checksum is intended by Windows and by other OS's as extending to only the present/defined in EFI PART partition entries or if it extends to sectors occupied by them (in other words whether the checksum in #16 above should be about 6 entries (128*6=768 bytes) or about 2 sectors or 1024 bytes.

 

It's a lot of testing, prone to errors and with possible roadblocks at each step that I cannot foresee, how to workaround them (if any) needs to be found out "on the spot".

 

:duff:

Wonko

 

 

[1] you need to edit the current version of the batch to calculate the crc32 on ONLY the sectors (or partition entries) used in the area between

 :pte_out

and 
crc32 (md)%sect_md%+%sectlen% | set crc_part=0x



#16 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 191 posts
  •  
    Netherlands

Posted 04 September 2021 - 05:41 PM


So, if you have:

1) an "expendable" USB stick
2) a dd image of it stored on some other media
3) the possibility of booting to grub4UEFI from the stick on your Dell computer

4) the possibilty of continuing booting to a PE or a Windows installation (residing on the internal disk)

 

OK, I will have to take things one step at a time...

Until now, for my tests I always boot Grub4UEFI straight from HD, and use a USB stick only for booting PE.

Please explain why it would be better to boot Grub4UEFI from USB?

 

Until now, I used Terabyte Image for Windows to make a backup of the EFI system partition, so I can quickly restore that partition when things go wrong. Until now, I never had to use this backup, because I didn't screw up thing bad enough beyond the point where I could simply boot PE from USB, [edit: assign letter to EFI part], copy back some files, etc.

Doing sector edits and things like that would be a different story of course. But as I said, the DELL is fully "expendable" and I can do any test on it.

 

So if we could start with these two points, please... (1) Why the separate USB and (2) Is an EFI backup like that good enough.

Thanks.



#17 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 191 posts
  •  
    Netherlands

Posted 04 September 2021 - 06:18 PM

Aah, I see, you want only the 4 partitions on the HD, not 5 (EFI part + 4x other)

 

 

No way we can do it with the EFI part on the HD?



#18 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15888 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 05 September 2021 - 07:21 AM

No, the USB idea is only as usually one has a normally bootable (from internal hard disk) system, that can also boot from USB, doing the experiments on the USB is only a way to make sure that a (very possible) mistake either in the theory or in the practice won't lead to an unbootable system :ph34r:.

 

If you have a tested way to recover the internal hard disk in case of problems you can use that just fine.

 

After all what will be changed in the experiments is only confined to the first 34 sectors (and possibly the last 34 ones) of the hard disk so if you have a (tested) way to boot froim alternate media (USB, CD/DVD, etc) *anything* capable of restoring these set (or sets) of sectors you can proceed just fine, though that there is still the risk that when booted with these different settings in the GPT the Windows will decide to "automagically" change *something* in drive letter assignements (or *whatever*) that might force you to perform a complete restore of the Windows partition.

 

Of course starting with 5 partitons instead of 4 is fine as well, you leave the 5th partition alone and still try "moving" the 3rd and 4th, only, for the sake of the test, you will need to move them to the 3rd sector (instead of the second), creating the "hole" not only on the first but also on the second sector, i.e. starting with 5 partitions you have (grouped by sector):

1st slot=1st partition

2nd slot=2nd partition

3rd slot=3rd partition

4th slot=4th partition

 

5th slot=5th partition

6th slot=empty

7th slot=empty

8th slot= empty

 

9th slot=empty

10th slot=empty

11th slot=empty

12th slot= empty

 

and you have to test:

 

1st slot=1st partition

2nd slot=2nd partition

3rd slot=3rd partition

4th slot=empty

 

5th slot=5th partition

6th slot=empty

7th slot=empty

8th slot= empty

 

9th slot=4th partition

10th slot=empty

11th slot=empty

12th slot= empty

 

and:

 

1st slot=1st partition

2nd slot=2nd partition

3rd slot=empty

4th slot=empty

 

5th slot=5th partition

6th slot=empty

7th slot=empty

8th slot= empty

 

9th slot=4th partition

10th slot=3rd partition

11th slot=empty

12th slot= empty

 

:duff:

Wonko


  • Atari800XL likes this

#19 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 191 posts
  •  
    Netherlands

Posted 05 September 2021 - 07:54 AM

If you have a tested way to recover the internal hard disk in case of problems you can use that just fine.

 

 

So I guess I will first have to check this "tested way to recover the internal hard disk".

 

My current partitions on the test laptop (DELL 3531) are:

p1=EFI 100mb --- p2="Windows1" 40gb --- p3="Windows2" 40gb --- p4="Windows3" 40gb --- p5="Windows4" 40gb

 

I have already created a working apply system that uses the previous system mentioned in the other thread, which means: Using Grub4UEFI for multiboot (without MS BCD menus etc), using NTFS drivers in UEFI, using extra EFI folders on the OS partitions, preventing unwanted drive letter assignments, etc. This is working quite nice actually, for example I have tested the latest Windows 11 preview, using apply directly from UUP files, no conversion to .wim or .iso necessary. All the time saved in that case, makes up for the slow internal hard disk... (But that's a different story).

I'm just mentioning this to repeat that the DELL harddisk is completely "expendable", everything can be restored either from backups, or from "scratch". Fully unattended, no "drive letter surprises", etc. Thanks again for all the help I got for that.

 

So now on to this NEW test, to see if we can find out a way to "better", "really" hide these GPT partitions.

 

I'm not very experienced in the advanced Grub4Dos batch commands, so I would like to run a test like this first:

(1) Boot PE from USB, make a (small and quick) Terabyte backup of the EFI system partition. Maybe also a "Drive Snapshot" backup. Also a full disk backup, I guess.

 

EDIT: Maybe an image of the EFI partition is not even necessary. After all, it's quite a "simple" partition, isn't it? If I recreate it with Diskpart from PE, all I need is to copy back the files. The only thing that might normally be "finicky" would be the BCD entries, but these are not needed when using the Grub4UEFI multiboot method....

 

(2) Boot from HD to the Grub4UEFI command line.

(3) Display the partition info

(4) Change the partition info

(5) Test (misc.)

(6) Restore the EFI partition from the Terabyte backup, to make sure everything is really reset to the original situation. If the backup doesn't restore things that we *assumed* it would, the backup is useless.

 

Would this be a good plan?

Can you help me with some code for items (3) and (4)?

 

Thanks!



#20 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15888 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 05 September 2021 - 11:35 AM

Yep, the plan is good.

 

No prob with assistance with #3 and #4

 

The 34 sectors are loaded by the current GPTcrc32.g4b to a memory drive that starts:

as device at (md)0x300

as memory at 0x60000

 

The batch, for *some reasons* sets the debug msg level to "no show" and turns the pager off, so you will be better, after having run it to reset them:

debug normal

pager on

when working on command line.

 

A sector is 1 sector on (md) device or 0x200 (please read as 512) bytes in memory.

First sector (the copy of LBA0 on the hard disk, i.e. the MBR) will be:

(md)0x300+1

or

0x60000 (and extend for 0x200 bytes)

 

i.e. you can see its contents by issuing either of:

cat --hex (md)0x300+1

or

echo --mem=0x60000=0x200

 

The actual sectors on which you will experiment are following, i.e. the "map" is:

LBA0 (MBR) = (md)0x300+1 = 0x60000

LBA1 (EFI PART) = (md)0x301+1 = 0x60200

LBA2 (1st set of 4 partition slots)= (md)0x302+1 = 0x60400

LBA3 (2nd set of 4 partitoin entries) = (md)0x303+1 = 0x60600

LBA4 (3rd set of 4 partition entries) = (md)0x304+1 = 0x60800

 

To write a value at a given position you can try "write", examples:

1) to write value "0x05" to field "partition entry numbers" you have to know:

a. the field is on LBA1 (EFI PART) so it's start is at 0x60200

b. the field is at offset 0x50 in that sector
c. the field is 4 bytes (or 32 bit value), so you use: write --offset=0x50 --bytes=4 0x60200 0x5

 

2) to write a new CRC32 for partition array

a. the area to be crc'ed starts on LBA2 (hence (md)0x302+1 = 0x60400)

b. the extension is to be tested, let's say that (since you have written 0x05 as number of partitions in the previous step you want the crc32 of the first 5 entries only)

c. that will be crc32 (md)0x302+2,640 (the 2 can be 3, 13 or 32, what matters is the 640 after the comma, that is 5*128=640)

d. you will get a hex number without the leading 0x

e. we know that that crc32 is on LBA1 (EFI PART, i.e. 0x60200 or (md)0x301+1) at offset 0x58 and that it is a 4 byte value

f. write --offset=0x58 --bytes=4 0x60200 0x<the number you got above>

 

3) to write a new CRC32 for EFI PART on LBA1 (i.e. 0x60200 or (md)0x301+1) at offset 0x10
a. you need first to 00 out existing crc

write --offset=0x10 --bytes=4 0x60200 0x0

b. you calculate the CRC32: crc32 (md)0x301+1,92

c.you will get a hex number without the leading 0x

d. write --offset=0x10 --bytes=4 0x60200 <the number you got above>

 

4) to copy a partition entry (for this example move 4th partition to 9th slot)

a. source, 4th partition is at offset 3*128=384 on first partition table sector (i.e. LBA2)

b. target, 9th partiion slot is at offset 0 on third partition table sector (i.e. LBA4)

c. dd if=(md)0x302+1 of=(md)0x304+1 bs=1 count=128 --skip=384 --seek=0

 

5) to 00 out a partition entry (for this example the 3rd partition

a. target third partition is at offset 2*128=256 on first partition table sector (i.e. LBA2)

b. the sector is at address 0x60400, but you have to start at offset 256=0x100 and write the byte 0=0x00 for 128=0x50 times

c. call Fn.24 0x60500 0x00 0x50

 

Ask if you have any doubt/need any clarification.

 

:duff:

Wonko



#21 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 191 posts
  •  
    Netherlands

Posted 05 September 2021 - 03:05 PM

Wonko, I just finished my first "real" test (with actual "writing back the sector to disk") and things look really promising!!!

 

But first off: I must admit I did a slightly different test than we agreed on, let me try to explain:

(1) The partitions on the HD were still the same as a previous test scenario, so that included two extra "data" partitions after the OS partitions. So that's 1xEFI, 4xOS, and 2x Data partitions. In total: **7** partitions.

(2) It took some time (again) to wrap my mind around the new (for me) Grub4Dos batch commands, at first my idea was to postpone any tests until I fully understood what was going on, but then my second thought was "to hell with it", let's just try it until it crashes (maybe I was looking forward to rebuilding everything from scratch as I mentioned before, must be some type of Stockholm syndrome).

 

So first let me mention the exact commands I used. I actually started to understand the command as I typed them :-) (Learning by doing, I guess).

As I typed the "0x5" for the number of partitions, I thought to myself "Hey, let's write this new info to disk right now and see how the "Data" partitions show up in Windows!")

So I hope you can forgive me that I changed this first test a bit.... I will of course do the full test later (might be some days though).

gptcrc32.g4b (hd0)0+34
debug normal
pager on
write --offset=0x50 --bytes=4 0x60200 0x5
crc32 (md)0x302+2,640
result: b8454d4c
write --offset=0x58 --bytes=4 0x60200 0xb8454d4c
write --offset=0x10 --bytes=4 0x60200 0x0
crc32 (md)0x301+1,92
result: 1acbcce5
write --offset=0x10 --bytes=4 0x60200 0x1acbcce5 (Wonko forgot the "0x" here)
dd of=(hd0)0+34 if=(md)0x300+34 bs=512 count=34
output:
buf_size=0x100000,  loops=0x1.  in_pos=0x0, out_pos=0x0
00000000

After this, I rebooted the laptop, then started Windows from OS partition 1 (using Grub4UEFI), then opened Disk Management.

Partitions 6 and 7 were gone. And this time they were **REALLY** gone! (Disk Management displays "Unallocated")

 

Next up I rebooted again, repeated the code above, but now I reset the number of partitions to 7.

Rebooted to Windows and the Data partitions were back!

 

So to me this was a very successful test, at least I got my "feet wet" with the Grub4Dos commands, and it gives me some confidence to start juggling around with the actual OS partitions.



#22 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15888 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 06 September 2021 - 08:01 AM

Well, that is the whole point of experimenting, (and actually of living :)):

Life is “trying things to see if they work”

 

(Wonko forgot the "0x" here)


... just checking if you were paying attention ... :innocent:   :whistling:

 

Results are encouraging :thumbsup: , though at the moment we don't know anything about changing order of partitions you have (tentatively) established that:

1) the partition array checksum covers the exact number of sectors declared in the field at offset 0x50 in EFI PART, and they can be an arbitrary, even odd (if you pardon me the pun) number such as 5 and not a multiple of 4

2) a modified GPT  (heck, today must be pun day) primary partition structure set of partitioning data can co-live with an outdated/different secondary one (the "backup" at the end of the disk), in the sense that the computer will boot using the primary one without exploding[1]

 

As you have seen, limiting the partition entries extension provokes the OS to believe that the area past the last partition in the partition table and the backup copy of the GPT data at the end of the disk is empty.

 

What has to be checked is whether the "normal" tools (let's say diskpart or disk management) can extend the partition table size and re-create new partitions in that space or if they are "locked out", as that could be a possible issue.

 

Once we (you) will have tested the already seen list of hypothized behaviours, it would make sense (to me at least) to see if one of the existing GUID's are "not liked" by Windows, i.e. if instead of hiding the partitions by cutting the partition table short, it wouldn't be better to change the partition ID toi something that is not liked or that is ignored (I am thinking of the MSR partiton ID or - say - the Haiku one).

In case we can invent our own partition ID, I vote for:

 

47727562-3445-4649-2052-756C65732120

:duff:

Wonko

 

[1] if this is confirmed and the backup actually works as backup, this gives us a good way to restore the disk to the initial status after any modification.



#23 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 191 posts
  •  
    Netherlands

Posted 06 September 2021 - 02:13 PM

It might be a few days before I can resume testing, but this is the new plan:

 

- There are 4 "OS" (Windows) partitions (p2,p3,p4,p5), let's try to "clear out" the entries for partition 3 (=Windows2).

 

- Expected result after booting Windows (from p2, p4 or p5), or from PE: there should be a section labeled "Unallocated" in Disk Management, where p3 used to be.

 

- In the partition table, the partition entry should be moved to a free "slot", as mentioned in Wonko's post above. This way, it can be moved back later, if all works as expected, the partition should then become active/ visible again, but this is not guaranteed.

 

- If I understand correctly, the CRC32 checksum is calculated only on the "active" (which might be a confusion term here) or "limited number of" partitions?

In this example: The checksum is for the table entries of p1 (EFI), p2 (Windows1), p3 (zeroed out), p4 (Windows3), p5 (Windows4), p6 (Data1) and p7 (Data2)

That would also mean the checksum in NOT calculated on the "placeholder" entry? (the p2 table entry that is "moved away" to a hopefully safe place further on in the table).



#24 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 15888 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 06 September 2021 - 04:22 PM

No,  if you "declare" 0x80 (128) partitions at offset 0x50 in EFI PART, the partition array consists of 0x80 entries (each of 0x80 or 128 bytes) and the checksum is calculated on the partition array, i.e. all of the 32 sectors after EFI PART. (no matter how many of these slots are empty, 128 00's are a non-partition, but the crc32 i s computed on those 00's as well)
 
From the results of your preliminary test it seems that this is valid for *any* number of partition entries, what you did was to declare 0x5 partitions and calculate the checksum on 5*128=640 bytes following the EFI PART sector.
 
So, everything is cool.
 
The doubt (maybe I was mis-reading the report on Rufus I cited earlier) was whether the number of partitions declared at offset 0x50 needed to be a multiple of 4 or - if you prefer - if the checksum was made on whole 512 bytes sectors (4*128=512) only.


:duff:
Wonko



#25 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 191 posts
  •  
    Netherlands

Posted 09 September 2021 - 03:36 PM

OK, I'm almost ready to get back to testing.

I think my first order of business is to automate the re-creation of the partitions on the hard disk. At one point I thought I would create a backup copy of the "LBA0/ MBR" sector, but I must admit I'm not sure if such a backup would be useful/ possible/ realiable, etc.

So my current plan is to create a batch like this:

 

(1) Diskpart script: clean disk, convert to GPT, create partitions (1=UEFI 100mb, 2,3,4 and 5: 40gb NTFS "OS" partitions, 6 and 7: Data partitions)

(2) Don't assign drive letters to the OS partitions, they would only get "in the way". Assigning a letter (C: in this case) should be done (much) later in the OS Apply phase. When we reach that apply phase, we will reset the "0x8..." attribute to the partition we assing letter "C:" to (this was explained in previous posts).

(3) Instead of drive letters, descriptive drive labels could be more useful (Windows1, Windows2, Windows3, Windows4). This should help (enough) to check which partitions end up hidden/ moved/ borked.

(4) Assign letters "C" and "D" (or "D" and "E", or whatever) to the data partitions. This shouldn't hurt anything. When we **manually** set drive letters to these partitions, the "0x8..." attribute is **cleared**, so that will mean that on a reboot (to OS, or in these test most likely: to PE), these partitions will be assigned a (new) letter anyway, which letter exactly will depend on the partition experiments.

(5) Assign letter S: to the UEFI system part. The UEFI partition is different from a normal partition, so it is never automatically assigned a drive letter (the "0x8..." attribute story is not applicable here).

(6) Copy all files needed for the experiments to S:

(Microsoft boot files, Grub4UEFI boot files, menu.lst, splash screen, font file etc., A1ive GrubFM, etc.)

 

(7) Go wild with shifting stuff around in the partition table. Test, test, test.

 

I don't think actual Windows installs are necessary for these (possibly "destructive") tests, at least not in the earliets stages, booting a PE should be enough to view the outcome of the "hide/ unhide" tests.

 

Hmmm, hope this will work, thank you for letting me think out loud...






1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users