Jump to content











Photo
- - - - -

[Release] GPTcrc32.g4b to (re-)calculate CRC32 hashes in GPT header


  • Please log in to reply
36 replies to this topic

#26 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 192 posts
  •  
    Netherlands

Posted 10 September 2021 - 01:36 PM


The actual sectors on which you will experiment are following, i.e. the "map" is:

LBA0 (MBR) = (md)0x300+1 = 0x60000

LBA1 (EFI PART) = (md)0x301+1 = 0x60200

LBA2 (1st set of 4 partition slots)= (md)0x302+1 = 0x60400

LBA3 (2nd set of 4 partitoin entries) = (md)0x303+1 = 0x60600

LBA4 (3rd set of 4 partition entries) = (md)0x304+1 = 0x60800


:duff:

Wonko

Sorry, did we ever discuss why the EFI PART is using up a lot more room in the partition table than the other basic partitions?

With you excellent new G4ub4Dos Sector Editor, it's all the more apparent the this one's using a lot more space, whereas the data partition are crammed together.

Or is it just a lot of "reserved for future use"?



#27 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 10 September 2021 - 04:14 PM

The sector "EFI PART" (LBA1) is NOT the partition table, it is ONLY the "header" and "index" for the partition table (that starts on LBA2).

 

Basically the good guys writing the specifications used a lot of bytes for either unneeded or redundant information, but even then couldn't use more than 92 bytes (i.e. less than the "normal" 128 bytes used in a partition entry) SO they "reserved for future use" the remaining 420 bytes.

 

As a matter of fact I have seen (here and there) quite a few disks where *something* was written in these last 420 bytes, but really cannot say who/which tool/OS does not respect the specs.

 

On one hand, "wasting" the space where 3 partition table entries could have been seems a lot, on the other, almost *any* modern OS wastes almost 2,000 sectors anyway, so reserving a whole sector for writing 92 bytes is not that bad.  

 

:duff:

Wonko



#28 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 192 posts
  •  
    Netherlands

Posted 11 September 2021 - 12:24 PM

OK, back to testing now. I have automated the creation of the partitions, etc. so no need for an image/ backup of the partition table, etc.

 

I repeated the steps in post #21 of this thread, it works consistently (change number of partitions directly in partition table, from 7 to 5, recreate checksums, etc.)

When I boot PE from a USB, I can check Disk Management to see if/ how things have changed. Changing the number of partitions from 7 to 5 leaves the right-most part "unallocated", changing it back to 7 shows the original partitions again.

So far so good.

 

Now it's time for the more interesting stuff.

As a first test, I just cleared out an entry in the partition table, even without recreating the crc32 checksums:

 

gptcrc32.g4b (hd0)0+34
(To read the partition table into memory)

 

call Fn.24 0x60600 0x00 0x80

(To clear the entry for one of the partitions. Not sure if these are the correct values, though...)

Wonko used "0x50" for the number of bytes to clear. Is that correct, or should I use 0x80 for 128 bytes?

 

EDIT: I used SectEdit to confirm that the bytes were cleared.

 

dd of=(hd0)0+34 if=(md)0x300+34 bs=512 count=34
(To write the table back to the hd).

 

Because this was a very blunt way of cutting, without even creating valid checksums, I was expecting as error of some kind, but the result was quite underwhelming: There was no visible change in te partition info (Disk management in PE).

My first suspicion was: Maybe this "incorrect" table was not used now, maybe the backup copy of the table was used instead.

 

As long as I'm learning some new stuff with each test, I will go on trying, I hope I will produce more interesting results with further tests.



#29 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 192 posts
  •  
    Netherlands

Posted 11 September 2021 - 12:45 PM

Aah, some news.

In my previous post I mentioned that there was "No visible change"  after that blunt cutting in the partition table.

But on a reboot (or only after power off and power on?) this is what I see in Disk Management under PE:

 

- There is one big partition on the disk now, called "(GPT Protective Partition)", size = 2048 GB

(Mind you: the harddisk is 465gb, so the 2048 GB is just some type of place holder).

 

I saw this one before in a borked test, but forgot to write down the details. I searched on "GPT Protective Partition" and saw that it was indeed some sort of sign of "Hey, I give up, things are screwed beyond repair".

 

So lesson learned: Don't just cut holes in the thing...



#30 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 192 posts
  •  
    Netherlands

Posted 12 September 2021 - 09:49 AM

This might be interesting:

 

- I never tried BootIce's "Backup / Restore partition table" feature before, but now that I'm a bit more confident in trying different scenarios, I thought I'd try it out. Of course, it works perfectly:

 

(1) Boot from PE, start Bootice, select harddisk, "Parts Manage", "Backup Partition Table"

(2) Using Disk Management: Remove partition 4 (which is the third OS partition). This is a "clean" remove, which shows up in Disk Management as "Unallocated". We haven't found a way yet to do a clean cut like this in Grub4Dos batch language.

(3) Make another Bootice partition table backup as in (1)

 

Both partition table backups can now be compared.

 

(4) This was very surprising to me: I used Bootice to *restore* the first partition table backup to the harddisk. All the contents of the partition was still there, and the OS was perfectly bootable. So to me this proves that when you use Disk Management to remove a partition, the deletion is clean enough to modify the partition table, but not the actual data on the partition.

 

Then this thought came to me:

- To *really* and *cleanly* hide the OS partitions from each other, I could boot to PE each time (prior to OS boot from HD using Grub4UEFI) and restore a suitable Bootice partition table backup. Possible names would be:

"A Show1-Hide2-Hide3-Hide4",

"B Hide1-Show2-Hide3-Hide4",

"C Hide1-Hide2-Show3-Hide4",

"D Hide1-Hide2-Hide3-Show4"

 

I think I will actually test that... Not at all practical, but maybe educational :-)

 

Now, wouldn't it be nice if we could do this in Grub4UEFI? And I think Wonko could actually do this!?

If we know the format of the Bootice partition table backup, wouldn't it be possible to restore the partition table in Grub4UEFI?

 

 

And of course, after typing out this idea, I realize it would also be possible to do the actual partition backups with Grub4UEFI as well, but for now it seems easier and more reliable to do the partition deletion and table backup in PE.

After all, the functionality is already there in Bootice, the extra functionality needed in Grub4UEFI would only be the restore part.

 

What do you think?



#31 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 192 posts
  •  
    Netherlands

Posted 12 September 2021 - 01:56 PM

Well, supporting the Bootice partition table backup format might have been a stupid idea...

 

But if you ever have the time to write some code to do a full table backup in Grub4Dos/EFI (to a file), I would be more than willing to test it, using a similar scenario as in the previous post.



#32 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 12 September 2021 - 04:36 PM

The issue (not a real issue, let's call it a limitation, you get it) is that NO file can be created or enlarged in grub4dos, unless (exception) on a FAT filesystem by using the FAT external module (and that is not fully-fully safe anyway).

 

There are no problems in grub4dos to read an existing file or to write to it as long as the amount of bytes written do not exceed the amount of bytes already on the file (file size).

 

Otherwise you have (on a "normal" GPT style disk) sectors LBA 34-2047 that are not used.

 

I don't think that if you use 34 of these  2013 unused sectors to save a backup anyone will be horrified, actually you can put up to 59 backups of the sectors LBA 0-33 in that space.

 

But as well, you can create any number of files in *any* OS with file creation support and then use them in grub4dos just fine (as long as they are at least 17408 bytes in size each), GPTBackup01.bin, GPTBackup02.bin ... etc..

 

Backup the GPT partition table of disk (hd0) to hidden area starting at sector 42

dd if=(hd0)0+34 of=(hd0)42+34 bs=512 count=34

or just:

dd if=(hd0)0+34 of=(hd0)42+34

 

Restore the backup:

dd of=(hd0)0+34 if=(hd)42+34 bs=512 count=34

or just:

dd of=(hd0)0+34 if=(hd)42+34

 

 

Backup the GPT partition table of disk (hd0) to an existing file GPTBackup01.bin on first partition

dd if=(hd0)0+34 of=(hd0,0)/gptbackup01.bin bs=512 count=34

or just:

dd if=(hd0)0+34 of=(hd0,0)/gptbackup01.bin

 

 

Restore the backup:

dd of=(hd0)0+34 if=(hd0,0)/gptbackup01.bin bs=512 count=34

you SHOULD NOT use the simplified form:

dd of=(hd0)0+34 if=(hd0,0)/gptbackup01.bin

unless you are sure that the file is exactly 17408 bytes in size.

 

I don't think that the above is overly difficult, with a minimum of attention, on plain grub4dos command line.

 

The matter is more complicated if/when you need to keep also a backup of the backup GPT partitoin table, as you need to find out where it is exactly (at the end of the disk).

 

:duff:

Wonko


  • Atari800XL likes this

#33 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 192 posts
  •  
    Netherlands

Posted 12 September 2021 - 06:12 PM

Thank you very much for that! I was sure you were fed up with this by now :-)

 

Those are some great examples you mentioned, and it was pretty much EXACTLY what I was looking for!

This will provide enough material for some interesting tests over the next days, I will let you know how things are going.

I have learned a lot over the last few weeks, so thank you again for your expert advice and patience.



#34 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 192 posts
  •  
    Netherlands

Posted 12 September 2021 - 07:08 PM

Wow!!! This looks VERY promising indeed!

 

My weekend is almost over, but I couldn't resist trying it out:

 

Partitions on hd are all "normal" and visible (p1=EFI, p2/3/4/5: OS partitions, p6/7: Data)

 

(1) In Windows, create file "carrier.dat" of exactly 17042 bytes (I used "FileFiller")

(2) Boot PE, copy file to harddisk (EFI partition).

(3) Boot Grub4UEFI, "c" for command line, create partition table backup with:

dd if=(hd0)0+34 of=(hd0,0)/carrier.dat bs=512 count=34

(4) Boot PE, copy (don't move) S:\carrier.dat to a new name, "ptb-1111.bin" (ptb= "partition table backup", all the "1's" indicate a "visible" partition, a "0" would indicate a hidden partition). So this particular backup has all partitions as "normal".

 

(5) Still in PE, use diskmanagement to remove "OS" partitions 2,3 and 4 (partition numbers 3,4,5). So now only the EFI partition is visible, one OS partition, and the two data partitions. This is the desired layout for when we want to boot "OS1/ Windows1" with Grub4UEFI!!!

 

(6) Reboot to Grub4UEFI, command line, create a backup of this particular partition table, same command line as (3)

This causes "carrier.dat" to be overwritten, which is as expected (and actually "necessary", as Wonko showed).

 

(7) Like (4), boot PE, copy (don't move) S:\carrier.dat to a new name, this time it's "ptb-1000.bin" (1=visible, 0=hidden).

 

At this point I first wanted to test a restore of the original partition table (ptb-1111.bin), and everything worked! Tried to boot Windows from partition 2 ("OS1/Windows1"), worked fine.

 

So now I have to repeat the steps above to complete the list:

ptb-1111.bin (safety backup, all partitions visible)

ptb-1000.bin (show OS1, hide others)

ptb-0100.bin (show OS2, hide others)

ptb-0010.bin (show OS3, hide others)

ptb-0001.bin (show OS4, hide others)

 

When all the files are created, I will test integrating them into menu.lst, which means adding this command for each OS selection:

dd of=(hd0)0+34 if=(hd0,0)/ptb-xxxx.bin bs=512 count=34

(replace "xxxx" with the desired config).

 

Until now, I only have "1111" and "1000", I will do the others later.

 

This might not be the "ultimate" solution for "simple" GPT partition hiding, but it is a very interesting concept, and until I learn more by studying the actual partition table backups, etc., I think this is not a bad solution!

All the "hard work" can be conveniently done in PE/ DiskManagement, Grub4UEFI doesn't have to do much "new" stuff, you just have to make sure you have your partition table backups copied to the correct filenames.

 

A very big thanks again to Wonko!



#35 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 13 September 2021 - 12:41 PM

(1) In Windows, create file "carrier.dat" of exactly 17042 bytes (I used "FileFiller")

 

Why?

34*512=17408

 

You failed to backup 17408-17042=366 bytes

 

It's fine, though :), those 366 byte are anyway all 00's, as they correspond to the 126th, 127th and 128th partitions (that you don't have)

 

Which leads us back to one of  the initial points, you don't really need to backup and restore 34 sectors if you know that only the first 4 of them (i.e. if you have up to 8 GPT partitions) are "populated" and the other 30 sectors are all 00 filled and the good guys that wrote the specifications provided us with the field at offset 0x50 in EFI PART that we can set to the max number of "used partitions" (rounded to sector) and thus limit the number of sectors to backup and restore.

 

:duff:

Wonko


  • Atari800XL likes this

#36 Atari800XL

Atari800XL

    Frequent Member

  • Advanced user
  • 192 posts
  •  
    Netherlands

Posted 13 September 2021 - 12:53 PM

Sorry, I meant to say 17408 bytes... I guess I got distracted...

 

Maybe 17042 is my bank PIN number :-)

 

Of course, you are correct about this backup being "too large" anyway, but I will just start working "downwards" from here (from a working and automated method), this will include "smaller"/ different backups, comparing backups, comparing these with the Bootice format, etc.

 

The surprising part for me was the fact that all the data on the "removed" partitions seems to be still there, after you restore the partition table to a "previous state".

If I had written that MS code, I would try to sneak in some "0"s here and there, just for fun :-)



#37 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 16066 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 13 September 2021 - 03:57 PM

The surprising part for me was the fact that all the data on the "removed" partitions seems to be still there, after you restore the partition table to a "previous state".

If I had written that MS code, I would try to sneak in some "0"s here and there, just for fun :-)

 

Well, there is nothing to be surprised about.

 

Imagine a book where first 5 pages are the index, something *like*:

Index ... Page 1

Chapter 1 ... Page 6

Chapter 2 ... Page 42

Chapter 3 ... Page 57

...

So you take the book and cover with correction ink the contents of the first 5 pages. :w00t:

 

But it is not like pages from 6 to the end of the book vanish with a poof.

 

What should actually surprise you is that up to XP the format command did the same thing to the indexes of volumes (i.e. FATs or $MFT) and all files were still there (and could be recovered easily as long as not fragmented), from Vista onwards the behaviour is the same, but ONLY if you use the quick format (or /q switch), otherwise volumes are actually zeroed out.

 

The idea (right or wrong) is/was that wiping sectors takes time, while zeroing out only the indexes is fast, and while the good MS guys may sometimes appear less than brilliant, they are not intentionally evil/spiteful.

 

:duff:

Wonko


  • Atari800XL likes this




1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users