Windows altering (and messing up) my GPT partition table behind the scene
#1
Posted 01 January 2017 - 12:38 PM
#2
Posted 01 January 2017 - 02:17 PM
I am not sure to understand.
At first sight the "good" file is not " pure GPT", is a "hybrid" MBR/GPT.
WHEN does "Windows" (which version) alters its contents?
The "good" has seemingly only one partition, whilst the "bad" has seemingly 4 of them.
BOTH the "good" AND the "bad" have also NO disk signature (which more or less should mean that it has been "generated" by *something* but never mounted.
Most of the differences (once set aside the different number of partitions) are due to checksums, and this is more or less "normal".
"good": Disk \\.\Physicaldrive8: 2097152 sectors, 1024.0 MiB Logical sector size: 512 bytes Disk identifier (GUID): C556E9BF-F1D5-412E-A1E2-9784B8B16E38 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 2097118 Partitions will be aligned on 64-sector boundaries Total free space is 2088957 sectors (1020.0 MiB) Number Start (sector) End (sector) Size Code Name 1 64 8191 4.0 MiB EF00 "bad": Disk \\.\Physicaldrive9: 2097152 sectors, 1024.0 MiB Logical sector size: 512 bytes Disk identifier (GUID): C556E9BF-F1D5-412E-A1E2-9784B8B16E38 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 2096606 Partitions will be aligned on 32-sector boundaries Total free space is 839261 sectors (409.8 MiB) Number Start (sector) End (sector) Size Code Name 1 64 8191 4.0 MiB EF00 2 8224 520191 250.0 MiB 0700 3 520224 1032191 250.0 MiB 0700 4 1032224 1257471 110.0 MiB FC00
You need to provide some more context ...
How exactly the "good" one was created, which partitions it should contain, etc.
Wonko
#3
Posted 01 January 2017 - 02:33 PM
Hi Wonko,
I installed esxi in a vmware machine (1gb disk) using ESXI Installable ISO.
Once installed and tested it was working fine (i.e I rebooted the vmware guest without the ISO), I converted the VMDK to a RAW disk.
And then finally wrote this RAW disk to a USB key.
If I insert this USB key under windows (or mount the image under windows), the GPT partition table gets modified.
I can also add that if I boot a winpe in my vmware guest (where the working 1gb disk sits), at next reboot, i can no longer boot on the disk and will again get "hypervisor not found".
So whether I mount the image under my full windows or boot my vm with a winpe, the GPT partition gets messed up.
good.dd comes from the original raw disk (which boots fine)
bad.dd comes from the same raw disk after it has been "touched" by windows (which no longer boots).
Regards,
Erwan
#6
Posted 01 January 2017 - 03:09 PM
Well, what you posted as "good" has a single partition, even if seen in a plain hex editor, maybe you posted the "wrong" file.
Anyway the "issue" is probably in:
"good": First usable sector is 34, last usable sector is 2097118
vs.
"bad": First usable sector is 34, last usable sector is 2096606
The first is OK, since the image is 2097152 sectors - 34 = 2097118
whilst the second makes little sense, as 2097152 - 512 - 34=2096606
this 512 sectors difference is "queer".
As well it is "queer" (though it may be due to the small size of the image) that the first partition starts at LBA 64 (as it normally would stat at 2048).
Additionally, both have "unbalanced CHS/LBA values, and you will have to explain how/why it is a "Hybrid MBR" (i.e. containing boot code).
Please do recheck the data you posted and please provide also the last three sectors of the image (besides the first three) so that I don't have to re-build the backup GPT (which should be OK, but that maybe introduces some variations, i.e.:
good_first3.txt
good_last3.txt
bad_first3.txt
bad_last3.txt
Wonko
#7
Posted 01 January 2017 - 03:19 PM
3 first and 3 last sectors for the good image (booting fine) and the bad image ("hypervisor not found").
3 sectors = 0x600 / 1536 bytes.
Unique and only difference between the 2 images : windows has seen / touched the second one making it a bad image.
Some extra hints from vmware forum :
-https://kb.vmware.co...ernalId=2012022
To resolve this issue, use a Live Linux CD of your choice. Relabel the local datastore and create a partition table with one partition. Then re-install ESXi.
-https://communities....tart=0&tstart=0
Attached Files
#8
Posted 01 January 2017 - 03:59 PM
Well, the "good" still has a single partition whilst the "bad" still has 4 of them.
The DMDE screenshot on your post #4 cannot reasonably be coming from this same "good" image, there is simply NO info on first sector of the GPT partition table.
Try again, this time taking 34 sectors from the beginning and 34 sectors from the end of each image.
I am starting to suspect that (for *whatever* reasons) the "good" image has the other three partitions on "slots" - say - #35, #42 and #110 and again (for *whatever* reasons) *something* re-orders them (possibly creating some other issue while doing that).
The report here:
https://communities....tart=0&tstart=0
talks about a single empty partition inserted, but if his was the case, sector LBA 2 of "good" image would still contain 3 partition entries (besides the inserted empty one, as each partition entry is 128 bytes)
Wonko
#9
Posted 01 January 2017 - 04:17 PM
Look at these 2 screenshots coming from the good and bad images.
The same images i used to extra the 3 first/last sectors.
The good image has 5 entries (other 3 blocks) but with 3 empty/dummy ones stuff in the middle : #1, #5, #6, #7, #8.
The bad image seems to have been reorganised (thank you windows...) with consecutive entries other 2 blocks : #1, #2, #3, #4, #5
Number of partitions and GUID are the same but not the partition ID's / setup differ.
What tool are you using to parse the GPT partition table?
Looks like your tool is not parsing correctly the GPT partition in the "good" image?
I will shortly attached the 34 first/last sectors of each image.
The error below seems to indicate that esxi is expecting a bank6 = entry/partition #6 which no longer exist since windows are "kindly" reorganized it all for me
Question remains : why the hell is windows bothering to alter/mess up my partition table?! and how to prevent that (i dont need to fix it as I can easily rewrite my raw image to my media).
Probably it believes this setup is wrong but at least it should ask me if I want to fix it or not.
Edit : I believe your tool gets "fooled" by the "good" GPT setup and only read one entry because of the 3 empty/dummy entries right after entry #1.
Therefore it ignores #5,#6,#7,#8.
Which may indicate that vmware setup breaks the GPT standards : entries should be consecutive?
#10
Posted 01 January 2017 - 04:30 PM
bad_last34.txt 17KB 609 downloadsAs requested, 34 sectors first/last on both images.
From thread here, i believe the issue lays there : "Second, I see that ESXi 5.0 leaves an empty partition entry slot(for untold reason) , and Windows shifts the entries behind to fill the gap after GPT modification. So ESXi's boot bank BANK6 gets screwed. I think a VMware so-called BANK is a synonym for a partition entry."
Attached Files
#11
Posted 01 January 2017 - 04:39 PM
Sure , but you provided only 3 sectors:The good image has 5 entries (other 3 blocks) but with 3 empty/dummy ones stuff in the middle : #1, #5, #6, #7, #8.
#1 the MBR
#2 the EFI PART
#3 the FIRST sector ONLY of the actual partition table that (since each slot is 128 bytes) contains slots #1, #2, #3 and #4
Rest assured my tool parses allright what is there, (but it cannot invent what it isn't there).What tool are you using to parse the GPT partition table?
Looks like your tool is not parsing correctly the GPT partition in the "good" image?
If slots #2, #3 and #4 are empty, and I have not the sector containing slot #5 onwards, the only partition I can see is the one in slot #1.
I'll check the 34 sectors files.
As a first guess, probably we can fill the empty slots with "fake" partitions so that everything is "kosher".
Wonko
P.S.: The good_first34.txt file is only partial, please try again.
#12
Posted 01 January 2017 - 04:44 PM
arg, sorry, i messed up ....
now attached the right file to the previous post.
#13
Posted 01 January 2017 - 05:18 PM
Make a copy of the "good" image.
Overwrite first and last 34 sectors with the files attached.
Attempt repeating the *whatever* sequence that corrupted the "original" good image.
I inserted three (slots #2, #3 and #4) 4 kb partitions (very, very small ) of the Haiku BFS type (which hopefully Windows and *anything else* but Haiku should "leave alone").
The "issue" may be that partition order is still not entirely "kosher" as the order in slots is not the same as on disk.
If the image still becomes corrupted it will be needed to slightly shrink (or move the beginning of the EF00 one to sector 34).
Disk \\.\Physicaldrive8: 2097152 sectors, 1024.0 MiB Logical sector size: 512 bytes Disk identifier (GUID): C556E9BF-F1D5-412E-A1E2-9784B8B16E38 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 2097118 Partitions will be aligned on 2-sector boundaries Total free space is 254053 sectors (124.0 MiB) Number Start (sector) End (sector) Size Code Name 1 64 8191 4.0 MiB EF00 2 34 41 4.0 KiB EB00 Haiku BFS 3 42 49 4.0 KiB EB00 Haiku BFS 4 50 57 4.0 KiB EB00 Haiku BFS 5 8224 520191 250.0 MiB 0700 6 520224 1032191 250.0 MiB 0700 7 1032224 1257471 110.0 MiB FC00 8 1257504 1843199 286.0 MiB 0700
Attached Files
#14
Posted 01 January 2017 - 06:49 PM
First part of the test : i replaced the sectors in the "bad" image and it now boots OK.
So the fix works.
#15
Posted 01 January 2017 - 07:18 PM
Second part of the test : i booted winpe with my "known to be working disk", could see my disk, browse partitions, etc.
Rebooted directly on that disk, and it is still booted fine whereas I know from previous experience that this would mess up my GPT disk.
So what does it tell us?
-Windows will rewrite the GPT partition table if it is not up to his taste? i.e empty entries will be removed?
-Workaround this behavior consists in rewriting the partition table (main and backup) with dummy entries with type = "Haiku BFS" ?
That windows behavior is really nasty IMHO.
Is that a bug from Windows, i.e not accepting a GPT "feature" (empty entries) or is that a bug from vmware i.e creating a non standard GPT partition table?
#16
Posted 01 January 2017 - 07:46 PM
Cannot say which is the culprit, most probably it is a 50/50 situation.
Surely it makes little sense to have "empty partition slots" (but this is not a good reason for Windows to attempt to "compact" the partition table)
The EB00 "Haiku BFS" is just an "unusual" ID, there are so many of them, a few examples:
Command (? for help): l 0700 Microsoft basic data 0c01 Microsoft reserved 2700 Windows RE 3000 ONIE boot 3001 ONIE config 4100 PowerPC PReP boot 4200 Windows LDM data 4201 Windows LDM metadata 7501 IBM GPFS 7f00 ChromeOS kernel 7f01 ChromeOS root 7f02 ChromeOS reserved 8200 Linux swap 8300 Linux filesystem 8301 Linux reserved 8302 Linux /home 8400 Intel Rapid Start 8e00 Linux LVM a500 FreeBSD disklabel a501 FreeBSD boot a502 FreeBSD swap a503 FreeBSD UFS a504 FreeBSD ZFS a505 FreeBSD Vinum/RAID a580 Midnight BSD data a581 Midnight BSD boot a582 Midnight BSD swap a583 Midnight BSD UFS a584 Midnight BSD ZFS a585 Midnight BSD Vinum a800 Apple UFS a901 NetBSD swap a902 NetBSD FFS a903 NetBSD LFS a904 NetBSD concatenated a905 NetBSD encrypted a906 NetBSD RAID ab00 Apple boot af00 Apple HFS/HFS+ af01 Apple RAID af02 Apple RAID offline af03 Apple label af04 AppleTV recovery af05 Apple Core Storage be00 Solaris boot bf00 Solaris root bf01 Solaris /usr & Mac Z bf02 Solaris swap bf03 Solaris backup bf04 Solaris /var bf05 Solaris /home bf06 Solaris alternate se bf07 Solaris Reserved 1 bf08 Solaris Reserved 2 bf09 Solaris Reserved 3 bf0a Solaris Reserved 4 bf0b Solaris Reserved 5 c001 HP-UX data c002 HP-UX service ea00 Freedesktop $BOOT eb00 Haiku BFS ed00 Sony system partitio ed01 Lenovo system partit Press the <Enter> key to see more codes: ef00 EFI System ef01 MBR partition scheme ef02 BIOS boot partition fb00 VMWare VMFS fb01 VMWare reserved fc00 VMWare kcore crash p fd00 Linux RAID
most probably *any*, including the MS ones such as 0700 would do.
But besides the "shift" or "compacting" of the slots, the "bad" GPT is actually "bad", meaning that some fields in the GPT structures are not correctly written (the backup EFI PART is different from "main", specifically it seems like the "bad" GPT backup has been NOT modified to reflect the changes in the main one, as a matter of fact the bad_last34.txt remains the same as the good_last34.txt) which is IMHO a rather "major" issue.
Besides, there is at least another issue:
Disk \\.\Physicaldrive9: 2097152 sectors, 1024.0 MiB Logical sector size: 512 bytes Disk identifier (GUID): C556E9BF-F1D5-412E-A1E2-9784B8B16E38 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 2096606 Partitions will be aligned on 32-sector boundaries Total free space is 253565 sectors (123.8 MiB) Number Start (sector) End (sector) Size Code Name 1 64 8191 4.0 MiB EF00 2 8224 520191 250.0 MiB 0700 3 520224 1032191 250.0 MiB 0700 4 1032224 1257471 110.0 MiB FC00 5 1257504 1843199 286.0 MiB 0700 Command (? for help): v Caution: The CRC for the backup partition table is invalid. This table may be corrupt. This program will automatically create a new backup partition table when you save your partitions. Problem: The secondary header's self-pointer indicates that it doesn't reside at the end of the disk. If you've added a disk to a RAID array, use the 'e' option on the experts' menu to adjust the secondary header's and partition table's locations. Identified 2 problems!
Again, last usable sector is (should be) total size - 34, while the "bad" GPT adds a 512 sectors "no man's land".
It could be a bug in Windows due to the (too small) size of the image, I wouldn't be surprised if with larger images it would work fine.
Wonko
#17
Posted 01 January 2017 - 08:10 PM
Well, thanks a lot for your time and expertise (again).
I now have a pretty good understanding of the situation and learned a few things around GPT.
Vmware coming up with an exotic GPT setup and Microsoft arbitrarily attempting to fix this setup is definitely not a good match.
Furthermore, that discussion may be helpful to others and not only related to Vmware.
In the long run, I may adapt clonedisk to this behavior (like not put the target disk offline before restoring which may prevent windows from "touching" the disk).
Thanks !
#18
Posted 01 January 2017 - 08:45 PM
Could it be something connected to the way the image is mounted?
Maybe the "full" size is wrognly accessed.
What is on the "bad" image in the 34 sectors following sector #2096606?
Likely there is the actual "backup" of the "bad" GPT , and if this is the case it means that *somehow the whole stuff was triggered by a size access issue.
I.e. the whole thing *somehow* has been mounted as having 2096640 sectors.
Then the Windows would not find the backup of the GPT, assume that it has been corrupted and - since there are holes in the partition table order - decides to optimize it while re-writing the backup GPT.
"Wrong", but starting to make (a little) sense.
Wonko
#19
Posted 01 January 2017 - 09:29 PM
I no longer have a bad image since i fixed it with your magical 34 sectors
Thus I may re start from scratch next week.
The image was never mounted (imdisk or else).
The source disk was generated by Vmware ESXI installer (the same installer that leaves empty entries in the partition table).
The "good" source image was obtained from converting a vmdk to a raw image (using libvmdk) and was left untouched since then.
The "bad" source image was obtained from a converting a vmdk to a raw image (using libvmdk) after it was "corrupted" by a winpe (booting within the VM).
"bad" and "good" come from the same source (virtual) disk.
Note that it seems to be a known issue on vmware forum that windows will mess up the vmware / gpt partition table : is it triggered by a missing/bad backup table or because windows does not like empty entries in a GPT table, I cannot tell.
I would be tempted to go for the second as I am pretty sure I have dealt (without issues) in the past with images without a backup table (but it was a MBR image, not a GPT).
It seems that windows behaves differently with GPT (more aggressive) compared to MBR.
Edit:
just in case, attached a screenshot of the "bad" image at position 2096606+34 sectors i.e 3FFBBC00+4400=3FFC0000
#20
Posted 02 January 2017 - 11:29 AM
Yep that is the good (and the bad) of GPT vs. MBR (and viceversa).
MBR has NO "backup", you lose first sector and it is game over.
But it is of course usually simple to rebuild the partition table contents in this case.
GPT not only has a backup, but the "main" table has besides an inner checksum, a field that tells you where the backup is:
http://thestarman.na...asm/mbr/GPT.htm
(offset 0x20 of the main EFI PART)
Which of course is one of the stupidest things that one could insert in a specification, if you have the "main" table you don't need the backup, and if you don't have it (or it is corrupted) there are little chances that by sheer magic the particular field at offset 0x020 is good, and even if it is, if any other part of the table is corrupted the checksum won't verify, an so you cannot trust it.
On the other hand, there is also this provision that the backup table should be put at the very end of the device (which actually in itself makes a lot of sense) and that all partitions should be between the "main" and "backup" tables:
https://en.wikipedia...able_Scheme.svg
So what may have happened is that for *whatever* reasons the Windows accessed the image without loading the last 512 sectors (why?), found out that there was no backup table, panicked and rebuilt the backup table and while at it reordered/compacted the used slots.
What has to be found out is why the image was loaded/accessed with a smaller size.
And no, rest assured Windows can be as stupid and as aggressive with MBR's also, see:
http://reboot.pro/to...itioning-issue/
Wonko
1 user(s) are reading this topic
0 members, 1 guests, 0 anonymous users