Jump to content











Photo
- - - - -

Windows altering (and messing up) my GPT partition table behind the scene

windows gpt

  • Please log in to reply
19 replies to this topic

#1 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1918 posts
  • Location:Nantes - France
  •  
    France

Posted 01 January 2017 - 12:38 PM

Hi,
 
Lately, I have created an image of an (installed, not installable) esxi hypervisor.
This way I can at will write it to any media (usb key, harddrive ...) and directly boot to an already installed esxi hypervisor without the hassle of installing it each time.
 
So far so good.
 
But when I burn this image under windows (using clonedisk), i could no longer boot it ("no hypervisor found" error) whereas burning under linux (with dd) was OK.
I first thought clonedisk was the issue except that simply inserting a working usb key under windows has the same effect : my usb key would no longer work afterwards.
 
So it appears windows does change the GPT partition entry behind the scene.
Not only does it change some boot flags but it also reshuffles the whole partition table!.
I have attached the before/after 1512 first bytes where the changes occur (change the extension bad to dd or raw).
 
What would be the recommandation to work around this nasty windows behavior?
-prevent windows from mounting drives? i can consider it under a winpe environement but then will I be able to write to the media?
-add an option in clonedisk to leave the media offline when done writing?
-add a flag to the gpt partition table to tell windows to ignore it (does such thing exist)?
 
Note that my scenario involves vmware esxi but this is probably irrelevant the overall question : how can I ensure (in general) that windows will not modify my GPT partition table after I am done writing an image or when I insert such a  media?
 
Any idea/feedback welcome :)
 
Regards,
Erwan

Attached Files

  • Attached File  bad.txt   1.5KB   49 downloads
  • Attached File  good.txt   1.48KB   51 downloads


#2 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 13497 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 01 January 2017 - 02:17 PM

I am not sure to understand. :dubbio:

 

At first sight the "good" file is not " pure GPT", is a "hybrid" MBR/GPT.

 

WHEN does "Windows" (which version) alters its contents?

 

The "good" has seemingly only one partition, whilst the "bad" has seemingly 4 of them.

 

BOTH the "good" AND the "bad" have also NO disk signature (which more or less should mean that it has been "generated" by *something* but never mounted. :dubbio:

 

Most of the differences (once set aside the different number of partitions) are due to checksums, and this is more or less "normal".



"good":
Disk \\.\Physicaldrive8: 2097152 sectors, 1024.0 MiB
Logical sector size: 512 bytes
Disk identifier (GUID): C556E9BF-F1D5-412E-A1E2-9784B8B16E38
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 2097118
Partitions will be aligned on 64-sector boundaries
Total free space is 2088957 sectors (1020.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              64            8191   4.0 MiB     EF00

"bad":
Disk \\.\Physicaldrive9: 2097152 sectors, 1024.0 MiB
Logical sector size: 512 bytes
Disk identifier (GUID): C556E9BF-F1D5-412E-A1E2-9784B8B16E38
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 2096606
Partitions will be aligned on 32-sector boundaries
Total free space is 839261 sectors (409.8 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              64            8191   4.0 MiB     EF00
   2            8224          520191   250.0 MiB   0700
   3          520224         1032191   250.0 MiB   0700
   4         1032224         1257471   110.0 MiB   FC00

You need to provide some more context ...

How exactly the "good" one was created, which partitions it should contain, etc.

 

:duff:

Wonko



#3 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1918 posts
  • Location:Nantes - France
  •  
    France

Posted 01 January 2017 - 02:33 PM

Hi Wonko,

 

I installed esxi in a vmware machine (1gb disk) using ESXI Installable ISO.

Once installed and tested it was working fine (i.e I rebooted the vmware guest without the ISO), I converted the VMDK to a RAW disk.

And then finally wrote this RAW disk to a USB key.

 

If I insert this USB key under windows (or mount the image under windows), the GPT partition table gets modified.

I can also add that if I boot a winpe in my vmware guest (where the working 1gb disk sits), at next reboot, i can no longer boot on the disk and will again get "hypervisor not found".

So whether I mount the image under my full windows or boot my vm with a winpe, the GPT partition gets messed up.

 

good.dd comes from the original raw disk (which boots fine)

bad.dd comes from the same raw disk after it has been "touched" by windows (which no longer boots).

 

Regards,

Erwan



#4 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1918 posts
  • Location:Nantes - France
  •  
    France

Posted 01 January 2017 - 02:35 PM

attaching a view from DMDE from the good image.

 

 

Attached Thumbnails

  • esxi_good.png


#5 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1918 posts
  • Location:Nantes - France
  •  
    France

Posted 01 January 2017 - 02:40 PM

and a view from DMDE from the bad image (i.e after it has been "touched" / seen by windows).

Attached Thumbnails

  • esxi_bad.png


#6 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 13497 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 01 January 2017 - 03:09 PM

Well, what you posted as "good" has a single partition, even if seen in a plain hex editor, maybe you posted the "wrong" file.

 

Anyway the "issue" is probably in:

"good": First usable sector is 34, last usable sector is 2097118

vs.

"bad": First usable sector is 34, last usable sector is 2096606

 

The first is OK, since the image is 2097152 sectors - 34 = 2097118

whilst the second makes little sense, as 2097152 - 512 - 34=2096606

 

this 512 sectors difference is "queer".

 

As well it is "queer" (though it may be due to the small size of the image) that the first partition starts at LBA 64 (as it normally would stat at 2048).

 

Additionally, both have "unbalanced CHS/LBA values, and you will have to explain how/why it is a "Hybrid MBR" (i.e. containing boot code).

 

Please do recheck the data you posted and please provide also the last three sectors of the image (besides the first three) so that I don't have to re-build the backup GPT (which should be OK, but that maybe introduces some variations, i.e.:
good_first3.txt

good_last3.txt

bad_first3.txt

bad_last3.txt

 

 

:duff:

Wonko



#7 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1918 posts
  • Location:Nantes - France
  •  
    France

Posted 01 January 2017 - 03:19 PM

3 first and 3 last sectors for the good image (booting fine) and the bad image ("hypervisor not found").

3 sectors = 0x600 / 1536 bytes.

 

Unique and only difference between the 2 images : windows has seen / touched the second one making it a bad image.

 

Some extra hints from vmware forum : 

-https://kb.vmware.co...ernalId=2012022

To resolve this issue, use a Live Linux CD of your choice. Relabel the local datastore and create a partition table with one partition. Then re-install ESXi.

-https://communities....tart=0&tstart=0

Attached Files



#8 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 13497 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 01 January 2017 - 03:59 PM

Well, the "good" still has a single partition whilst the "bad" still has 4 of them.

 

The DMDE screenshot on your post #4 cannot reasonably be coming from this same "good" image, there is simply NO info on first sector of the GPT partition table.

 

Try again, this time taking 34 sectors from the beginning and 34 sectors from the end of each image.

I am starting to suspect that (for *whatever* reasons) the "good" image has the other three partitions on "slots" - say - #35, #42 and #110 :w00t: and again (for *whatever* reasons) *something* re-orders them (possibly creating some other issue while doing that).

The report here:

https://communities....tart=0&tstart=0

talks about a single empty partition inserted, but if his was the case, sector LBA 2 of "good" image would still contain 3 partition entries (besides the inserted empty one, as each partition entry is 128 bytes)

 

:duff:

Wonko



#9 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1918 posts
  • Location:Nantes - France
  •  
    France

Posted 01 January 2017 - 04:17 PM

Look at these 2 screenshots coming from the good and bad images.

The same images i used to extra the 3 first/last sectors.

 

The good image has 5 entries (other 3 blocks) but with 3 empty/dummy ones stuff in the middle : #1, #5, #6, #7, #8.

The bad image seems to have been reorganised (thank you windows...) with consecutive entries other 2 blocks : #1, #2, #3, #4, #5

 

Number of partitions and GUID are the same but not the partition ID's / setup differ.

 

What tool are you using to parse the GPT partition table?

Looks like your tool is not parsing correctly the GPT partition in the "good" image?

I will shortly attached the 34 first/last sectors of each image.

 

The error below seems to indicate that esxi is expecting a bank6 = entry/partition #6 which no longer exist since windows are "kindly" reorganized it all for me :(

 

Question remains : why the hell is windows bothering to alter/mess up my partition table?! and how to prevent that (i dont need to fix it as I can easily rewrite my raw image to my media).

Probably it believes this setup is wrong but at least it should ask me if I want to fix it or not.

 

snap5186.png

 

Edit : I believe your tool gets "fooled" by the "good" GPT setup and only read one entry because of the 3 empty/dummy entries right after entry #1.

Therefore it ignores #5,#6,#7,#8.

Which may indicate that vmware setup breaks the GPT standards : entries should be consecutive?

Attached Thumbnails

  • good.png
  • bad.png


#10 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1918 posts
  • Location:Nantes - France
  •  
    France

Posted 01 January 2017 - 04:30 PM

Attached File  bad_last34.txt   17KB   39 downloadsAs requested, 34 sectors first/last on both images.

 

From thread here, i believe the issue lays there : "Second, I see that ESXi 5.0 leaves an empty partition entry slot(for untold reason) , and Windows shifts the entries behind to fill the gap after GPT modification. So ESXi's boot bank BANK6 gets screwed. I think a VMware so-called BANK is a synonym for a partition entry."

Attached Files



#11 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 13497 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 01 January 2017 - 04:39 PM

The good image has 5 entries (other 3 blocks) but with 3 empty/dummy ones stuff in the middle : #1, #5, #6, #7, #8.

Sure :), but you provided only 3 sectors:
#1 the MBR
#2 the EFI PART
#3 the FIRST sector ONLY of the actual partition table that (since each slot is 128 bytes) contains slots #1, #2, #3 and #4
 

What tool are you using to parse the GPT partition table?

Looks like your tool is not parsing correctly the GPT partition in the "good" image?

Rest assured my tool parses allright what is there, (but it cannot invent what it isn't there).
If slots #2, #3 and #4 are empty, and I have not the sector containing slot #5 onwards, the only partition I can see is the one in slot #1.

I'll check the 34 sectors files.

As a first guess, probably we can fill the empty slots with "fake" partitions so that everything is "kosher".

:duff:
Wonko

P.S.: The good_first34.txt file is only partial, please try again.

#12 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1918 posts
  • Location:Nantes - France
  •  
    France

Posted 01 January 2017 - 04:44 PM

arg, sorry, i messed up ....

now attached the right file to the previous post.



#13 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 13497 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 01 January 2017 - 05:18 PM

Try the attached.
Make a copy of the "good" image.
Overwrite first and last 34 sectors with the files attached.
Attempt repeating the *whatever* sequence that corrupted the "original" good image.
I inserted three (slots #2, #3 and #4) 4 kb partitions (very, very small ;)) of the Haiku BFS type (which hopefully Windows and *anything else* but Haiku should "leave alone").
The "issue" may be that partition order is still not entirely "kosher" as the order in slots is not the same as on disk.
If the image still becomes corrupted it will be needed to slightly shrink (or move the beginning of the EF00 one to sector 34).
Disk \\.\Physicaldrive8: 2097152 sectors, 1024.0 MiB
Logical sector size: 512 bytes
Disk identifier (GUID): C556E9BF-F1D5-412E-A1E2-9784B8B16E38
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 2097118
Partitions will be aligned on 2-sector boundaries
Total free space is 254053 sectors (124.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              64            8191   4.0 MiB     EF00
   2              34              41   4.0 KiB     EB00  Haiku BFS
   3              42              49   4.0 KiB     EB00  Haiku BFS
   4              50              57   4.0 KiB     EB00  Haiku BFS
   5            8224          520191   250.0 MiB   0700
   6          520224         1032191   250.0 MiB   0700
   7         1032224         1257471   110.0 MiB   FC00
   8         1257504         1843199   286.0 MiB   0700

Attached Files



#14 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1918 posts
  • Location:Nantes - France
  •  
    France

Posted 01 January 2017 - 06:49 PM

First part of the test : i replaced the sectors in the "bad" image and it now boots OK.

So the fix works.



#15 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1918 posts
  • Location:Nantes - France
  •  
    France

Posted 01 January 2017 - 07:18 PM

Second part of the test : i booted winpe with my "known to be working disk", could see my disk, browse partitions, etc.

Rebooted directly on that disk, and it is still booted fine whereas I know from previous experience that this would mess up my GPT disk.

 

So what does it tell us?

-Windows will rewrite the GPT partition table if it is not up to his taste? i.e empty entries will be removed?

-Workaround this behavior consists in rewriting the partition table (main and backup) with dummy entries with type = "Haiku BFS" ?

 

That windows behavior is really nasty IMHO.

Is that a bug from Windows, i.e not accepting a GPT "feature" (empty entries) or is that a bug from vmware i.e creating a non standard GPT partition table?



#16 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 13497 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 01 January 2017 - 07:46 PM

Cannot say which is the culprit, most probably it is a 50/50 situation.

Surely it makes little sense to have "empty partition slots" (but this is not a good reason for Windows to attempt to "compact" the partition table)

The EB00 "Haiku BFS" is just an "unusual" ID, there are so many of them, a few examples:
 



Command (? for help): l
0700 Microsoft basic data  0c01 Microsoft reserved    2700 Windows RE
3000 ONIE boot             3001 ONIE config           4100 PowerPC PReP boot
4200 Windows LDM data      4201 Windows LDM metadata  7501 IBM GPFS
7f00 ChromeOS kernel       7f01 ChromeOS root         7f02 ChromeOS reserved
8200 Linux swap            8300 Linux filesystem      8301 Linux reserved
8302 Linux /home           8400 Intel Rapid Start     8e00 Linux LVM
a500 FreeBSD disklabel     a501 FreeBSD boot          a502 FreeBSD swap
a503 FreeBSD UFS           a504 FreeBSD ZFS           a505 FreeBSD Vinum/RAID
a580 Midnight BSD data     a581 Midnight BSD boot     a582 Midnight BSD swap
a583 Midnight BSD UFS      a584 Midnight BSD ZFS      a585 Midnight BSD Vinum
a800 Apple UFS             a901 NetBSD swap           a902 NetBSD FFS
a903 NetBSD LFS            a904 NetBSD concatenated   a905 NetBSD encrypted
a906 NetBSD RAID           ab00 Apple boot            af00 Apple HFS/HFS+
af01 Apple RAID            af02 Apple RAID offline    af03 Apple label
af04 AppleTV recovery      af05 Apple Core Storage    be00 Solaris boot
bf00 Solaris root          bf01 Solaris /usr & Mac Z  bf02 Solaris swap
bf03 Solaris backup        bf04 Solaris /var          bf05 Solaris /home
bf06 Solaris alternate se  bf07 Solaris Reserved 1    bf08 Solaris Reserved 2
bf09 Solaris Reserved 3    bf0a Solaris Reserved 4    bf0b Solaris Reserved 5
c001 HP-UX data            c002 HP-UX service         ea00 Freedesktop $BOOT
eb00 Haiku BFS             ed00 Sony system partitio  ed01 Lenovo system partit
Press the <Enter> key to see more codes:
ef00 EFI System            ef01 MBR partition scheme  ef02 BIOS boot partition
fb00 VMWare VMFS           fb01 VMWare reserved       fc00 VMWare kcore crash p
fd00 Linux RAID

most probably *any*, including the MS ones such as 0700 would do.

 

But besides the "shift" or "compacting" of the slots, the "bad" GPT is actually "bad", meaning that some fields in the GPT structures are not correctly written (the backup EFI PART is different from "main", specifically it seems like the "bad" GPT backup has been NOT modified to reflect the changes in the main one, as a matter of fact the  bad_last34.txt remains the same as the good_last34.txt)  which is IMHO a rather "major" issue.

Besides, there is at least another issue:



Disk \\.\Physicaldrive9: 2097152 sectors, 1024.0 MiB
Logical sector size: 512 bytes
Disk identifier (GUID): C556E9BF-F1D5-412E-A1E2-9784B8B16E38
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 2096606
Partitions will be aligned on 32-sector boundaries
Total free space is 253565 sectors (123.8 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              64            8191   4.0 MiB     EF00
   2            8224          520191   250.0 MiB   0700
   3          520224         1032191   250.0 MiB   0700
   4         1032224         1257471   110.0 MiB   FC00
   5         1257504         1843199   286.0 MiB   0700

Command (? for help): v

Caution: The CRC for the backup partition table is invalid. This table may
be corrupt. This program will automatically create a new backup partition
table when you save your partitions.

Problem: The secondary header's self-pointer indicates that it doesn't reside
at the end of the disk. If you've added a disk to a RAID array, use the 'e'
option on the experts' menu to adjust the secondary header's and partition
table's locations.

Identified 2 problems!

Again, last usable sector is (should be) total size - 34, while the "bad" GPT adds a 512 sectors "no man's land". :ph34r:

 

It could be a bug in Windows due to the (too small) size of the image, I wouldn't be surprised if with larger images it would work fine. :dubbio:

 

 

:duff:

Wonko



#17 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1918 posts
  • Location:Nantes - France
  •  
    France

Posted 01 January 2017 - 08:10 PM

Well, thanks a lot for your time and expertise (again).

 

I now have a pretty good understanding of the situation and learned a few things around GPT.

 

Vmware coming up with an exotic GPT setup and Microsoft arbitrarily attempting to fix this setup is definitely not a good match.

Furthermore, that discussion may be helpful to others and not only related to Vmware.

 

In the long run, I may adapt clonedisk to this behavior (like not put the target disk offline before restoring which may prevent windows from "touching" the disk).

 

Thanks !



#18 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 13497 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 01 January 2017 - 08:45 PM

Could it be something connected to the way the image is mounted?

Maybe the "full" size is wrognly accessed.

 

What is on the "bad" image in the 34 sectors following sector #2096606?

 

Likely there is the actual "backup" of the "bad" GPT :dubbio:, and if this is the case it means that *somehow the whole stuff was triggered by a size access issue.

 

I.e. the whole thing *somehow* has been mounted as having 2096640 sectors.

 

Then the Windows would not find the backup of the GPT, assume that it has been corrupted and - since there are holes in the partition table order - decides to optimize it while re-writing the backup GPT. :unsure:

"Wrong", but starting to make (a little) sense.

 

:duff:

Wonko



#19 erwan.l

erwan.l

    Gold Member

  • Developer
  • 1918 posts
  • Location:Nantes - France
  •  
    France

Posted 01 January 2017 - 09:29 PM

I no longer have a bad image since i fixed it with your magical 34 sectors :)

Thus I may re start from scratch next week.

 

The image was never mounted (imdisk or else).

 

The source disk was generated by Vmware ESXI installer (the same installer that leaves empty entries in the partition table).

The "good" source image was obtained from converting a vmdk to a raw image (using libvmdk) and was left untouched since then.

The "bad" source image was  obtained from a converting a vmdk to a raw image (using libvmdk) after it was "corrupted" by a winpe (booting within the VM).

 

"bad" and "good" come from the same source (virtual) disk.

 

Note that it seems to be a known issue on vmware forum that windows will mess up the vmware / gpt partition table : is it triggered by a missing/bad backup table or because windows does not like empty entries in a GPT table, I cannot tell.

 

I would be tempted to go for the second as I am pretty sure I have dealt (without issues) in the past with images without a backup table (but it was a MBR image, not a GPT).

It seems that windows behaves differently with GPT (more aggressive) compared to MBR.

 

Edit:

just in case, attached a screenshot of the "bad" image at position 2096606+34 sectors i.e 3FFBBC00+4400=3FFC0000

Attached Thumbnails

  • bad_1073462272+34.png


#20 Wonko the Sane

Wonko the Sane

    The Finder

  • Advanced user
  • 13497 posts
  • Location:The Outside of the Asylum (gate is closed)
  •  
    Italy

Posted 02 January 2017 - 11:29 AM

Yep that is the good (and the bad) of GPT vs. MBR (and viceversa).

 

MBR has NO "backup", you lose first sector and it is game over.

 

But it is of course usually simple to rebuild the partition table contents in this case.

 

GPT not only has a backup, but the "main" table has besides an inner checksum, a field that tells you where the backup is:

http://thestarman.na...asm/mbr/GPT.htm

(offset 0x20 of the main EFI PART)

 

Which of course is one of the stupidest things that one could insert in a specification, if you have the "main" table you don't need the backup, and if you don't have it (or it is corrupted) there are little chances that by sheer magic the particular field at offset 0x020 is good, and even if it is, if any other part of the table is corrupted the checksum won't verify, an so you cannot trust it.

 

On the other hand, there is also this provision that the backup table should be put at the very end of the device (which actually in itself makes a lot of sense) and that all partitions should be between the "main" and "backup" tables:

https://en.wikipedia...able_Scheme.svg

 

So what may have happened is that for *whatever* reasons the Windows accessed the image without loading the last 512 sectors (why?), found out that there was no backup table, panicked and rebuilt the backup table and while at it reordered/compacted the used slots.

 

What has to be found out is why the image was loaded/accessed with a smaller size.

 

And no, rest assured Windows can be as stupid and as aggressive with MBR's also, see:
http://reboot.pro/to...itioning-issue/

 

 

:duff:

Wonko






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users