Jump to content











Photo
- - - - -

How to super-compress raw image without corrupting it?


  • Please log in to reply
10 replies to this topic

#1 AnonVendetta

AnonVendetta

    Silver Member

  • Advanced user
  • 655 posts
  • Location:A new beginning.....
  • Interests:Self-development, computing

Posted 30 January 2018 - 10:44 AM

I have a raw image of my Android phone's /system partition, captured with dd from TWRP recovery:

 

dd if=/dev/block/platform/soc/624000.ufshc/by-name/system of=/external_sd/system.img

 

This was captured after doing a restore of stock firmware from ZTE's website, so it is 100% official and unmodified.

 

I would like to get a high compression rate on this image to obtain a smaller file size (it's about 5GB now), but without corruption. The resulting file will also be written back to the partition with dd, after it is zeroed out.

 

The big issue is that Android OTA updates are done on a block/sector level, and the system partition is checked before an update is applied. If anything is different than what it expects to see (i.e. user modifications with root access or TWRP or whatever), the update will fail. Android also performs dm-verity checks (by default, which can be disabled with hackery) on this partition at bootup, via hash checking, again if anything is amiss then the device wont boot. So I need it to be an exact match to what is expected.

 

Thanks!



#2 erwan.l

erwan.l

    Gold Member

  • Developer
  • 2343 posts
  • Location:Nantes - France
  •  
    France

Posted 30 January 2018 - 12:23 PM

If under linux, you could compress/decompress it to tgz (or tar.gz) on the fly.

 

Under windows, for storage purposes, you could use any compression tool but would probably have to do it (reading/writing to disk) several manual steps.



#3 AnonVendetta

AnonVendetta

    Silver Member

  • Advanced user
  • 655 posts
  • Location:A new beginning.....
  • Interests:Self-development, computing

Posted 30 January 2018 - 12:54 PM

@erwan.l: I did say it would have to be written back with dd, so I'm not sure tgz or tar.gz will work here. Unless dd can handle those, not sure.



#4 erwan.l

erwan.l

    Gold Member

  • Developer
  • 2343 posts
  • Location:Nantes - France
  •  
    France

Posted 30 January 2018 - 01:08 PM

i am not a linux guru but this should workd (writing to disk) : tar xzOf disk.tar.gz | dd of=/dev/sdb bs=1M .

this is what i meant by "on the fly".



#5 AnonVendetta

AnonVendetta

    Silver Member

  • Advanced user
  • 655 posts
  • Location:A new beginning.....
  • Interests:Self-development, computing

Posted 30 January 2018 - 03:48 PM

dd is a native command in TWRP, I don't think tar is.



#6 cdob

cdob

    Gold Member

  • Expert
  • 1402 posts

Posted 30 January 2018 - 05:23 PM

Tar seems to be known at TWRP.
https://twrp.me/faq/...exclusions.html

with "tar" when your internal storage is mounted in TWRP).


Can you test tar at used TWRP version?

TWRP installs zip archives too.
Is zip and unzip included too? At *nix both works on the fly too and supports piping.

#7 erwan.l

erwan.l

    Gold Member

  • Developer
  • 2343 posts
  • Location:Nantes - France
  •  
    France

Posted 30 January 2018 - 06:25 PM

from an phone/android platform, options are limited : cpu, memory, available softwares ...

and it even more challenging from twrp which is a stripped out environement.

 

from a generic linux, you can also do : dd if=/dev/sda | gzip -1 - | ssh user@local dd of=image.gz

bzip2 (or any command line compression tool for that matter) is another alternative to gzip.

you can also combine dd with netcat (instead of ssh).

 

EDIT:

another alternative using zlib : dd if=if=/dev/sda  | openssl zlib -d > image.tar



#8 erwan.l

erwan.l

    Gold Member

  • Developer
  • 2343 posts
  • Location:Nantes - France
  •  
    France

Posted 30 January 2018 - 06:37 PM

A possibly nice article touching a similar topic : http://tadej.ja.nez....age-backup.html .



#9 alacran

alacran

    Silver Member

  • .script developer
  • 648 posts
  •  
    Mexico

Posted 31 January 2018 - 05:23 AM

Use wimlib-imagex ( Just Google for it) and select -solid to get the highest compression rate, there is Windows and Linux versions, This is what I use for my Win. OS backups, never had a problem when restore.



#10 erwan.l

erwan.l

    Gold Member

  • Developer
  • 2343 posts
  • Location:Nantes - France
  •  
    France

Posted 31 January 2018 - 08:53 AM

I am out of this thread : insulting people, whatever the context is, goes against my values.



#11 AnonVendetta

AnonVendetta

    Silver Member

  • Advanced user
  • 655 posts
  • Location:A new beginning.....
  • Interests:Self-development, computing

Posted 31 January 2018 - 10:28 AM

OK, bye then, see you later, the French won't be missed by anyone anyway.






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users