42 replies to this topic

[steve6375]

I think I have found a bug in grub4dos on NTFS filesystems

 

The problem only occurs if the TARGET file is fragmented

 

I used

 

dd if=()/file1.cfg  of=()/file2.cfg

 

dd reported that all of file1.cfg was copied to file2.cfg  (which was larger than file1.cfg) and no error

 

However, on inspection, only the first 0x2000 bytes were copied  (i.e. 2 clusters ??)!

 

Now I have copied very large files to another Contigous file and that has worked (I can copy a 400MB fragmented ISO to a contiguous 500MB file and that works).

 

When I make file2.cfg contiguous, then dd works correctly. I have not tested for this bug on FAT32 filesystems.

 

Can anyone confirm this please? You need to make sure your target file is NOT contguous to test it.

 

 

 

A second bug was to do with SplashImage which I have reported here.

[tinybit]

Can this work? Use the blocklist notation to replace the NTFS filename mentioned.

[steve6375]

Not sure, but it sounds more likely to corrupt files using the (hd0,0)x+y notation as surely that would not follow any file cluster  arrangement (i.e. it would overwrite clusters not belonging to the file if the file was not contiguous)?

 

P.S. I reported a bug with exFAT also - partnew (hd0,3) 0x0 /xx.img  returns 'file not contiguous' if the file is over 4GB. Files under 4BG are OK and work. (all files are contiguous as reported by blocklist and WinContig)..

[tinybit]

Not sure, but it sounds more likely to corrupt files using the (hd0,0)x+y notation as surely that would not follow any file cluster  arrangement (i.e. it would overwrite clusters not belonging to the file if the file was not contiguous)?

 

 

You use "(hd0,0)x1+y1,x2+y2,...,xn+yn" just as the blocklist command reported for the file, and it is equivalent to the filename notation.

[steve6375]

I have managed to make the problem reproducible

 

see https://code.google....s/detail?id=193

 

using blocklist for dd seems to work OK

 

 

 

Attached Thumbnails

• 

[steve6375]

I have repeated the test on a FAT32 formatted drive and although I got 6 fragmented Sn files, none of them gave the problem. 

[steve6375]

Issue 172 which I reported previously

 

cat --hex (md)0x4e+100 > /s1
cat --hex --length=0x100 --skip=0x1f80 /s1

 

show same problem on target files which are fragmented in same way. (NTFS only)

 

blocklist /s1

(hd0,0)1044904+8,6004392+93

 

fails

 

blocklist /bigger

(hd0,0)1047112+8,600436+56,6004944+40,6002526+56,6005744+43

 

fails

 

A good way to make a fragmented file is to use Windows console command

copy /b s+s s1

[tinybit]

Try this build for testing the NTFS issue.

 

 

Attached Files

•  grub4dos-0.4.5c-2014-08-12.7z   254.47KB   422 downloads

[steve6375]

Seems to be fixed!       well done!

[steve6375]

Latest 0.4.5c version is here http://grub4dos.chen...ries/downloads/

 

Note that this fixes a serious NTFS bug, so I recommend everyone to use the latest version - it may fix some 'strange' issues which seemed to go away if you reformat the drive...

[sixcentgeorge]

does your grub4dos is able to handle gzip files with size bigger than 4 Go and copy them in ram ?

[steve6375]

It's not my grub4dos - the developers are chenall, tinybit + others.

 

Did you report the issue? Which one is it?

https://code.google....all/issues/list

[sixcentgeorge]

it is an old bug , i had it myself withsome vhd for win7 in ram , others reported it there .

so you do not have troubles with vhd gzipped and then loaded in ram by gru4dos ?

[steve6375]

I never tested that - if you find a bug , please report it on the site as an issue and give details - then it will be fixed.

[tinybit]

grub4dos does not support gzip files with decompressed size greater than or equal to 4G.

 

The format of gzip is less recommended than lzma format. You should use lzma instead of gzip.

 

Note that I don't know if the lzma would work fine for files larger than 4G. I only know that the "large" gzip will not work with grub4dos, and I don't think there would be a fix in the near future.

[steve6375]

@tinybit

Did you see issue #198 I reported? Issue 198: all grub4dos variables lost using some AMI BIOSes

https://code.google....s/detail?id=198

 

The user who found the issue is still doing some more testing for me, but this could explain why several people have said that 'Easy2Boot doesn't work' and just given up on it.

 

There is some weird BIOS + grub4dos issue going on and it seems to be with AMI BIOSes.

 

Any idea what could be causing this?

[sixcentgeorge]

grub4dos does not support gzip files with decompressed size greater than or equal to 4G.

 

The format of gzip is less recommended than lzma format. You should use lzma instead of gzip.

 

Note that I don't know if the lzma would work fine for files larger than 4G. I only know that the "large" gzip will not work with grub4dos, and I don't think there would be a fix in the near future.

i wonder if i should use a special parameter when file is using the lzma ?

my vhd is about 7Go , after using lzip.exe with best compression , i got a file of 1.6Go but latest grub4dos does not recognize it as lzma and think it is a vhd because it applies default setting for heads and cylinders....as a result it copies but not decompress the file in ram ...so boot does not work .

i used lzip from there : http://lzip.nongnu.org/lzip.html

may be i should use a file made by 7zip , i have no idea what setting i should use : zip or 7zip with lzma ?

[steve6375]

download lzma.exe from https://code.google....l?name=lzma.exe
usage:
                lzma.exe e  BlueLight.xpm  BlueLight.xpm.lzma

 

You may need to use an .lzma file extension  (unsure???)

[tinybit]

@steve

 

I cannot deal with the variables issue. I will tell chenall about it.

 

@sixcentgeorge

 

check the lzma file header:

 

offset 00, one byte, should be less than or equal to 0xE0.

offset 01 to 04, four bytes(DWORD), must be a power of 2 to work with grub4dos, that is, 2^n.

offset 05 to 0C, eight bytes(QWORD), must not be (-1) to work with grub4dos, that is, must not be 0xFFFFFFFFFFFFFFFF.

 

Reportedly 7zip works fine and creates lzma files compatible with grub4dos. EDIT: it was misleading, sorry.

 

The .lzma filename extension is not required.

[sixcentgeorge]

i tested 7zip and lzma from steve6375 that seems to be the 9.20 from sdk : http://www.7-zip.org/sdk.html ; plus the 9.22 beta .

 

 

my vhd is 6.815.745 kb

with lzma 9.20 and 9.22 , i get files of same size : 1.731.152 kb ; i did 3 tests : 2 with 9.22 and multithread or only one , one with 9.20 after i found that grub4dos was not reading well the lzma

with lpzip file is 1.643.989 kb

with 7z using zip base for settings i got a file of 1.319.858 kb .

the file with 7 zip was made , like with lzma.exe , in one hours but using 11 Go of memory while lzma used only 100 Mo .

the 7zip file was not read by grub4dos

the 9.22 exe lzma create a file but some parts are "missing" like files in the quick launch bar . with 7zip , files are correctly compressed

 

 

same goes with start menu

 

 

there is a lot of improvements between lzma from 9.20 and what can do 7zip , the files  size difference is 324.131 kb : that gives  25 % of reduction ;']

lets hope you will be able to update grub4dos or explain how to use 7 zip to set right settings

[steve6375]

I don't think 7zip can make .lzma files that grub4dos can understand.

7Zip can decompress lzma files made with lzma.exe  but it cannot make a .lzma file.

I don't know why lzma.exe is not working correctly.

 

Did you actually successfully boot any lzma file with grub4dos?

[sixcentgeorge]

the file you gave is the only one that create well read file by grub4dos .

that would be cool if grub4dos could read 7zip zip file made with lzma

Edited by sixcentgeorge, 24 August 2014 - 01:46 PM.

[tinybit]

I am so sorry. I never created lzma files, and my memory was at fault. Sorry again for having wasted a lot of your time. Let's suppose the working software is the mentioned lzma.exe. Thanks, Steve6375.

[steve6375]

Not sure, but it sounds more likely to corrupt files using the (hd0,0)x+y notation as surely that would not follow any file cluster  arrangement (i.e. it would overwrite clusters not belonging to the file if the file was not contiguous)?

 

P.S. I reported a bug with exFAT also - partnew (hd0,3) 0x0 /xx.img  returns 'file not contiguous' if the file is over 4GB. Files under 4BG are OK and work. (all files are contiguous as reported by blocklist and WinContig)..

The exFAT bug should be fixed now

see https://code.google....s/detail?id=194

[Icecube]

It would be better to use xz instead of lzma:

 

 

The .xz file format is a container format for compressed streams. There are no archiving capabilities, that is, the .xz format can hold only a single file just like the .gz and .bz2 file formats used by gzip and bzip2, respectively.

Compared to a few other popular stream compression formats, the .xz format provides a couple of advanced features. At the same time, it has been kept simple enough to be usable in many embedded systems. Here is a summary of the features:

 

• Streamable: It is always possible to create and decompress .xz files in a pipe; no seeking is required.
• Random-access reading: The data can be split into independently compressed blocks. Every .xz file contains an index of the blocks, which makes limited random-access reading possible when the block size is small enough.
• Multiple filters (algorithms): It is possible to add support for new filters, so no new file format is needed every time a new algorithm has been developed. Developers can use a developer-specific filter ID space for experimental filters.
• Filter chaining: Up to four filters can be chained, which is very similar to piping on the UN*X command line. Chaining can improve compression ratio with some file types. Different filter chain can be used for every independently compressed block.
• Integrity checks: Integrity of all headers is always protected with CRC32. The integrity of the actual data may be verified with CRC32, CRC64, SHA-256, or the check may be omitted completely. It is possible to add new integrity checks in future, but there is no possibility for developer-specific check IDs like there is for filter IDs.
• Concatenation: Just like with .gz and .bz2 files, it is possible to concatenate .xz files as is. The decompressor can decompress a concatenated file as if it was a regular single-stream .xz file.
• Padding: Binary zeros may be appended to .xz files to pad them to fill e.g. a block on a backup tape. The padding needs to be multiple of four bytes, because the size of every valid .xz file is a multiple of four bytes.

Once a new filter or integrity check has been added to the .xz file format specification, it won't be removed. This is to ensure that all .xz files, that use only the filters defined in the .xz file format specification, can always be decompressed in future.

New filters, integrity checks, or other additions to the .xz file format are unlikely to occur very often. Useless bloat can be avoided when new filters are added to the official list only when the new filters are clearly useful.

 

 

http://tukaani.org/xz/format.html

 

 

 

LZMA Utils are legacy data compression software with high compression ratio. LZMA Utils are no longer developed, although critical bugs may be fixed as long as fixing them doesn't require huge changes to the code.

Users of LZMA Utils should move to XZ Utils. XZ Utils support the legacy .lzma format used by LZMA Utils, and can also emulate the command line tools of LZMA Utils. This should make transition from LZMA Utils to XZ Utils relatively easy.

 

 

http://tukaani.org/lzma/

 

 

As far as I remember the xz file format has a "better header" than lzma with also a longer magic sequence.