145 replies to this topic

[karyonix]

I have spent a few weeks implementing PAE support for GRUB4DOS.
It is based on grub4dos-0.4.4-2009-10-16-src.zip from http://nufans.net/grub4dos/ .

Modification:

• Implemented PAE support in int13_handler and in map, dd, cat commands.
• Added map option --mem-max=MAX and --mem-min=MIN.
  Allow user to prevent map --mem command from using memory address above MAX or memory address below MIN.
• Added map option --add-mbt= .
  If =0, map --mem will not add master boot track to memdrive.
  If =1, map --mem will add master boot track to memdrive.
  Changed default behavior to add master boot track only if image have BPB with recognized filesystem.
• Allow map to reuse memory above existing memdrives in same block.
• Changed probed_total_sectors value to use last partition ending instead of calculated from geometry.
  map --mem will not warn when image size is not multiple of cylinder size.
• Parse and print 64-bit integer.
• Parse number-of-sectors with K,M,G suffix.
• Display progress while reading large file.

I will test a little more and post it here tonight if no serious bug found.
--------

This is not official GRUB4DOS release. It is a modified version.
It is still in alpha stage, not well tested, may be buggy, or may cause loss of data.
Not recommended for inexperienced user.
-- removed due to serious bug found --
Bug: direct map does not work.

RAM above and below 4GB address are split to different blocks.
If you have 4GB RAM it may be split to 3.25GB (below 4GB address) and 0.75GB (above 4GB address).
If you have 6GB RAM it may be split to 3.25GB (below 4GB address) and 2.75GB (above 4GB address).
If you have 8GB RAM it may be split to 3.25GB (below 4GB address) and 4.75GB (above 4GB address).

Normally, map --mem will use the first block that have enough space.
To force mem drive to above 4GB address, use --mem-min= option.

map --mem-min=4G

map --mem (hd0,0)/hddimg1.img (hd1)

map --mem-min=0

map --mem (hd0,0)/hddimg2.img (hd2)

This modified GRUB4DOS does not support large mem drive that span across two blocks.

[tinybit]

good work.

builds under http://nufans.net/grub4dos/tinybit/ also can load image onto 4G+(but use 64-bit compared to PAE).

If you would like, you may give a patch to those versions.

[karyonix]

Today, map --mem command in my modified GRUB4DOS works well. I can access the drive above 4GB in Windows 7 RC 32-bit (with firadisk) and still have full 3.25GB low RAM for Windows to use.
But direct map command without --mem in my modified GRUB4DOS doesn't work it just keeps silent indefinitely.

I think that, instead of debugging my current code, it would be better to move on and try to make change base on tinybit's new versions and make a patch for latest version.

It's time to go to work now. I will do that after I come back from work this evening.

[Galapo]

I like the idea of adding a progress bar. Thanks for that.

Regards,
Galapo.

[karyonix]

The bug is because my move memory function do not deny write to destination address NULL or 0.
Now the bug has been fixed. It should work.
binary : grub4dos-0.4.4-pae1-2009-12-29.zip MF
source : grub4dos-0.4.4-pae1-2009-12-29-src.7z MF
It is still base on old GRUB4DOS version.
I will read tinybit's new version tonight.

@Galapo
I thought about progress bar but decided to just print progress in plain simple number.
They are almost functionally equivalent.

[ireneuszp]

In tinybit's new version
http://nufans.net/grub4dos/tinybit/

or grub4dos-chenall
http://nufans.net/grub4dos/chenall/
http://code.google.c.../downloads/list

gfxmenu doesn't work !!!

[Galapo]

I thought about progress bar but decided to just print progress in plain simple number.
They are almost functionally equivalent.

Gday karyonix,

For me, it doesn't matter whether you implement a progress bar or printed progress percentage. Sorry, I'd assumed by "Display progress" in your notes that you meant "progress bar". Printed progress is enough for me too. It is very helpful to see that an image is still being loaded when under usb 1 systems.

Thanks,
Galapo.

[tinybit]

In tinybit's new version
http://nufans.net/grub4dos/tinybit/

or grub4dos-chenall
http://nufans.net/grub4dos/chenall/
http://code.google.c.../downloads/list

gfxmenu doesn't work !!!

This was really a problem before the build 2009-12-20. But it should have been resolved since then.

Try the latest build 0.4.5a-2009-12-28. If the problem would go on as before, you may upload your menu.lst and your message file for us to reproduce the problem. Thanks.

[ireneuszp]

This was really a problem before the build 2009-12-20. But it should have been resolved since then.

Try the latest build 0.4.5a-2009-12-28. If the problem would go on as before, you may upload your menu.lst and your message file for us to reproduce the problem. Thanks.

Version 0.4.5a-2009-12-28 does not run completely.
for example

find --set-root --ignore-floppies /ntldr



Error 27: Unrecognized command

and gfxmenu doesn't work too!!

my menu.lst and gfxmenu files:
http://www.wikiforti...rub4dos-test.7z



and here is quemu grub4dos test:
http://www.wikiforti...emu-g4d-test.7z

extract and run grub4dos-test.cmd

Edited by ireneuszp, 30 December 2009 - 06:10 PM.

[tinybit]

＠ireneuszp

Thanks for your upload.

By tracing, I finally find out the hang is caused by the programming code in your message file.

If you can gain the source code of your message file, you may have a look at it and hopefully solve the problem.

The newish grub4dos versions use a new GDT layout. If the gfx message code blindly used the old GDT layout to swich between real mode and protected mode, it will hang for certain. The message code should build its own GDT table.

In addition, the newish grub4dos'es use part of memory in the physical address range 3M-4M as grub4dos's code space. If the gfx message code also (blindly)used this space, it will hang because of the conflict.

[karyonix]

@tinybit
I made a patch for your grub4dos-0.4.5a-2009-12-28-src .
(removed) patch Attached File patch.zip ( 7.07K ) Number of downloads: 14

Changes
map --add-mbt= option to be used with --mem. If =0 master boot track will not be added automatically.
map --top option to be used with --mem. map --mem will try to allocate memory at highest available address.
map --mem-max=, map --mem-min options to be used before map --mem. Allow user to manually limit range of address that map --mem can use.
safe_parse_maxint_with_suffix function parses K,M,G,T suffix after number.

I have not changed image loading part of map function and _read functions.
There is a problem. When map --mem allocate memory above 4GB. Disk image is not loaded into that address.

[tinybit]

currently only implemented 64-bit memmove function. It requires a 64bit CPU. if not, only loaded into 32bit address space. The function mem64 is in asm.S. It needs testing. If there are bugs, I hope you would fix.

[ireneuszp]

@ireneuszp

Thanks for your upload.

By tracing, I finally find out the hang is caused by the programming code in your message file.

If you can gain the source code of your message file, you may have a look at it and hopefully solve the problem.

The newish grub4dos versions use a new GDT layout. If the gfx message code blindly used the old GDT layout to swich between real mode and protected mode, it will hang for certain. The message code should build its own GDT table.

In addition, the newish grub4dos'es use part of memory in the physical address range 3M-4M as grub4dos's code space. If the gfx message code also (blindly)used this space, it will hang because of the conflict.

Repaired with version http://nufans.net/gr...-2009-12-30.zip

I changed also my file gfxmenu and all runs now, thanks and happy new year.

[karyonix]

There are multiple possible ways to implement reading/writing in memory above 4GB.

1. Use intermediate buffer in lower 4GB address space
It is the workaround I did in my previous PAE attempt.
All file read/write are done in lower 4GB address.
Use unmodified 32-bit fsys read functions.
grub_read will have to call memmove64 a few time to copy large block data between intermediate buffer and high-address buffer.

2. Modify all fsys read functions to use 64 bit address and use memmove64, memset64.
Processsor will have to switch between 32-bit flat protected mode and long mode (or maybe PAE) everytime memmove64/memset64 is called.

3. Another possibility is to keep paging enabled and let 32-bit fsys read functions operate in virtual address space.
grub_read map high-address buffer into 4GB virtual address apace.
fsys can access buffer easily using its virtual address.
fsys may call BIOS through devread/rawread/biosdisk functions.
If BIOS are called through real mode, then paging and old mapping should be restored when processor come back to protected mode before returning to fsys.
If the grub_read want to read/write more bytes than it can map at a time. It must split the request and call fsys read multiple times.

I like the third way. What do you think ?

[tinybit]

By your description, I am using method 2.

About method 3:

BIOS is frequently called: the timer, the keyboard, the screen.... all through BIOS. If for each call we have to restore paging once, I worry about the efficiency. And if we use long-mode, then the (frequent)switch between real-mode and long-mode is time-consuming.

Generally PAE could support 64G memory whereas long-mode will support much more memory. So the long-mode is preferable.

About method 2:

Processsor will have to switch between 32-bit flat protected mode and long mode (or maybe PAE) everytime memmove64/memset64 is called. Yes. But the switch is not frequently. It happens mainly on I/O, and only when address of 4G+ is needed to access.

About method 1:

This needs an intermediate buffer. It is the drawback.

[karyonix]

Early processor that support PAE (Pentium Pro) have only 36-bit physical address width.
According to Intel's document, PAE paging mode and IA-32e paging mode (including long mode) both have up to 52-bit physical address limited by MAXPHYSADDR (CPUID.80000008H:EAX[7:0]).

Long mode is still preferable when available. movsq in Long mode is faster than movsl.

[karyonix]

There is a bug about add_mbt processing in my previous patch. This patch should fix it.
Also added PAE paging code for grub_memmove/cmp/set64.
Currently, mem64 does not work with my Core 2 Duo PC. Legacy PAE paging will be useful for a while.

(removed) karyonix20100102_patch_for_grub4dos_0.4.5a_2010_01_02.diff.zip ( 5.27K ) Number of downloads: 20

[karyonix]

Hello.
I made some more modification.
- PAE support in int13_handler. I think it should also work in 32-bit processors with PAE support. However, I have only tested it in 64-bit Intel Core2 Duo.
- Support reading block-list larger than 4GB.
- Support reading non-compressed larger-than-4GB file in NTFS partition.
I installed 8GB RAM in my computer. The largest region seen in displaymem output is address=0x100000000 (4GB) length=0x130000000 (4.75GB). I loaded 4.74 GB image file with map --mem command successfully.

source diff file (removed) patch20100122.zip ( 13.08K ) Number of downloads: 13

[chenall]

patched.

http://grub4dos-chen...-2010-01-22.zip

[TheK]

I tested grub4dos-0.4.5a-2010-01-22. Mapping an uncompressed image works, but it faild to map gziped image to memory (--mem with and without --top). the same compressed image works with 0.4.4.


Tested on a AMD Phenom II system.

[ireneuszp]

patched.

http://grub4dos-chen...-2010-01-22.zip

@chenall in your version
http://grub4dos-chen...-2010-01-22.zip
splashimage doesn't work !!

in version grub4dos-0.4.5a-2010-01-21 works ok

[chenall]

@chenall in your version
http://grub4dos-chen...-2010-01-22.zip
splashimage doesn't work !!

in version grub4dos-0.4.5a-2010-01-21 works ok

did you used on ntfs?
maybe some bug on fsys_ntfs.c

http://bbs.znpc.net/...583...=1&page=4

[ireneuszp]

did you used on ntfs?

Yes, on ntfs

[chenall]

@karyonix

in fsys_ntfs.c
read_block func

ss = (ctx->target_vcn << log2_spc) + ctx->vcn_offset + nn;
ctx->target_vcn = ss >> log2_spc;
ctx->vcn_offset = ss & (spc-1);

[chenall]

http://grub4dos-chen...-2010-01-23.zip