The checksum "by the book" is that of the whole isolinux.bin file minus the first 64 bytes.
This is how I calculated it (and verified it against the isolinux.bin 2.12-3.86) and veriified it agains "mkisofs massaged" isolinux.bin's and tested successfully in the 4.07 tests I made with the batch.
In a nutshell, there are two checksums:
1) is called in isolinux.asm "FirstSecSum" (and is the checksum of bytes 64-2048, and more or less is/should be a kind of verification of proper loading of the first sector, which is 2048 bytes and corresponds to the 4*512 bytes sectors that -boot-load-size 4 refers)
2) is called in isolinux.asm "bi_csum" (probably stands for binary image checksum) and is initially set, in isolinux.asm to 0xDEADBEEF (like the length of the file itself and the 10 following dwords while the LBA address of the binary image (in the .iso) is set to 0x00000000),
; This table hopefully gets filled in by mkisofs using the
; -boot-info-table option. If not, the values in this
; table are default values that we can use to get us what
; we need, at least under a certain set of assumptions.
bi_pvd: dd 16 ; LBA of primary volume descriptor
bi_file: dd 0 ; LBA of boot file
bi_length: dd 0xdeadbeef ; Length of boot file
bi_csum: dd 0xdeadbeef ; Checksum of boot file
bi_reserved: times 10 dd 0xdeadbeef ; Reserved
Now, after compilation of isolinux.asm to isolinux.bin, all these fields remain as above.
This would be OK, as long as mkisofs (or however a program that fixes the bootinfotable) is used with the -boot-info-table switch that:
1) writes in bi_file the LBA address (in the .iso)
2) writes in bi_length the actual length of the boot file
3) writes in bi_csum the actual checksum from offset 64 to the end of the boot file
4) for *whatever* reasons writes all 00's to bi_reserved
The news in Isolinux 2.13 is that the good Peter Anvin introduced in the code a smart way to attempt to boot even if the user forgot the -boot-info-table switch.
So *something* at build time post processes the just assembled isolinux.bin as follows:
1) leaves in bi_file the address 0x00000000 (which value is later used to check whether -boot-info-table has been used or not, as that value won't ever be 0 if -boot-info-table is used)
2) writes in bi_length the actual length of the boot file (after having padded it with 00's to the next multiple of the CD sector, i.e. 2048 bytes)
3) writes in bi_csum the actual checksum from offset 64 to the end of the boot file just padded
4) for *whatever* reasons leaves the bi_reserved to 0XDEADBEEF
In checksumiso.pl the process is very clear.
Since version 4.00 the checksumiso.pl is no more and as said seemingly the same chores are done by the prepcore.c.
But it is not like re-posting it you make me understand the (stupid) C syntax, I can stare at those lines all day long, but I will never be able to actually understand what (the heck) they do, I take for good that you can understand that code and that it actually (wrongly) computes only the range 64-2048, though I am not convinced at all that the checksum there is calculated for bytes 64-2048, simply because if I recalculate it it doesn't match (while if I recalculate the right range checksum it matches, meaning that my half-@§§ed and blatantly copied/pasted/adapted algorithm in Tiny Hexer works just fine).
In any case it is to be confirmed that it is actually that prepcore.c file the culprit and that the code in it is wrong (there could be another zillion things in the build procedure that changes those fields).
may be I should split this thread to a isolinux one?
Yep, why not?