5 replies to this topic

[Wonko the Sane]

RE:

 

 

3. LZMA2 Compression
WinBuilder 082 compresses attached file with deflate (used in .zip, .gz) algorithm.
For better compression rate, PEBakery now supports LZMA2 (used in .7z, .xz) algorithm.
Ex) Encode command was updated, see manual

 

As often happens, some critique.

 

The "embedding" inside a .script of a files was (originally) intended in Winbuilder as a quick way to "attach" something (very small in size) within the .script to avoid cluttering the "source tree".

 

Historically a large number of developers (wrongly) used it to embed in .scripts every kind of (large) program, often embedding in .scripts something non-redistributable or of very dubious source, this new feature seems like another incentive to put inside the .script every kind of stuff, making stupidly large .scripts, including MB's of encoded data.

 

If the feature was used "as intended" here should be no particular issue with old zlib2 compression because the amount of compressed data should be very little.

 

Anyway it makes (to me) little sense to offer an option that breaks so dramatically the compatibility with Winbuilder and - possibly - using yet another format.

 

For Winbuilder .script embedded files we managed (after as usual quite a lot of bickering) to obtain a self-standing de-encoder/decompressor (some references here):

http://reboot.pro/to...elease-unwbzip/

 

Maybe it is the case (if needed) of releasing a corresponding similar tool for this new format (possibly with a simpler syntax, avoiding the *need* of a half-@§§ed batch to actually use it)

 

Wonko

 

 

 

 

[ied206]

Thanks for your opinion, wonko.

 

Historically a large number of developers (wrongly) used it to embed in .scripts every kind of (large) program, often embedding in .scripts something non-redistributable or of very dubious source, this new feature seems like another incentive to put inside the .script every kind of stuff, making stupidly large .scripts, including MB's of encoded data.

 

If the feature was used "as intended" here should be no particular issue with old zlib2 compression because the amount of compressed data should be very little.

 

Anyway it makes (to me) little sense to offer an option that breaks so dramatically the compatibility with Winbuilder and - possibly - using yet another format.

I acknowledge about adding another archive format would break compatibility with WB082.

But I think this feature will help saving internet bandwidth (common problem of webmasters), and it is worth it.

 

In theory, only small files should be included in script, just like you said. But I don't think every developer can keep this rule. System scripts can follow this rule, since the project trees can include required file before being shipped to users. But for application scripts (distributed seperate from project), I found lots of user can benefit from embedding files, since it only one drag-and-drop will just work. In this case LZMA2 is useful.

 

And if a project intends to use PEBakery exclusively, then can benefit from LZMA2 no matter what.

 

For Winbuilder .script embedded files we managed (after as usual quite a lot of bickering) to obtain a self-standing de-encoder/decompressor (some references here):

http://reboot.pro/to...elease-unwbzip/

 

Maybe it is the case (if needed) of releasing a corresponding similar tool for this new format (possibly with a simpler syntax, avoiding the *need* of a half-@§§ed batch to actually use it)

In fact, I am planning to write these kind of tool, too.

 

 

My ultimate goal is to design new file format for scripts, to solve inefficieny caused by current file embed design.

My current approach is to make script file 'immutable compressed archive' and save setting elsewhere, but better design may exist. It would break the compatibility for sure, but thinking of planned new script language, it cannot be avoided. What do you think about this approach?

[Wonko the Sane]

Well, in the "old" scheme (and with all the limitations and - with all due respect to the original developer - foolish choices taken at the time) encoding (large) data into the script had some reasons, but nowadays, given that you are going before or later to break compatibility, it makes even less sense (to me at least).

 

The bandwith excuse is not a good one, a whole project can be compressed in a .7z archive for download with a similar or better efficiency, but if you are going to break backwards compatibility, you can well get the d@mn original data, and simply 7-z it together with the .script.

 

Your new program can well be made capable of reading the .script inside the .7z and access the rest of the .7z archive, this way you have not the 64ENCODE (and decode) steps.

 

There is no need to design a new compressed format a "plain" .7z would do just fine as I see it.

 

Wonko 

[ied206]

There is no need to design a new compressed format a "plain" .7z would do just fine as I see it.

 

Wonko 

Of course, I won't design a new 'compress format'. I am thinking something like OOXML or deb, rpm files.

And to be strict .7z is not suited for that task, because of solid compression.

[Wonko the Sane]

Of course, I won't design a new 'compress format'. I am thinking something like OOXML or deb, rpm files.

And to be strict .7z is not suited for that task, because of solid compression.

It seems to me an unnneeded complication.

 

But I also cannot see what solid compression has to do with this.

 

I may well be wrong, but the scope should be making things as simple as possible (though not simpler than needed), a typical Winbuilder (or PEbakery) .script basically does a small number of simple things:

1) issue some commands, some of which dedicated to preparing the filesystem, creating the directory structures, etc.

2) write (mainly copying them from compressed source) some files

3) edit some settings (in the Registry and in other files)

4) rinse and repeat

 

Loosely - but not so far from reality - the same things an installer or *any* scripting environment would do, I would actually go a little further, stating that *any* program can be replaced by an installer/scripting language and a set of command line tools.

 

So now (Winbuilder) we have a .script format that is plain text (IF the stupid use of commas and quotes can be categorized as "plain" ) that can contain, BASE64 encoded/zip compressed binary data.

Sometimes the amount of these data, inefficiently encoded/compressed, is extremely large.

 

The interpreter for this command can read these "plain" text files and extract from them the encoded/compressed data.

 

So, in a nutshell, the .script is - besides a list of a given sequence of commands for the interpreter - a container for the data.

 

I don't see any reason why we cannot use a "proper" container (a known, efficient archive format) to contain both the "plain text" script and the data (efficiently compressed).

 

The interpreter needs three features:

1) ability to uncompress the .script from the archive (since the .script would NOT contain anything but the actual sequence of commands its size will be small)

1) ability to interpret the .script (same feature already existing)

2) ability to extract from the archive the data (same feature already existing, that can can currently decode and extract the embedded data)

 

Wonko

[Wonko the Sane]

As a side-side note and before I forget:

 

http://consolesoft.com/p/bhx/

 

 

 

BHX v5.6 - Encode a binary file in a batch script for rebuild it. BHX v5.6
Encode a binary file in a batch script for rebuild it.

By default, the generated batch script name is mybin.cmd.
Is recommended convert your file in a cabinet
with compression, created with this command line:
Makecab /d compressiontype=lzx file file.cab
and use the generated cabinet as source.
This can save many bytes in the output script.

BHX source [/ne] [/o:out] [/y]

source Binary to encode.
/ne If the source file is a cabinet, not write code
for expand it.
/o:out out is the output filename instead of mybin.cmd.
/y Overwrite the output file.
/hex Encode binary data using 16 hexadecimal characters.
The default encoder uses 85 printable characters.

BHX /author
display the author of the program.

 

Wonko