67 replies to this topic

[homes32]

As Nuno suggested in this topic this topic is for the discussion of ways to enable external programs to integrate with winbuilder.exe

• How to pass existing script, global, and permanent variables to this new (let's call it) capi.exe?
• How to have capi.exe update project global variables (ie Set,%var%,NewValue,GLOBAL)?
• How to have capi.exe update project permanent variables (ie Set,%var%,NewValue,PERMANENT)?
• How to have capi.exe update the winbuilder log?

[MedEvil]

In theory we could always call the capi.exe with shell execute and hand over some parameters.
If that is a viable option to speed up WB or if this will slow it even more down, can only psc answer.

Better imo would be a CAPI.dll, which integrates with WB.

[pscEx]

And the best would be to revise the CAPI.

Is it e.g. "very senceful" to currently use the CAPI with reg_add > RegWrite,HKLM

API definitions should make app scripts platform independent.
And it increases WB processing time when API is only used to shorten inputs which are the same for ALL platforms.

Peter

[Brito]

How to pass existing script, global, and permanent variables to this new (let's call it) capi.exe?

One way would be passing them as parameters like MedEvil said or writing them inside another file when the external file is launched.

Also, do remember that most variables are already stored inside script.project and each script also holds his own set of variables as a INI section. If you're worried about values that might be modified while the project is running, you can also write them back to a INI file.

If you need to change a considerable number of static variables while the project is running then I'd say that this is more of a structural problem in the project than anything else.

How to have capi.exe update project global variables (ie Set,%var%,NewValue,GLOBAL)?

From the external program, write the intended variables to a INI section and when returning to winbuilder call "AddVariables" to add multiple variables on a single operation: http://winbuilder.ne...ml#AddVariables

How to have capi.exe update project permanent variables (ie Set,%var%,NewValue,PERMANENT)?

These variables are located on script.project and you can edit them there directly.

How to have capi.exe update the winbuilder log?

The only influence you can have on wb's log is using Echo. I'd suggest that the program writes it's own log apart from wb's log.

If an error appears, I'd suggest a window popup with a timed counter to press the "ok" button. Or since capi is project-wide intended, call it on the finalization scripts to warn users about errors that need attention.

----------------

From experience, writing all variables to a fresh file takes a few milliseconds and can be read by any external app using the standard INI read. You can also reverse the process to understand what went good or wrong with the external program that was called.


As a side note:

The default INI read/write code that comes with most compilers is notoriously slow compared to the custom read/write code found inside WinBuilder. This occurs mostly because wb ignores any duplicate sections and quits reading the rest of the file once the targeted section is found.

I can help you create faster INI read/write code to get better performance if needed.

[homes32]

The only influence you can have on wb's log is using Echo. I'd suggest that the program writes it's own log apart from wb's log.

If an error appears, I'd suggest a window popup with a timed counter to press the "ok" button. Or since capi is project-wide intended, call it on the finalization scripts to warn users about errors that need attention.

I can see that being extremely confusing for end users and frustrating for developers/people trying to help the poor confused user. it is much easier to debug something if all the logs are inline with one another instead of trying to mentally or manually put a time line of events together from 2 or more logs.

[Galapo]

It seems all very intricate, to say the least, and can be broken at any point along the chain.

Nuno's suggestions seem rather logical, but possiby may have to involve a rebuild of variables to be successful. Sometimes that can produce unexpected results.

capi.exe sourcecode should be opensource so we can all make changes/updates/suggestions as necessary.

Regards,
Galapo.

[Galapo]

A agree with homes32. One of the advantages of using WB is that we get an extended log. To then not have this available, or available only in some second log, is not an advance.

Regards,
Galapo.

[Brito]

it is much easier to debug something if all the logs are inline with one another instead of trying to mentally or manually put a time line of events together from 2 or more logs.

Yes, I fully understand what you mean. Tried my best to think on a better proposal.

For the moment, please bear with the current limitation and proceed with an external log if necessary even if not the most elegant or integrated solution.

The hard-core way would be acessing the memory pointer where winbuilder.exe stores the log lines and add the desired text. This would function across all wb's without changes but requires intrinsic knowledge of how to use memory pointers.

Not indicated for the faint of heart but blazzing fast logging..

------

Nevertheless, will keep an eye to solve this shortcoming in the future. Defining a flag on the echo command wouldn't also be a idea to implement along the way.

[paraglider]

Using an external exe to me sounds wildly inefficiant. Much better to use a dll which is all in process and will remain loaded for more than one call.

For example require_file if coded as an external dll everything needed could be passed as parameters.

Access to the log file as already pointed out would need to be addressed. Could be done by winbuilder providing all log access through a api dll which it uses and which other external dlls could also use.

[Galapo]

So it'd be interesting to see if an external app or dll run by 080 would be faster than 077rc2 using api by script.

Certainly it's good to look for improvements with CAPI. But this shouldn't be at the neglect of searching for why 080 is about 2x slower than 077rc2. An external CAPI app run by 077rc2 would currently be the fastest build method, but it would be advantageous to have an updated winbuilder.exe running at 077rc2 speed (or better) also processing api commands with something external. The pressure for such a solution, as I've said before, will be felt more by projects no longer supporting or able to support the speedy 077rc2 version.

Regards,
Galapo.

[pscEx]

But this shouldn't be at the neglect of searching for why 080 is about 2x slower than 077rc2.

My measurement with TempPE.Script:
080_3: 2.657 sec
077_RC2: 2.203 sec.
That is not 2x slower, that is 20%!

With high propability the If command is one reason.

There was a bug which only could be fixed by a lot of additional checks:

Set,%var%,Bla

If,1,EQUAL,1,StrFormat,FILENAME,FullFilename,%Var%

Rather than executing

StrFormat,FILENAME,FullFilename,%Var%

the bug executes the line

StrFormat,FILENAME,FullFilename,Bla

wich leads to an error message.

BTW: Maybe this sample runs under 77 RC2. It should only explain the principle. I do not remember the original bug situation.

The fix checks evey If command, whether the following action contains a variable name which must not be replaced by the variable value.

Check 100 * the line

If,A,EQUAL,B,Echo,YES

It will take double time in 080 / 081 compared with 77 RC2.

Peter

BTW: for those who are interested:

// search variable names not to translate

	if ScriptMemo.Count > 1 then

	  m1 := ScriptMemo[1]

	else

	  m1 := '';

	GetVarIndex(command, m1, varIndex, secondIndex);

	varName := '';

	secondName := '';

	if varIndex <> -1 then

	  varName &#58;= TransInner&#40;ScriptMemo&#91;varIndex&#93;&#41;;

	if secondIndex <> -1 then

	  secondName &#58;= TransInner&#40;ScriptMemo&#91;secondIndex&#93;&#41;;

// translate variables

	for i &#58;= 1 to x - 1 do

	begin

	  ScriptMemo&#91;i&#93; &#58;= TranslateParams&#40;ScriptMemo&#91;i&#93;&#41;;

	  if &#40;i <> varIndex&#41; and &#40;i <> secondIndex&#41; and not AnsiStartsText&#40;&#39;OUT&#58;&#39;, ScriptMemo&#91;i&#93;&#41; then

	  begin

		ScriptMemo&#91;i&#93; &#58;= TranslateVariables&#40;ScriptMemo&#91;i&#93;&#41;;

		if ForceWarning and doVarCheck and ExecBlock and &#40;AnsiPos&#40;&#39;%&#39;, ScriptMemo&#91;i&#93;&#41; > 0&#41; and isVariable&#40;ScriptMemo&#91;i&#93;&#41;

		  then

		  InstantMsg&#40;LangMan.LangLog&#40;&#39;!#2482&#39;&#41; + &#39; &#91;&#39; + ScriptMemo&#91;i&#93; + &#39;&#93;&#39;, msgIcon_Warning&#41;;

	  end;

	end;

[Galapo]

My measurement with TempPE.Script:
080_3: 2.657 sec
077_RC2: 2.203 sec.
That is not 2x slower, that is 20%!

Hi Peter,

I could not reproduce your good result. See my logs here: http://www.boot-land...?...st&p=102991.

077rc2 processes the script in 3.484 seconds, while 080 processes the script in 5.938 seconds -- that is close enough to 2x faster to call it 2x faster. Most app scripts will process close to 2x faster as well with 077rc2.

With high propability the If command is one reason.

I assume you're doing testing on functions -- you'll notice that the issue isn't restricted to the 'If' command but is basically across the board.

Regards,
Galapo.

[pscEx]

Hi Peter,

I could not reproduce your good result. See my logs here: http://www.boot-land...?...st&p=102991.

077rc2 processes the script in 3.484 seconds, while 080 processes the script in 5.938 seconds -- that is close enough to 2x faster to call it 2x faster. Most app scripts will process close to 2x faster as well with 077rc2.

Because "Single Tests" may depend on some host activity etc., I made a test series with TempPE.Script. Every value is the average of 5 tests.

I got:

In LiveXP with 077RC2: 2.125 sec
In LiveXP with 081RC1: 2.825 sec. That is a difference of 33%
In nativeEx_Barebone with 077RC2: 1.625 sec
In nativeEx_Barebone with 081RC1: 2.190 sec. That is a difference of 35%

The differences are technically identic.

But the "general" difference between LiveXP and nativeEx_Barebone is remarkable. It is also about 30%!

I assume you're doing testing on functions -- you'll notice that the issue isn't restricted to the 'If' command but is basically across the board.

You are right. The "Check variable function" mentioned above, is done in EVERY script line. Theoretically the bug "replace variable name by content" can occur in every statement.

Peter

[Brito]

Using an external exe to me sounds wildly inefficiant. Much better to use a dll which is all in process and will remain loaded for more than one call.

To me, using an external dll sounds wildly inefficient. It is much better to use a single winbuilder.exe which is all in a process and will remain loaded for more than one call. ( )

---------------------

The goal is improve the speed of scripts with these priorities in mind:

• Use the current winbuilder.exe
• Work with previous winbuilder.exe versions
• Avoid requiring any new function for a future version.

---------------------

A small console executable, even if called hundreds of times to perform a repetitive task (create a shortcut, or whatever task), is still far faster than creating the same results with script commands that involve thousands of IF's and string processing commands.


Also, moving some of the scripting to a executable is only part of the equation to improve the performance.

We still need to look better on the structure of projects and see what can be put on a "diet" so that we can keep scripts in a good running shape.

[MedEvil]

How about simply forgetting the CAPI idea?

It was known from the start, that the API and later CAPI, will have a huge impact on performance.

The only upside was, that it makes writing scripts much easier, which at the time, seemed like a good thing to bait some more members into becoming script developers.


Well, it didn't work this way, the number of script developers hasn't skyrocketed or at least increased.

By simply creating scripts with Pe1, PE2, Pe3 sections, we would also get cross project App scripts, but without any impact on performance.

[MedEvil]

To me, using an external dll sounds wildly inefficient. It is much better to use a single winbuilder.exe which is all in a process and will remain loaded for more than one call. ( )

Since when is a dll less efficient, than having the functionality in a monolithic exe?

[Brito]

The only upside was, that it makes writing scripts much easier, which at the time, seemed like a good thing to bait some more members into becoming script developers.

Actually, the main goal was writing scripts that don't need to be rewritten in the future.

Otherwise we'd have to write all scripts for PE4, PE5, PEnn.. Weren't you around at that time in 2006~2007 when we had to recode scripts for LiveXP, VistaPE, NativePE and so forth?

Since when is a dll less efficient, than having the functionality in a monolithic exe?

For a start, we avoid getting into a "DLL hell" with a single exe..

-------------

API scripts are a very good concept when they are kept simple as needed.

[MedEvil]

Actually, the main goal was writing scripts that don't need to be rewritten in the future.

Otherwise we'd have to write all scripts for PE4, PE5, PEnn..

Yeah, right. And nobody around here complains about app scripts with outdated versions!

For a start, we avoid getting into a "DLL hell" with a single exe..

Dll hell - script hell, what's the difference?

[pscEx]

API scripts are a very good concept when they are kept simple as needed.

I have to comment a lot.
But tomorrow, not now.

Currently I'm starting to view the FIFA championchip Argentinia - Greece.

I wish "Good luck" to my friends Smiley and Billonious.
(And the German trainer Otto)

Peter

[pscEx]

I wish "Good luck" to my friends Smiley and Billonious.
(And the German trainer Otto)

OOPS!

Peter

[Brito]

Yeah, right. And nobody around here complains about app scripts with outdated versions!

That's one of the advantages of free volunteer work. Personally, I only upgrade to newer versions whenever absolutely necessary if the project already has all app scripts I need and I just want to try another project.

But this is not the point of discussion nor it would be solved here. I think it's more important to look at the scripts that are taking 6 minutes just like paraglider mention and see what can be done to improve them.

Shall we start?

[Galapo]

Hi Peter,

I made a test with LiveXP recommended as off the server. Only changes was: disabled mass storage drivers script, disabled emulation and CD burning scripts.

Results:

077rc2 completed in 17 min 11 sec.
080 completed in 25 min 39 sec.

 Test_logs_LiveXP_rec.rar   553.99KB   467 downloads

080 in this test takes 55% longer to process the project.

But this is not the point of discussion nor it would be solved here. I think it's more important to look at the scripts that are taking 6 minutes just like paraglider mention and see what can be done to improve them.

Nuno, based on the tests above, it should be clearly seen that the biggest hurdle to fast builds is winbuilder.exe itself. The current stable is running around 50% slower than previously. By using 077rc2, a project can cut build time dramatically even without a single script optimised (if possible).

The priority as far as I can tell should therefore be with winbuilder.exe itself rather than work-arounds to the real problem which is a slowdown with winbuilder.exe script processing.

Regards,
Galapo.

[paraglider]

Whilst 7rc2 is faster I still see it as slow - 17 minutes 11 seconds is far too long.

With pebuilder I can build a 746mb xp based pe iso in about 3.5 minutes.

[MedEvil]

080 in this test takes 55% longer to process the project.

Only a bit over 49%.

@paraglider
Though i love the speed of pebuilder too, the speed comes clearly from the very simple scripts that are used.

[Galapo]

Only a bit over 49%.

Hi MedEvil,

Maths isn't my strong suit, but I assume it isn't yours either.

17 min 11 sec = 1031 secs
26 min 39 sec = 1599 sec

Half of 1031 = 515.5.
1031 + 515.5 = 1546.5
1546.5 is less than 1599, therefore more than either 49% and 50%.

55% of 1031 = 567
1031 + 567 = 1598
1598 is close enough to 080 time of 1599, therefore time increase of 55%.

Regards,
Galapo.