3 replies to this topic

[sbaeder]

After making some comments on WB development and WB80, I sugested we do a more formal test plan. As much as possible could/should be automated. And where not automated (impractical/impossible), we have a checklist of sorts on the things we want to see...

So, After being referred to the poasting Nuno created on the overall plan, I thought it would be a good idea to have a place to discuss testing.

SO, let the comments begin...

[amalux]

Hi sbaeder

Much calmer here (for now ), I'm taking this opportunity to offer some thoughts on WB testing and my overall feeling about its development. People can take or leave what I say as I am really an outsider looking in and that's the way I like it; it helps me to stay objective and calm, able to see the big picture a bit clearer. I have nothing but respect for the developers (they know who they are) even though I may find the development going in directions I don't always understand (at the time). Regardless, I see an overall improvement in WB and I am grateful for those improvements and for the continued free use of this great tool.

I see WB somewhat as a 'diamond in the rough', takes some effort and hard work to make it shine but shine it does, brighter than all the rest and I'm a bit afraid of polishing it too much. If WB gets too easy to use, too 'packaged', too controlled, it may restrict the 'development' at the user end. This end product 'evolution' is probably, largely overlooked by the core developers but is really a large part of the pay off for the rest of us. I think there has to be both and they need to be somewhat separate; the more one focuses on 'core' development, new features, bug fixes etc. the harder it is to see the big picture; the end products (boot disks) usability, customization and addition, improvement of tools.

I'm not sure what I'm trying to say other than, let's lighten up a bit, stand back and see that the big picture is really pretty damn good! Sure, there's more efficient ways of doing things and of course we all want WB to keep getting better (it is) but let's not throw out the baby..., things are good and getting better every day; I don't know if a complete overhaul is necessary or even productive. There needs to be those that focus on bug fixes and core development but not at the expense of stability and usability for the rest of us. I find testing every new release of WB like re-inventing the wheel over and over again, I think the wheel works good enough to focus on building a better cart and eventually a car around it and that's what I focus my time on; the car is looking pretty nice from where I sit.

[sbaeder]

Thanks for the feedback, and I agree 100%...the engine seems to be pretty complete...Maybe a few tweaks here and there, but for the most part it's in really good shape - just look at the quality of the projects it has allowed to shine.

On the other hand, it wouldn't hurt to be a bit more formal in the testing process. For example, if (as stated in the other post on this (Post 101 on WB80) that LiveXP is a good "stress" test of the syntax and processing, then that should be the "benchmark" for release.

I have also seen some discussion of making sure we have enough "strange" (at least to me as an English version system and spaeker) characters in the paths and scripts, etc. Even things like a longer path (which causes the delete to fail on some of the cached files) are all good things to put in place as a check list before release of even an "alpha" build.

But I agree, we also need to find ways to better organize and support the projects that layer on top of this "diamond"...

[Brito]

I propose that our test cases begin by covering the basic functionality steps: http://winbuilder.ne...tax.html#syntax

We begin writing a script that tests file operations (filecopy, filedelete, filerename, filecreateblank, etc).

For each wb operation that is under stress, we should perform a series of tests that will output the result as successful or not.

For example, testing filecopy:
- check if it is working in very long directory names
- test if it is working with space on the name of folders
- test how it will react to the cases where no source file is found
- test the cases where it has no permissions to write on the target folder.
- test what happens if the target folder has no more disk space
- test overwrite of files (and possible switches)
- .... (more to be added)

My idea is that each script should test a specific wb operation to allow us add more test cases in the future.

It would be even better if we could somehow sum the result into a score.

For example, wb 081 passed with 92% of compatibility to the available tests to help us understand and pinpoint what is changing between versions.
(source: http://www.boot-land...?...st&p=94395)