Quality in OpenEmbedded

Determining the Quality of OpenEmbedded is quite hard as we depend on 4000 pieces of software. So OpenEmbedded quality is aggregated from the quality of all such independant projects. To make things worse we use these things in many different combinations. By guess we have +10 target distributions +30 target machines, +4000 package/build descriptions and we should work on all available Linux distributions and somehow even non GNU UNIX platforms. I don't want to do the math to think about each possible combination but trust me this is more that we currently have computing power to compile. Then people would need to run these apps and their tests on target platforms and some how all these results must me managable.

The idea on how to test it and manage quality. The main strategy is to track difference and notice early when things break, notice them before a user would have a chanche to notice. The most common issues are missing source, failed configure, failed builds or wrongly packaged.

• Test if this application/image in this configuration with that toolchain properly compiles and links. We can track binary size differences, the build times and monitor differences.
  



• Test the packaging. Test if some files are in a package where they shouldn't be. This applies to .so files or .debug directories. Also test .la and .pc files if they point to the right directories. Check if the architecture of the file is compatible with the TARGET.
  



• Test -native packages after we have compiled them and before we stage them. Run the tests suits of these applications and refuse to work when they are broken. This is tricky as there are sometimes known issues, e.g. with gcc where a lot of tests will fail but are known to fail. So we can only fail if we really have an unexpected failure.
  



• The real nice thing to test is if this application actually works. It may not work because this is a known software issue or we have done something wrong or the application makes bad assumptions on the target architecture. But we need test cases for these things. Many of the GNU utilities have something to test them, these testcases must be able to be packaged and execute on a phone using the installed software. This is actually tricky as well. If we manage to get the testcase to execute on the target platform we still need to get the results back. With DogTails
  



• Track test coverage of the software we build. Always report how many lines are untested.
  

The architecture to solve some of these issues consists out of patches to our metadata and bbclasses. E.g. the "Cross Compile Badness" check which looks at some known wrong include directories when cross compiling and aborts. This way applications will fail when they have -I/usr/include in their INCLUDEPATH. Packagers will be made aware of this issue when they create the package and add it to OpenEmbedded. Also with the insane BitBake class we will be able to see the following issues: insecure RPATH in the binaries, wrongly packaged files, packages where ARCH and content are different, wrong depends. Together with BitTest we will be able to see undocumented keys, missing sources, wrong RDEPENDS for -native packages. I think we slowly get to the point where we have the tools to secure the quality of our OpenEmbedded product.

On the server side we are currently lagging behind. We have the crappy tinderbox3 from the Mozilla Foundation but I have started writing a new one using python and Django. This setup will help us to relate Bugtracker, IRC, Build- and Testresults. You will be able to do queries like. Show me everything about the package gimp-foo. You will be able to see test results, compilations, you will be able to filter for weeks/month/architecture.

But why is this important? A complete setup will allow us to spot regressions fast, inform the developers immediately, e.g. through RSS feeds or IRC. It will allow to see which software works in which configurations so it will be possible to declare known good configurations. E.g. compile servers could make their images available which get regularily booted and ran on QEMU devices for ARM, MIPS, PowerPC and other real devices which send back reports to the system, e.g. including coverage data and memory footprints. So we will always know that the applications compiled and ran. We only need the discipline to add tests after having found issues and encourage the projects we compile to create test suites themselves.

Oh and finally it is fun to create and setup such a system. The current status is okay as well. We have a Bonsai system which relates to Bugtrackers. Going to here you will be able to do simple queries. If a string like fix #123 is found you will be redirected to the bug tracker. This bonsai will be used to relate build results to changes in the SCM.

Imagine we track all the SCMs we build against. Imagine you build the gimp and the memory footprint during the tests increased by two megabytes and you want to track it down. You will be able to see all the changes made to the dependencies of the gimp between this build and the last known one. You should be able to find the source of this issue quickly.
For the tinderbox itself I have written a first model which I will try to fill with some data to find out how it performs. Actually I'm really excited about how well these things can be integrated.

SoftwareEngineering and QA for Free Software

There are two aspects of Software Engineering that currently interest me most. One is taking Usability into account when creating applications. What tasks needs to be solved, who is our target user, how often are these tasks executed. All these questions will heavily influence the User Interface and behaviour of your application. I started as a geek adding geek features to applications. So dear lazyweb and specially Free Software developer. If you write a new application ask yourself what these primary usecases are, talk with usability experts as they will help you to write better software. If you plan adding this very important feature to an existing application, please ask yourself how this fits with the primary usecase of the application and the target user.
No the addressbook for your grandmother doesn't need easy access to the GPS coordinates of her weapons of mass destruction. Such patches and features wishes should always be ignored. If you are in doubt if such a feature is appropriate consult an expert. We are experts on technology and sometimes by accident we create an usable userinterface and this brings me to another subject, one where I have more domain knowledge.

Quality and Quality Control in Free Software projects. The important question is always what is quality? What is good quality? Technically I always try to achieve the perfect quality but from a QA point of view this term is undefined. The Software we create should not crash, it should not have security issues and you should not lose data. I think this is mandantory quality that everyone agrees on. But what about questions such as usability? Can this application be used to do task X and how easy it is to get task X done. Do people know that task X can be solved using this way or do they do it in a more complicated way?


This leads us to my main question? How to check and assure the quality of your software? Regression tests, memory usage, coverage tests are well known types of tests. They are a good a tracking the quality of your software but these are only on a technical level. They answer questions like does this application crash, leak memory, or loses data again? But they don't answer the really important question. Can this application be used by our target and will he be happy? I have no idea how we could test this though.

h.

Not responding to Lorn...

Hey Lorn,
I'm not going to abuse blogs as mailinglist or forum but I have some random thoughts. So Lorn Microsoft is a Free Software supporter because they hired the Gentoo Founder? I always thought that Microsoft doesn't like Free Software too much, after applying your logic I recognize my failure. I like old cars as well, I used to drive an old VW beatle during summer and winter. Yes it was old, easy to repair and I think I understood most parts of the car. Oh and it has had serve security issues as well. But in contrast to cars, software isn't likely to stay the same for more than fourty years and as our products are virtual I prefer to fix security issues. I prefer to fix issues that annoy users, I prefer to fix issues where a user can lose personal data. So in contrast to my old VW Volkswagen Qtopia 2.2 is not usable for many reasons. I get spare parts for the beatle, I do not get support for Qtopia 2.2, fabricating issues of the beatle are well documented, they are not documented at all for Qtopia, there is a local community of beatle drivers helpimg me out in need... I think if I would have a commercial Qtopia2.2 license and file bug reports your support will likely to tell me that these things are fixed in Qtopia4 and ask me to upgrade. So even your support will ask me to not use the unmaintained Qtopia2.2.
And there is no word in my previous post that blames Qtopia for something. Opie's error was the lack of differentiation to Qtopia. People thought both were Free Software and failed to see the difference. The difference was that Opie secured the Freedom of the Platform and created this community were issues were documented and fixed. This included public mailinglists, bugtracker and source control management.

And to the compability. I seriously wonder if you are a core developer, and you found issues why didn't you file bug reports? Or as you are a core developer why didn't you write a fix and shared it with us? This was what Max and Me did when making Opie-Console work for the Sharp.ROM. Do you know about weak symbols? These things make it possible to compile apps against Opie's libraries and let them work on Sharp.ROM and Qtopia1.7. So really if you don't have a clue about how our and Qtopia's library work internally don't make false statements.


Oh and yes, shared libraries are fluff. Instead of sharing an implementation between 15+ apps it would be better to write 15 different implementations with different bugs and different issues. And if you don't see the value of a sane and flexible debugging framework talk to the creators of Qt and ask them why they did something similiar in Qt4. Have you ever debugged on a device? And yes getting debug messages across a network is a feature I have regulary used when debugging on devices. This can be used when running test cases, it can be used when you don't have much space on rootfs and don't want to use NFS for keeping logs. But such features are only obvious if you actually test stuff on devices.


love and peace
zecke

PS: luckily I didn't respond

Joys of Freedom, R.I.P Opie

Another period of not blogging. A lot of things have happened. I attented the KDE conference in Dublin, was at OEDEM, went to the awesome Qt DevDays, visited a couple of other facilities and the term just started.
I'm mentally back to KDE development and my personal goals for KDE4 are better QA, smaller memory footprint, faster apps, less bloat, more usable enduser applications and sane error reporting. I still have nightmares understanding why GPG, KMail and Kubuntu won't work together. "Error initializing the backend" is IMHO not a usable error message and these things need to be changed. I have started writing my first web application using the Django Framework it is called Bonsai and allows you to query SCMs. E.g. the primary usecase is to find deleted files as most modern systems don't have something like an Attic we are all used to from CVS.

Back to the Freedom and Qtopia Greenphone. It is an incredible device and if you buy one from Trolltech as Free Software developer you even get the sourcecode of the applications. And who needs the source for the kernel, libc, Qtopia libraries and the toolchain itself anyway. I think sourcecode is overrated anyway. Well I'm just kidding.

I took a quick look at the Qtopia Release Notes and I just recognized that only nine more days are missing and Qtopia hasn't had a GPL release for another year. From 17 documented releases 7 have a GPL release. And the last six releases are without any GPL release at all. From my point of view this makes Qtopia a proprietary platform like Symbian or Windows Mobile. I seriously wish Trolltech commercial success with their platform but from a Free Software point of view the Qtopia platform is just as interesting as Windows Mobile. And Windows Mobile has sadly one advantage, you can actually buy devices running Windows Mobile in Europe. And with handhelds.org we have a project specialised in liberating these devices which secures our freedom. So lazyweb do me a favor and stop believing Qtopia is Free Software.

I find it amusing guyhlem asked Opie to die and do you know what? Opie is dead, unmaintained, stuck with Qt2. So as Opie is dead all people use Qtopia now, right? Sadly this is wrong. The short answer is people that care about freedom, or a maintained platform just use GPE an Gtk+/X11 based environment. The longer answer is when Opie was actively developed the developers used the applications, integrated Opie to new devices and as their were users they knew about the issues and shortcomings. So a lot of real life issues and shortcomings were found and the important ones were hopefully addressed. This is the classic Quality Assurance of Free Software Projects. Code gets released, people test it and find all kind of issues which you are trying to fix. The result is a better integrated, more stable system. I have tried a lot of times to merge Qtopia improvements back into Opie and either I'm completely stupid or most of the stuff that was advertised in the Release Notes didn't work properly. My favorite example is the Config Cache in the Qtopia Config class. Config is a small Map which maps keys to values using QMap. The keys belong to a specific group. For the current group they have an iterator on this map. Now on write they destroy the QMap and invalidate the iterator. So if you now read or write through this iterator your application will just segfault. So for Opie I have merged and later patched their class to not crash, send patches to Trolltech and do you know what? Config::write in Qtopia will still leave a QMap::Iterator around which is invalid. The other interesting issue was with the fifteen game. They enchanched the graphics of this game and I decdied to merge it. Merging was again straight forward but the application didn't look right on 640x480. Hey one of the engineers mixed up width with height and we all know these things can happen. So I have written a patch against Qtopia sent it to the opie-forum@trolltech.com and the feedback again was none. I don't know if they have fixed this issue, fixed it independly or whatever. So the issue with Qtopia until now was the engineers at Trolltech probably don't have the time to listen to us, to the needs of the community or even accept patches.
Imagine we would have used Qtopia2.1.0 when it was released and ditched Opie. Imagine that we had the resources to make it work well enough on the available platforms (mainly iPAQs, Zaurus). We would have to maintain a patch set for all the bugs we find, no patches from us would be applied. For futher releases we just have to hope that they will do the right thing(tm) and without talking to us still doing what we need. So Qtopia2.1.1 gets releases and we don't know what bugs were fixed as no ChangeLog will be made available. So we would have to merge all our patches to Qtopia2.1.1 again and maybe even have new patches. And trust me merging 100+ patches to new releases is no joy. And you will do the same thing for Qtopia2.2, Qtopia2.3, Qtopia3.0, Qtopia4.0.... And you sincerly hope that they will do the right thing and fix the bugs you have fixed but reaility is different.We would end up with an ever growing patch set as we find new issues, start polishing applications and do you know what? Trolltech will do the same. So at your next merge point you can either throw your improvements away or try to merge them again. So at one point you might recognize that they won't do the right thing(tm) when they don't communicate and there is no way forcing them to.

So what would you do then? I have learned my lessons during Opie development so I'm thankful for this experience and I'm better educated than ever. Probably the most import lesson is "Don't play with a team when the team doesn't want to play with you". So what does that mean? We have always advertised Opie as Qtopia compatible. And as Opie was Free Software, many people thought and still think Qtopia is Free Software as well and we have taken the public preassure away from Trolltech. So we have advertized their platform and created public awarenes even if they didn't want to play with us. We went to fairs and demonstrated the Free Qtopia, talked with many people from different companies and projects and some of those evolved into paying gigs for Trolltech. And Trolltech went as far as advertising thousands of available Free Software for their platform.

R.I.P Opie, it was a pleasure to hack on you. But there is still hope. There is still the hope that Trolltech will release Qtopia 4.2 GPL and if they somehow promises to regulary release Qtopia sources we might see a Community around Qtopia again.

And Qt4.2 is full of incredible stuff. New feature likes Cascading Style Sheets for Widgets/Applications and the QGraphicsView are just awesome. E.g. the painting engine in Qt can probably be compared to cairo but there is a huge difference. Qt has a fixed point implementation for this engine since day one. So Qtopia4.2 might just be right for your device so let us hope Trolltech will recognize the value of the community (again).