Some Blog

QtMediaHub on MIPS with Qt5 and DirectFb

2012-01-10T14:00:00.003+01:00

Ever since the start of the Qt project I am working on DirectFB in Qt5 (and Qt4) with the remote goal of getting QtMediaHub to run. It started with catching up with the rather nice refactoring of lighthouse in Qt5, fixing memory and resource leaks in Qt5, DirectFB and mostly in the DirectFB lighthouse plugin.

It moved to dealing with broken "make install", broken QML2 examples and documentation, figuring out how to get patches for QtV8/V8 into the project, adding MIPS code for the Qml mode (a new global object), global compare for MIPS and finally working on OpenGL integration for QML2.

In this specific case I had an OpenGL ES 2.0 library coming from a vendor and created the 'directfbegl' plugin to use EGL to go from IDirectFBSurface to an eglSurface. I think in this specific case there is unified 2D and 3D buffer space which should allow a lot of cool stuff.

It mostly works, QML2 has some way to go to work on battery powered devices but it is looking quite nice.

OpenOCD and SIMtrace

2011-11-09T16:26:00.004+01:00

This is just a short note to myself (and then copied into our SIMtrace wiki).

This is working for OpenOCD (74558296d1e185fc4a7522f08832eceed460cbd7) and the Amontec JTAGKey (but should work with any JTAG adapter). The easiest way is to start OpenOCD with default config files.

The SIMtrace v1.0 hardware is powered by the Atmel SAM7S microcontroller and the closest match for this platform is the atmel_at91sam7s-ek.cfg board configuration file.

First try

$ sudo openocd -f interface/jtagkey.cfg -f board/atmel_at91sam7s-ek.cfg
...
Error: An adapter speed is not selected in the init script. Insert a call to adapter_khz or jtag_rclk to proceed...

Second try

You can patch one of the two files, create a new config file and load it with -f
or specify the command. The below example specifies it on the command line.

$ sudo openocd -f interface/jtagkey.cfg -f board/atmel_at91sam7s-ek.cfg -c "adapter_khz 30"

Final config

$ echo "adapter_khz 30
> arm7_9 dcc_downloads enable
> init
> reset init
> " > myinit.cfg

$ sudo openocd -f interface/jtagkey.cfg -f board/atmel_at91sam7s-ek.cfg -f myinit.cfg

One of the calls appear to fix 'halt' not really halting the SoC, I have not investigated which of the options is doing it.

Flashing First try

$ telnet 127.0.0.1 4444
> flash write_image /tmp/main_simtrace.samba 0 bin
wrote 0 bytes from file /tmp/main_simtrace.samba in 0.238727s (0.000 KiB/s)

With a quick look into the NOR Flash code, it appears to fail as it does not
find a flash region at the address it is looking for one and then reports to
you the success of 0 bytes (instead of a failure).

Flashing Working

$ telnet 127.0.0.1 4444
> flash write_bank 0 /tmp/main_simtrace.samba 0
wrote 37020 bytes from file /tmp/main_simtrace.samba to flash bank 0 at offset 0x00000000 in 39.175068s (0.923 KiB/s)
> reset init
> resume

DirectFB contribution to the Qt Project

2011-10-21T21:33:00.002+02:00

The Qt project was launched today, I got my 15 DirectFB patches merged, got some first experience with Gerrit, created an account for the Qt wiki, fixed some documentation, so all in all I think it is the great start of the Qt Project we have been waiting for! So thank you very much for all you involved with it!

Now to something completely else, somehow I like to see the parts that are not great yet. But most importantly it is a great opportunity for everyone to get involved with the Qt project and improve things. So here is my short wishlist for the Qt project.

Single account for the Bugtracker, Wiki, other services.

Read-Only access to gerrit.

Public CI based on Jenkins, right now build failures will still point to internal Nokia servers. I assume KDE can help a bit with the Jenkins setup.

Make it possible for non-mainstream QPA backends and platforms to be part of the CI System, if proven stable be considered core builders.

The wiki being part of the Qt project should be part of the Qt Project, the license should probably be made compatible with the license of the Qt documentation, to allow copying from one to the other.

Once again, thank you Nokia, thanks everyone involved!

First days with Ubuntu 11.10

2011-10-17T17:27:00.002+02:00

Crypto Password for /home is echoed back... no canonical mode set..

A single process can take down a multi-core system... poor scheduler performance?

New volume applet has no support for changing volume by keyboard... and no way to change the volume of the microphone

GTK+ file dialog has wrong margins... visual offsense... left and right area have different height

Creating a small GUI for the SIMtrace application

2011-10-13T20:22:00.003+02:00

Earlier this year I created a Hardware company with a friend to supply to our GSM community and beyond. One of our first products is the SIMtrace Hardware (CC Licensed, actually it should work with any Smartcard). Today I had to wait a bit and decided to convert the CLI application that talks to the firmware to a GUI application. I did this by running the CLI part in a QThread and using QMetaObject::invokeMethod to callback to my GUI.

I started off creating the Qt application with VIM and a browser to help me to remember some function names. After a while I had enough of this and used Qt Creator and enjoyed the auto completion, it had no problem reading our C header files of libosmocore and provided auto completion for these too. Once again I am amazed how nice Qt Creator is and how little code I needed to create the below.

From October 13, 2011

QPixmap and what I did not know

2011-09-27T13:28:00.002+02:00

In general QPixmap is the class I dislike the most in the Qt graphics stack. It is not because it did anything wrong, or has annoying bugs. It is merely that painting on a QPixmap can produce a huge slowdown. Maybe it is even the immediate painting model I dislike so much.

In theory the QPixmap should represent the graphics memory and in theory painting this to another piece of graphics memory should be very fast. But reality is different, e.g. with directFB/X it is possible that your graphics memory resides in the host memory because you have allocated too much. X (EXA, UXA...) has migration strategies, directFB will just allocate it in system memory and it will stay there. So your graphics memory class might not be always allocated in graphics memory.

The athur painting capabilities of Qt4 were very nice but Hardware/X/directFB might not be capable of handling all the advanced concepts (perspective transformation, drawing other pixmaps scaled) and the paintengines then need to fallback into the raster path, but the raster path needs to work on a QImage and this is where stuff get terrible. There will be a download of the memory (slow), there might be a change in color (ARGB -> pre multiplied ARGB?), then the fallback, and then uploading of the memory again. So even classes like QGraphicsBlurEffect go through QPixmap -> QImage -> QPixmap (or at least used to).

So the basic problem with QPixmap and the immediate painting model is, only once you are done with painting everything, you know if starting with a QPixmap would have made sense. If Qt5 wouldn't be all about OpenGL, I would have lobbied hard to make QPixmap an internal class and use it to 'cache' QImage that have peen painted a lot.

After all this love/hate with QPixmap there were still things I didn't know. For Qt5 I am working on the directFB lighthouse plugin and started to work on make more tests pass. The thing I did not know was that by default the QPixmap has no alpha channel, only if either a mask is set or the pixmap is filled with a color that has an alpha channel the QPixmap will end up having an alpha channel and be ARGB. This means that a lighouse plugin providing a QPlatformPixmap implementation needs to be able to provide an ARGB and a RGB based pixmap and switch between them.

GNU Smalltalk deployment with images and image resumption

2011-09-24T14:37:00.000+02:00

Once up on a time I was sitting in a cold hall at the Barcelona exhibition ground, a power outage has taken down several DVB-H platforms (racks consisting of servers, streamers, RF equipment...) and once power was restored red LEDs were blinking, systems not coming up automatically, hordes of engineers trying to boot the right kernel, trying to remember the multicast routes they had typed in by hand, chaos, hectic. It was interesting to witness that as we could lay back, continue with our work, as our platform was configured to come up automatically and it did.

One has to assume that stuff breaks, software, disk, RAM, CPU, power supply, anything. The only answer to that is be prepared, build your software to deal with failure (or nicely try to restart), make sure all systems start automatically, have backups, have a plan.

For my GSM (telephony) in Smalltalk components I didn't do that yet, mostly because my code was not very mature, I was still building up the trust relationship and so my deployment consisted of manually running GNU screen, then GNU Smalltalk, manually loading my code, starting things up. It was about time to correct that.

The tool of choice is the gst-remote application, it will run a GNU Smalltalk image, it can daemonize and it will listen for input from other systems. While moving to this deployment my fellow Smalltalkers were kind enough to fix some bugs on the way. There was an issue of gst-remote exiting when saving an image, the Delay class not functioning properly on resume, the wrong FileDescriptors being closed on image resumption. It is very nice to get help that fast!

My deployment now consists of building GNU Smalltalk packages, having a Smalltalk script to load the package and call start method of that package followed by taking a snapshot and exiting. I can then resume the image and if the software is prepared to deal with image resumption it will work.


Eval [
  | name image |
  name := Smalltalk arguments first.
  image := Smalltalk arguments second.

  (PackageLoader packageAt: name)
 fileIn;
 start.
  ObjectMemory
        snapshot: image;
        quit.
]

Qt, QPA and some notes to myself

2011-09-18T17:21:00.002+02:00

QT_QPA_PLATFORM_PLUGIN_PATH, QT_QPA_PLATFORM and QT_QPA_FONTDIR variables are used by QApplication/QPA to find the plugin. They are also available as arguments but there is no help entry for them (Qt5 needs some polishing).

When building a plugin and Qt insists to tell you that the plugin is not there, you are likely to have some undefined symbols, the best hint is to build your plugins with -Wl,--no-undefined in the linker line. Obviously we should plan to use dlerror to also report why Qt is seeing the file but not listing it as available plugin.

cheers

shop for SIMtrace

2011-08-28T15:28:00.002+02:00

The WebShop for the SIMtrace hardware can be found here. We are using a CA-Cert signed SSL certificate and your browser vendor might not like it.

Setting up, or mostly modifying the webshop was my first encounter with rails. In some ways it is a great framework, in others with a Smalltalk background there are some tears in my eyes. This is using spreecommerce because I wanted something that is not done in PHP and is not following the OpenCore business model.

The manual of spreecommerce is actually great from an engineering point of view, they describe the concepts, the models and how to modify it. But they are bit short on setting it up. There are some config options where no Graphical way to change them exists yet. So it is best to look into the app_configuration.rb of spree_core, e.g. to modify the default country.

There are still some bits that I would like to change, the code for my modification can be found on gitorious.

Going down the Sangoma rabbit hole

2011-08-22T13:22:00.002+02:00

I ended up needing to configure FreeSWITCH to use a Sangoma ISDN card. At first we were foolish and used the setup script to install libraries into our system.

Cleaning up

It took dpkg -S and manual filtering to clean this mess. On a debian system there are files in /usr/ that do not belong to packages, these can be links to /etc/alternatives, or in case of python be bytecode files (*.pyc).

Second attempt with building packages

The next attempt was to use checkinstall -D make install but the source distribution of wanpipe is such a mess that checkinstall refuses to work, I have created a handmade debian package with some split up. The most annoying issues are the installation of .svn, .deps, .libs directories, using /etc/wanpipe for examples and firmware (they belong somewhere else), also installing the package requires to overwrite files of other packages (Linux comes with a wanrouter.ko too). There also appears to be some fishy licensing part, the source distribution is shipping with some .o files, the module claims to be GPL though.

First attempt with FreeSWITCH

FreeSWITCH allows to build debian packages out of the box, and it worked, it also built the freetdm module. But my module was missing support for wanpipe, adding the wanpipe-dev, rebuilding fixed the problem.

Second attempt with FreeSWITCH

After configuring the ISDN card using the wancfg_fs utility (and not allowing it to mess with the free switch config, as it is breaking it), I saw the cards but I was missing a library to actually use the card for Voice Calls.

ISDN libraries

I needed to get the libsng-isdn library, of course without sourcecode, also without a LICENSE file at all. Who may use this library? Under which circumstances? I started with checkinstall but this does not handle shlib providers very well, so I had to package the library myself

Third attempt with FreeSWITCH

I was able to recompile the FreeTDM module, and I could configure the ISDN trunk and use the the fs_cli to orginate a call.

Summary

FreeSWITCH and FreeTDM are quite powerful, also having support for creating debian packages is great too. The Sangoma experience is less exciting.

PayPal must die

2011-08-17T11:26:00.003+02:00

This is another rant, today about PayPal. It all started when I tried to buy some credit for the skype callout feature. The overlords at PayPal decided that besides the Post Ident I did to open my account, the verified bank account that is stored in my data I will now need to enter my German Identity Card number. Too bad that the number they have stored is from a card that expired maybe five years ago.

How comes they think the Number is secret after hotels, car rentals, dvd rentals, airlines, ..., started making photo copies of these documents? But to make it worse they have implemented this 'security check' in a way that prevents me of using the normal contact form.

So I took it up with the support, and they proposed me to reset my password, they kept on proposing to reset my password. They offered me to call me, I gave them my hotel + room number but they were not able to figure out the prefix for dialing to China. After three more support emails and proposals to reset my password I know have the instruction to close my account and transfer my balance to my reference account. All they want is my first born though.

I was a bit lazy and have not collected the information they need yet, today I wanted to buy something online, the company is using PayPal Express Checkout, and I am not allowed to pay with my VISA card as Paypal knows some combination of my name and forces me to log into my paypal account (which of course does not work).

Please, someone, fix this payment mess.

Finally trying the OpenSUSE build service

2011-07-17T09:46:00.004+02:00

For the upcoming SIMtrace hardware (e.g. see the communication between SIM and Phone) I want to provide a patched Wireshark and packages for our host utilities to make the installation (and upgrading) very easy.

Being a happy Ubuntu user I started with launchpad, trying to create my own wireshark package. It was very easy to get source of the wireshark package, add a patch, upload it to my ppa. Then I spent some hours figuring out that launchpad does not allow 3.0 (git) git packages (it simply rejects the upload without a proper error message). My biggest issue is to figure how with one upload I could end up with packages for multiple series. E.g. simply copying the package to another series does not work due version numbers (and the need to be able to easily upgrade). So how do people provide to multiple series?

The next thing I tried was the OpenSUSE Build Service. I know that Cornelius is working on it for many years but so far I have always failed using it via the Web Interface. This time I started with an installation of OpenSUSE and used the osc application. The documentation is very sparse (there are many commands and some inconsistencies in naming) and I decided to start with the easy task of modifying an existing package. I have picked wireshark again (it was a bit difficult to find the authorative copy of it), added my patch, uploaded, and I ended up with SuSE packages. The next task was to build packages for the other RPM based distributions and as expected just adding a repository for Fedora didn't work. But here comes the power of this system. First of all the interface shows me that I have a build failure or some missing dependencies and using osc build DISTRO ARCH I can start a local build and take a look at things. awesome! And the build farm of OpenSUSE is fast as well!

The biggest problem I have with OBS is finding documentation, or google not showing it to me. E.g. the cross distribution howto always fails to come up in searches but is very valuable when trying to make your package work on different distributions.

One thing that impressed me a lot was adding a new package. I used osc mkpac placed my sourcecode there and then I had to write my first ever RPM spec file from scratch. So I typed vim libosomocore.spec and somehow I had a skeleton of the spec file present. This is very nice!

OsmoST SIP (SIP in Smalltalk)

2011-07-06T20:52:00.003+02:00

Intro

The last couple of days, to some degree weeks I was implementing a SIP stack for GNU Smalltalk to be used in my Smalltalk GSM project. The first time I encountered SIP was around 2001 when we were excited to run a SIP Phone on our Linux powered iPAQs (Linphone on Opie).

For Smalltalk I started with looking at SipStack by Frank Shearar (who was very responsive to questions regarding his code and SIP in general). The main problem was that his stack was incomplete and as usual it is difficult to pick things up without knowing SIP and not knowing where the code was heading.

Grammar

I began with mapping the BNF SIP Grammar to rules for the PetitParser parsing framework. I have shortened some rules (e.g. because the generic rule is a catch-all one and I don't need the specialized form yet).

Complexity

The next big task was to deal with the concepts of a SIP Dialog, a SIP Session, a SIP Transaction and getting the relationship of these entities right. E.g. to place a call one creates an unconfirmed Dialog, prepares the INVITE Transaction and passes it to the transport layer. During this transaction one can get a reply that leads to a confirmation of the dialog. In fact one can end up with multiple confirmed dialogs (called forking, the target phones start ringing in multiple places and can be picked up multiple times as well). To make matters worse some answers need to be sent sent within the same transaction, some open a new one. My code seems to work now but there are some things I don't do properly (e.g. routing). I have documented this and if I have a need for those I will address them.

Processes

In the last couple of years I was mostly dealing with a single select IO model, coming to Smalltalk makes me use Processes to read and write from sockets. My basic model is to have a RX Process to use Socket>>#next and a TX process that will do SharedQueue>>#next and then Socket>>#nextPutAll. But with having multiple processes one is required to do proper locking, e.g. have a simple locking hierarchy (first pick the lock of the CallAgent, then the lock of a Dialog/Session). This all works nicely until I reached the point that I have two sub systems talking to each other.

For setting up a call I might have one leg coming from GSM and use the MGCP Protocol, the other leg might be SIP. To hang-up I will need to inform the other leg of the call about it, but this means I will need to have a locking hierarchy that first takes all locks of the left leg of a call, then all locks of the right one which in turns means me I need to dispatch messages without any locks being held. Now besides having RX from maybe a couple of different Processes, I also deal with timeouts that will come from different Processes as well.

Central dispatch

To put an end to this (and the complications in the code) I created a simple queue that will BlockClosure>>#value a block from a central dispatch process, this way I have serialized everything and can mostly ignore locking. I hope that Smalltalk VMs will have a nice M:N Process model in the future and that my code will evolve into having different dispatchers running on these N native processes.

Using the code

st> PackageLoader fileInPackage: #OsmoSIP.
st> transport :=SIPUdpTransport startOn: '172.16.254.55' port: 5061.
st> useragent := SIPUserAgent createOn: transport.
st> transport start. "Starts the RX/TX process"

st> call := SIPCall fromUser: 'sip:1000@on-waves.com'
host: '172.16.1.72' port: 5060 to: 'sip:9198@172.16.1.72' on: useragent.
st> call createCall: 'SomeSDP file'.
st> call cancel. "E.g. before it is connected"
st> call hangup. "E.g. when the call is connected."
st> call terminate. "End the call somehow.."

Cross compiling Qt and dealing with teams

2011-06-17T14:38:00.003+02:00

Let me share some notes on how Embedded Linux programming can be, if you decide the war/pain is over. We do this by looking at the desktop build and then look at the tons of stuff that you need for embedded programming.

Desktop

$ pkg-config --cflags libpng12
...
$ qmake && make
...

Embedded
$ . /usr/local/angstrom/arm/environment
$ pkg-config --cflags libpng12..
...
$ qmake-qt4 && make

Where is the difference?
There is little, you just use a cross compiler. The GNU/Linux Toolchain is prepared for cross compilation, pkg-config works with cross compilation.

What if I want different options
You can decide to use different -mtune and -march values, or you can easily regenerate the SDK targeting a different CPU architecture or just the instruction set level of it.

How is this possible
One creates a level of abstraction. The question is why do you want to suffer and don't use the abstraction known from the desktop? The above is given to you by the Yocto/OpenEmbedded and details can be found in the OpenEmbedded Manual. Read here for more details on creating a Qt Toolchain/SDK and using it.

But I want to recompile Qt
Maybe you want to change the qconfig.h, or you have a bug in Qt where you want to test things more interactively. You could either do this in the build directory of Qt of Yocto/OpenEmbedded but then again the created SDK contains all the dependencies for building Qt. All that is missing is the generic qmake spec (that takes its values from the environment) that would need to be copied into mkspecs.

But my application needs libpainful-to-build
The SDK is deployed from header files of packages, you can install more -dev packages into your toolchain. If you use any of armv4t, armv5te, armv6, armv6-nofp, armv7, armv7a, ... chances are high that there is already a build for your machine...

So why is it so painful?
I really don't know, tell me.

Going to the Qt Contributors' Summit

2011-06-15T12:02:00.002+02:00

I will leave Reykjavik around midnight, arrive in the early morning in Berlin and then try to get to the venue. This means I will most likely miss the keynotes but should be there by lunch time.

Besides being interested in technical topics like the networking stack, signals for Qt5, I think I will try to focus on the license part of the contribution model. With recent events like Lodys going after individual developers I wonder if there is something Nokia can do to protect individual developers that acted in good faith.

Calligra Flow

2011-06-10T10:23:00.002+02:00

Currently I am mostly working on (FreeSoftware) GSM networks, adding software probes, monitors, filtering and patching of certain messages. With any complex system this is spanning across multiple applications, multiple hosts.

Because we collectively know that a picture is worth a thousand words my technical documentation should have a picture, showing the computers, their IPs, some details of how the messages flow and such. Doing this kind of work with Free Software is still a lot harder than it should be.

Currently I am getting the job done with dia, the Cisco stencils are great, many other parts (printing, exporting, scaling, text editing) are not as great and this is where Calligra Flow enters the picture. I compiled a version some months ago and there were almost no stencils, printing was broken...

Yesterday I compiled a newer version of Calligra and there is some very nice progress with Flow, printing does work, saving did work too (no crashes), there are a tons stencils. So thanks a lot for the progress and I will try again in some months.

Collection of WebKit ports

2011-04-29T20:41:00.004+02:00

WebKit is a very successfull project. It is that in many ways. The code produced seems to very fast, the code is nice to work on, the people are great, the partys involved collaborate with each other in the interest of the project. The project is also very successfull in the mobile/smartphone space. All the major smartphone platforms but Windows7 are using WebKit. This all looks great, a big success but there is one thing that stands out.

From all the smartphone platforms no one has fully upstreamed their port. There might be many reasons for that and I think the most commonly heard reason is the time needed to get it upstreamed. It is specially difficult in a field that is moving as fast as the mobile industry. And then again there is absolutely no legal obligation to work upstream.

For most of today I collected the ports I am aware of, put them into one git repository, maybe find the point where they were branched, rebase their changes. The goal is to make it more easy to find interesting things and move them back to upstream. One can find the combined git tree with the tags here. I started with WebOS, moved to iOS, then to Bada and stopped at Android as I would have to pick the sourcecode for each android release for each phone from each vendor. I think I will just be happy with the Android git tree for now. At this point I would like to share some of my observations in the order I did the import.

Palm

Palm's release process is manual. In the last two releases they call the file .tgz but forgot to gzip it, in 2.0.0 the tarball name was in camel case. The thing that is very nice about Palm is that they provide their base and their changes (patch) separately. From looking at the 2.1.0 release it looks that for the Desktop version they want to implement Complex Font rendering. Earlier versions (maybe it is still the case) lack the support for animated GIF.

iOS

Apple's release process seems to be very structured. The source can be downloaded here. What I think is to note is that the release tarball contains some implementations of WebCore only as .o file and Apple has stopped releasing the WebKit sourcecode beginning with iOS 4.3.0.

Bada

This port is probably not known by many. The release process seems to be manual as well, the name of directories changed a lot between the releases, they come with a WML Script engine and they do ship something they should not ship.

I really hope that this combined tree is useful for porters that want to see the tricks used in the various ports and don't want to spend the time looking for each port separately.

Random notes on building Android

2011-04-24T09:24:00.002+02:00

This is mostly a note to myself. The easy way to build a profilable image for Android (e.g. Gingerbread). I am applying the patch from the bottom to build with framepointers. I am build an unstripped image, I use the environment script and then launch the emulator. With the default engineering mode opcontrol and oprofiled will be installed only the events will be missing from the image.


$ make TARGET_STRIP_MODULE=false
$ . ./build/envsetup.sh
$ help
$ setpaths
$ ./out/host/linux-x86/bin/emulator

diff --git a/core/combo/TARGET_linux-arm.mk b/core/combo/TARGET_linux-arm.mk
index ae1997c..66e55fd 100644
--- a/core/combo/TARGET_linux-arm.mk
+++ b/core/combo/TARGET_linux-arm.mk
@@ -56,7 +56,7 @@ TARGET_LD := $(TARGET_TOOLS_PREFIX)ld$(HOST_EXECUTABLE_SUFFIX)
 TARGET_NO_UNDEFINED_LDFLAGS := -Wl,--no-undefined
 
 TARGET_arm_CFLAGS :=    -O2 \
-                        -fomit-frame-pointer \
+                        -fno-omit-frame-pointer \
                         -fstrict-aliasing    \
                         -funswitch-loops     \
                         -finline-limit=300
@@ -68,7 +68,7 @@ TARGET_arm_CFLAGS :=    -O2 \
 ifeq ($(ARCH_ARM_HAVE_THUMB_SUPPORT),true)
 TARGET_thumb_CFLAGS :=  -mthumb \
                         -Os \
-                        -fomit-frame-pointer \
+                        -fno-omit-frame-pointer \
                         -fno-strict-aliasing \
                         -finline-limit=64
 else

Switching back to Ubuntu/Debian from Fedora

2011-04-18T14:20:00.003+02:00

My initial reason to try Fedora Linux was that a lot of good things (glibc, SELinux, systemtap, gdb) are mostly developed by Red Hat and put into Fedora. So it felt natural to try Fedora Linux to see the state of things. Now after more than a year with Fedora I am back to Ubuntu/Debian. For some simple reasons:

yum is ultra slow. When I try to search a package it decides to update the package database over the network, it is slow to start, it is slow to carry out simple operations. The other is that the various databases on the system are quite big and take a lot of space.

My upgrade to Fedora15 was hosed. Not so much to complain about as the distribution is not meant to be released the next week or such. So I was experiencing integration issues (e.g. setroubleshootd causing SELinux denies which will start the settroubleshootd), systemd not logging to /var/log/messages.

With Fedora15 I also had the opportunity to try GNOME 3.0 and in many ways it is promising, I don't think it is ready for end users yet though.

The biggest issue with Ubuntu right now is to get SELinux up and running, their policy modules are quite outdated.

Playing with a MSC over the A-link

2011-04-10T21:08:00.003+02:00

One of the benefits of having a dynamic language like Smalltalk is that it becomes easier to experiment. For one of my current projects I am dealing with some simple USSD message handling and want to see how much load my application and the backend system can handle. Using the Smalltalk TestPhone this becomes quite easy.

The first thing I do is to make up an Identity. This includes the IMSI and the AuKey (e.g. COMPv1 as found in libosmocore) and the next is to establish the A-link to the MSC.


> phoneConfig := PhoneConfig new.
> phoneConfig imsi: 'XXXX...' auKeyv1: 'AAAAAAA'.
> (server := IPAConfig new addr: 'A.B.C.D' port: 5000) connect serve.

The next part is to do 'procedures' (SCCP connections with a specific goal). Right now there is an implementation of the LU Procedure, the beginning of Call Control (no way to hang up or such) and and sending some bits of USSD (processUnstructuredData Request). To warm up I can execute a simple USSD query. The below code will wait until the operation has completed, it could also print out if it was considered a success or a failure.


> (server doUSSD: phoneConfig nr: '**1234#')  execute

Now to produce load I can repeat this, or create multiple processes. In the below code I am going to create four processes that each will do the query 100 times.


> 4 timesRepeat: [:time |
     [
        100 timesRepeat: [:each | (server doUSSD: phoneConfig nr: '**1234#') execute]
     ] fork.

Another thing missing is to add a timeout on each of these operations, e.g. if the MSC does not answer fast enough or such but I will blog about transactions and dealing with time-outs in another blog.

Tsunamies and other things

2011-03-15T15:56:00.003+01:00

This is just some random thoughts after the news from Japan, hiking the Taroko National Park in Taiwan, driving with the stop train on the Coast Line in Taiwan, studying chinese, and being away from Computers for a day or two.

My idea for the tsunami would be something like an instant balloon. In Taiwan every floor has an emergency exit through the window and a device to use to safely get to the ground, there is also the famous lantern festival were hundreds of them fly to the sky. So in my idea groups could board an emergency balloon and escape the water, and some minutes later land on the more calm water.

Angelique's idea was having a transparent Tsunami Ball like structure. This way one would float on the water and the structure would be strong enough to withhold the enormous pressure the moving water has.

On something unrelated I have been playing a bit with Android, a very badish game idea, and XMLVM. So it appears that JavaScript is somehow getting the new low-level virtual machine language.

How to make the GNU Smalltalk Interpreter slower

2011-02-13T21:03:00.002+01:00

This is another post about a modern Linux based performance measurement utility. It is called perf, it is included in the Linux kernel sources and it entered the kernel in v2.6.31-rc1. In many ways it is obsoleting OProfile, in fact for many architectures oprofile is just a wrapper around the perf support in the kernel. perf comes with a few nice application. perf top provides a statistics about which symbols in user and in kernel space are called, perf record to record an application or to start an application to record it and then perf report to browse this report with a very simple CLI utility. There are tools to bundle the record and the application in an archive, a diff utility.

For the last year I was playing a lot with GNU Smalltalk and someone posted the results of a very simplistic VM benchmark ran across many different Smalltalk implementations. In one of the benchmarks GNU Smalltalk is scoring last among the interpreters and I wanted to understand why it is slower. In many cases the JavaScriptCore interpreter is a lot like the GNU Smalltalk one, a simple direct-threaded bytecode interpreter, uses computed goto (even is compiled with -fno-gcse as indicated by the online help, not that it changed something for JSC), heavily inlined many functions.

There are also some differences, the GNU Smalltalk implementation is a lot older and in C. The first notable is that it is a Stack Machine and not register based, there are global pointers for the SP and the IP. Some magic to make sure that in the hot loop the IP/SP is 'local' in a register, depending on the available registers also keep the current argument in one, the interpreter definition is in a special file format but mostly similar to how Interepreter::privateExecute is looking like. The global state mostly comes from the fact that it needs to support switching processes and there might be some event during the run that requires access to the IP to store it to resume the old process. But in general the implementation is already optimized and there is little low hanging fruits and most experiments result in a slow down.

The two important things are again: Having a stable benchmark, having a tool to help to know where to look for things. In my case the important tools are perf stat, perf record, perf report and perf annotate. I have put a copy of the output to the end of this blog post. The stat utility provides one with number of instructions executed, branches, branch misses (e.g. badly predicted), L1/L2 cache hits and cache misses.

The stable benchmark helps me to judge if a change is good, bad or neutral for performance within the margin of error of the test. E.g. if I attempt to reduce the code size the instructions executed should decrease, if I start putting __builtin_expect.. into my code the number of branch misses should go down as well. The other useful utility is to the perf report that allows one to browse the recorded data, this can help to identify the methods one wants to start to optimize, it allows to annotate these functions inside the simple TUI interface, but does not support searching in it.

Because the codebase is already highly optimized any of my attempts should either decrease the code size (and the pressure on the i-cache), the data size (d-cache), remove stores or loads from memory (e.g. reorder instructions), fix branch predictions. The sad truth is that most of my changes were either slow downs or neutral to the performance and it is really important to undo these changes and not have false pride (unless it was also a code cleanup or such).

So after about 14 hours of toying with it the speed ups I have managed to make come from inlining a method to unwind a context (callframe), reordering some compares on the GC path and disabling the __builtin_expect branch hints as they were mostly wrong (something the kernel people found to be true in 2010 as well). I will just try harder, or try to work on the optimizer or attempt something more radical...

$ perf stat gst -f Bench.st
219037433 bytecodes/sec; 6025895 sends/sec

Performance counter stats for 'gst -f Bench.st':

17280.101683 task-clock-msecs # 0.969 CPUs
2076 context-switches # 0.000 M/sec
123 CPU-migrations # 0.000 M/sec
3925 page-faults # 0.000 M/sec
22215005506 cycles # 1285.583 M/sec (scaled from 70.02%)
40593277297 instructions # 1.827 IPC (scaled from 80.00%)
5063469832 branches # 293.023 M/sec (scaled from 79.98%)
70691940 branch-misses # 1.396 % (scaled from 79.98%)
27844326 cache-references # 1.611 M/sec (scaled from 20.02%)
134229 cache-misses # 0.008 M/sec (scaled from 20.03%)

17.838888599 seconds time elapsed

PS: The perf support probably works best on Intel based platforms and the biggest other problem is that perf annotate has some issues when the code is included from other c files.

Nokia and Windows Phone 7

2011-02-11T09:05:00.002+01:00

I had the opportunity to play with MeeGo for Handsets 1.1 and the upcoming 1.2 in December of last year on a Nokia N900. I very much felt reminded of Openmoko before I had joined them. There were a lot of promises, dreams but the reality looked differently. The Handset 1.1 release was not working at all besides being very slow the xterm and the window manager couldn't quiet agree, the pre releases of 1.2 worked a bit better but is still far away from being usable. The SDK situation is not much better. The madde approach is very promising, the tools to recreate the SDK actually work but the Qt installation is different depending on if you are using the X86 or the ARM SDK. E.g. the WebKit headers are only in the X86 SDK but were not in the ARM SDK. The reason is that both SDK package descriptions are in different files and they do not get synced.

Nokia announced today to enter a strategic partnership with one of the biggest software companies in the world, Microsoft. I am glad Nokia doesn't try to turn Symbian into a real operating system, I understand that with throwing out the GTK+ platform and restarting with Qt MeeGo is not close to challenge Android/iPhone. Nokia has missed out on purchasing WebOS from Palm and the best alternative is really to go with Microsoft Windows 7 Phone. On the other hand it is scary that Windows 7 Phone is the best alternative.

Wireshark dissector for TETRA

2011-02-02T21:57:00.002+01:00

The Professional Mobile Communication Research Group of Beijing Institute of Technology (BIT) was kind enough to send us their TETRA Wireshark dissector. They went through the specification and created ASN1 files out of the tables, I helped with the integration and cleaning to get the code into wireshark and the wireshark developers were kind enough to do a fast review and the code is now merged.

The next part is to extend the GSMTAP dissector to pass on the tetra bits to the TETRA decoder, I have already written the code, now I need to get it merged into the wireshark codebase too.

My TETRA agenda includes figuring out majority logic decision decoding with syndromes which is required for the shorted Reed Muller code used in the AACH, create a proper GNU Radio block for the decoder emitting soft bits that we can feed into our viterbi decoder... maybe I will manage to do this during my vacation in Taiwan... who knows.

Cleaning up the cellmgr_ng and turning it into a MGW/STP

2011-01-22T22:27:00.002+01:00

One year ago I was starting the on the BSC NAT application from a Hotel room in Munich, I was flying from Taiwan to Munich and was happy to see the snow from my window. Shortly after this I began to implement something we called cellmgr_ng. The job was to take MTPLevel3 coming out of a library and put the SCCP payload into the IPA protocol and send it to a MSC. As part of MTP Level3 I had to implement the link bring up and such as well (once again without having too much time to read all the specs).

The problems were as most of the times not the ones one would expect. The system behind the E1/T1 link was sending wrong SCCP messages, when a SCCP connection was not closed fast enough it would send out its own release message but was confused about the source local and destination local addresses. As a result I had to add connection tracking to the codebase, I am also rewriting/removing Point Codes to avoid crashes and make the MSC happy.

Last month and this month I had the pleasure to resume the work on this. My knowledge of MTPL3 is at least a bit bigger now, we have done the libosmocore split so I can use more features without copying code around and there was new demand.

The first big thing is to support a LinkSet with multiple Links in it. This allowed me to do a lot of cleanups, some are still to do but now the abstraction is a lot better and I will do testing and force link failures and see how the system is behaving. This will also end with VTY commands to manage links via the telnet interface, more statistics collected and reported, being able to start writing PCAP files for the linksets from the VTY command.

The second big feature is the addition of M2UA to the codebase. The code does not support fail over scenario's yet but this should be relatively easy to add inside the structure now.

The result is that we now have something that could be used as a MediaGateway and convert MTPL2/MTPL3/{SCCP,ISUP} into SCTP/M2UA/MTPL3{SCCP,ISUP} and convert the audio to and from RTP, the other part is a Signalling Transfer Point (STP) with some basic capabilities.

Using systemtap userspace tracing...

2011-01-17T12:30:00.005+01:00

At the 27C3 we were running a GSM network and during the preparation I noticed a strange performance problem coming from the database library we are using running. I filled our database with some dummy data and created a file with the queries we normally run and executed time cat queries | sqlite3 file as a mini benchmark. I also hacked this code into our main routine and ran it with time as well. For some reason the code running through the database library was five times slower.

I was a bit puzzled and I decided to use systemtap to explore this to build a hypothesis and to also have the tools to answer the hypothesis. I wanted to find out if if it is slow because our database library is doing some heavy work in the implementation, or because we execute a lot more queries behind the back. I was creating the below probe:


probe process("/usr/lib/libsqlite3.so.0.8.6").function("sqlite3_get_table")
{
  a = user_string($zSql);
  printf("sqlite3_get_table called '%s'\n", a);
}

This probe will be executed whenever the sqlite3_get_table function of the mentioned library will be called. The $zSql is a variable passed to the sqlite3_get_table function and contains the query to be executed. I am converting the pointer to a local variable and then can print it. Using this simple probe helped me to see which queries were executed by the database library and helped me to do an easy optimisation.

In general it could be very useful to build a set of probes (I think one calls set a tapset) that check for API misusage, e.g. calling functions with certain parameters where something else might be better. E.g. in Glib use truncate instead of assigning "" to the GString, or check for calls to QString::fromUtf16 coming from Qt code itself. On second thought this might be better as a GCC plugin, or both.

In the name of performance

2010-12-17T14:22:00.003+01:00

I tend to see people doing weird things and then claim that the change is improving performance. This can be re-ordering instructions to help the compiler, attempting to use multiple cores of your system, writing a memfill in assembly. On the one hand people can be right and the change is making things faster, on the other hand they could use assembly to make things look very complicated, justify their pay, and you might feel awkward to question if it is making any sense.

In the last couple of weeks I have stumbled on some of those things. For some reason I found this bug report about GLIBC changing the memcpy routine for SSE and breaking the flash plugin (because it uses memcpy in the wrong way). The breakage is justified that the new memcpy was optimized and is faster. As Linus points out with his benchmark the performance improvement is mostly just wishful thinking.

Another case was someone providing MIPS optimized pixman code to speed-up all drawing which turned out to be wishful thinking as well...

The conclusion is. If someone claims that things are faster with his patch. Do not simply trust him, make sure he refers to his benchmark, is providing numbers of before and after and maybe even try to run it yourself. If he can not provide this, you should wonder how he measured the speed-up! There should be no place for wishful thinking in benchmarking. This is one of the areas where Apple's WebKit team is constantly impressing me.

GSM in Smalltalk - a GSM Toolkit

2010-12-15T04:44:00.004+01:00

I started to play with smalltalk somewhere in February, more specific with the GNU Smalltalk implementation. Like it is with any new language and class library it takes a while to get productive and it took me until somewhere the last month where I finally started to do GSM handling in Smalltalk and thanks to laf0rge the code is now in a public repository and hosted along the other Osmocom projects. You can see all the subprojects over here.

So far I can create and parse the ipaccess IPA protocol, SCCP, BSSAP, BSSMAP and messages for Call Control and Mobility Management of GSM48. I also have a small Web-UI to connect to a MSC and then do a Location Updating Request and a Mobile Originated Call for a configurable IMSI (you will need to know your AuKey for that). It is very easy to parse and create the whole stack of messages, adding new GSM48 messages is easy as well (i still consider writing a program to get this directly from the GSM48 PDF).

This code is probably very well suited to do load tests of a MSC, e.g. find out how many LUs it can do a second, also to play easily with various quirks with it. Thanks to the nature of smalltalk one can easily change the creation of messages at runtime without needing to reconnect to the MSC. The next thing would probably to use it for fuzzing a MSC...

One note of caution, it is my first Smalltalk project and there are probably plenty of things I do wrong, I already know some of them and will work on refactoring it to a better structure. The next stop is nice ASN1 support, I will base it on the LDAP work.

Qt Mobility - Note to myself on making QCamera work

2010-12-14T08:43:00.002+01:00

Just a small note on Qt Mobility when using GStreamer to capture the image. You will need a QCamera, a QCameraImageCapture. So far so easy. You will also need to call start() but you will also need to create a QCameraViewfinder and set it on the QCamera. If you do not set a view finder, it will not will work. Maybe I will remember this the next time.

osmo-bsc in OpenBSC master

2010-11-22T09:05:00.003+01:00

A small service announcement. The bsc_msc_ip from on-waves/bsc-master is now dead and one can use osmo-bsc from master. It has the same functionality but is implemented in a cleaner way making it more easily extandable.

The biggest benefit of a open/free BSC equipment is the flexibility. If your network is a bit different to what people have thought out 30 years ago, you are now in the position to easily change that. E.g. your BSC could have different MSC connections and send different subscribers to different MSCs.

I am going to work on some new BSC features as well. With the new internal BSC API it should be possible to have different call routing based on some criteria, implement IMSI access control on the BSC, provide CBS services...

Best Future ever

2010-11-18T10:49:00.003+01:00

I'm attending some classes in University to finish up my degree. The classes in itself are close to an abuse and punishment (they are mandantory) but on the commute I manage to read one paper in each direction. I was reading Reflective Facilities in Smalltalk-80 and saw the best implementation of a Future ever. In their Smalltalk-80 dialect all message sends go through a #dispatchMessage: selector and the future is implemented in there.

dispatchMessage: aMessage

...

semaphore wait.

result become: self

^super dispatchMessage: aMessage

This implementation waits for a signal in the semaphore and then turns itself into the result of the operation. This allows one to write code like this

| f |

f := Future promising: [2.0 + 2.0].

f printString.

The execution of the message sent to the Future will block in the #dispatchMessage: until the result of 2.0 + 2.0 is available and then send the printString to the result. There is only one downside, most smalltalk dialects don't have a dispatchMessage: like this.

bahn.de and broken web services

2010-11-02T09:02:00.002+01:00

Once more I need to buy stuff online, now I wanted to buy a train ticket online as I am afraid the discounted price will be gone once I am at the trainstation...

The payment option I want to use is only available when you are registered, so I registered, using a '+' in my email address and then it didn't work. I assumed like many shitty websites they believe a '+' is not a valid sign in an email.. in this case my account name was taken. Then they tell me that under "My Train" I can register my bank details, well they don't provide a Link to that page... so I had to spend time searching it. Then I say I want to enter my bank details and after three pages in their dialog it fails because apparently it thinks it already has my bank details... well... no... I believe this error is due me not having registered my bahncard, so I tried to register my bahncard but it requires a pin code that they will now send me by snail mail...

Why do companies always hire the most crappy web porgrammers on this planet? Why do they invest more money to make stuff go wrong, than to just make it work. Another example is the train search and going back... if they would use a Web Framework like Seaside it would remember my source and destination, my bahncard details.. etc. Why is bahn.de paying for a bad website? Maybe they should hire experts, it can't be more expensive than these companies hiring idiots....

Searching for Cell Broadcast

2010-10-17T21:49:00.003+02:00

Is there someone that can provide me with a trace with Cell Broadcast Services on the Abis link? I am not really figuring out what should be provided to the BTS... I assume that the nanoBTS takes care of scheduling the message and one provides one big message instead of the four separate things?

Please comment, drop me an email or such.

Back from the DevDays

2010-10-17T18:59:00.002+02:00

Like many others I was at the DevDays. I happened to be in Munich for GSM related things but managed to attend half a day on Monday and half on Tuesday. I was very happy to reconnect with friends from Australia, Oslo, Brazil and the US and I am always impressed by the team that the Qt unit has recruited and still manages to recruit. Creating awesome technology is just the consequence of that and as written in earlier posts, Qt Quick is great and the Qt Creator integration is getting good as well.

While still having your attention. Is there any interest in low-level GSM stuff in the KDE Community? The OsmocomBB project is close to make phone calls in a commercial GSM network and we could use a simple GUI running on a Linux host OpenBSC on the other hand is the swiss-army-knife of a GSM network and ideal for tests. E.g. testing emergency calls, security research, generating stuff that is difficult on a commercial network, do prototypes...

And on a private note... I am back to Berlin...

misc notes on using tools

2010-10-05T17:58:00.002+02:00

gdb-heap is a new project to analyze the heap of a running application. Right now it only knows about python
addr2line can help you to go from a SEGFAULT/SIGBUS/Alignment trap to file and line, it needs some tricks to make it work on libraries..
you can use gdb with the app and use info sym 0xb7f1121d and info line *0xb7f1121d to at least get the function of where an application crashed.. (thanks to people in #gdb)

Excited by the idea of Soft Processors

2010-10-02T20:52:00.002+02:00

I was spending some time the other day looking at Qi Hardwares' latest project Milkymist One. It is a SoC with the mico32 core (CPU). The mico32 is a free Soft Processor that runs on various FBGAs.

From my point of view the biggest benefit of a Soft Processor is the ability to accelerate dynamic languages with high level (CISC) instructions. The primitive (natively implemented routines) of languages like Python, Smalltalk (SqueakVM, GST), Ruby try to remain stable over time. With a Soft Processor one could share the knowledge of the object layout with the CPU and handle certain routines directly in the CPU.

This is the old battle between CISC, RISC, the speed of RAM and complexity. If things are easy things are faster (RISC) vs. if the CPU knows more it can do things in a better way. I think for script languages (with its own instruction fetch, decode, execute) an implementation in the CPU could probably do a lot better.

The question will be how easily one can reconfigure the Soft Processor, how easily binutils will handle new instructions and how open projects will be to implementations in hardware.

I am excited about the Milkymist...

Deploying WebKit, common issues

2010-10-01T17:48:00.005+02:00

From my exposure to people deploying QtWebKit or WebKit/GTK+ there are some things that re-appear and I would like to discuss these here.

Weird compile error in JavaScript?

It is failing in JavaScriptCore as it is the first that is built. It is most likely that the person that provided you with the toolchain has placed a config.h into it. There are some resolutions to it. One would be to remove the config.h from the toolchain (many things will break), or use -isystem instead of -I for system includes.

The best way to find out if you suffer from this problem is to use -E instead of -c to only pre-process the code and see where the various includes are coming from. It is a strategy that is known to work very well.

No pages are loaded.

Most likely you do not have a DNS Server set, or no networking, or the system your board is connected to is not forwarding the data. Make sure you can ping a website that is supposed to work, e.g. ping www.yahoo.com, the next thing would be to use nc to execute a simple HTTP 1.1 get on the site and see if it is working. In most cases you simply lack networking connectivity.

HTTPS does not work

It might be either an issue with Qt or an issue with your system time. SSL Certificates at least have two dates (Expiration and Creation) and if your system time is after the Expiration or before the Creation you will have issues. The easiest thing is to add ntpd to your root filesystem to make sure to have the right time.

The possible issue with Qt is a bit more complex. You can build Qt without OpenSSL support, you can make it link to OpenSSL or you can make it to dlopen OpenSSL at runtime. If SSL does not work it is most likely that you have either build it without SSL support, or with runtime support but have failed to install the OpenSSL library.

Depending on your skills it might be best to go back to ./configure and make Qt link to OpenSSL to avoid the runtime issue. strings is a very good tool to find out if your libQtNetwork.so contains SSL support, together with using objdump -x and search for _NEEDED you will find out which config you have.

Local pages are not loaded

This is a pretty common issue for WebKit/GTK+. In WebKit/GTK+ we are using GIO for local files and to determine the filetype it is using the freedesktop.org shared-mime-info. Make sure you have that installed.

The page only displays blank

This is another issue that comes back from time to time. It only appears on WebKit/GTK+ with the DirectFB backend but sadly people never report back if and how they have solved it. You could make a difference and contribute back to the WebKit project.

In general most of these issues can be avoided by using a pre-packaged Embedded Linux Distribution like Ångström (or even Debian). The biggest benefit of that approach is that someone else made sure that when you install WebKit, all dependencies will be installed as well and it will just work for your ARM/MIPS/PPC system. It will save you a lot of time.

OpenBSC updates

2010-09-20T00:00:00.003+02:00

I am almost back from a trip to iceland, I was fortunate enough to focus exclusively on GSM. In the last week I have continued to build osmo-bsc (a true BSC based on code from the on-waves/bsc-master) branch, I have worked on the NAT to make the IMSI filtering more useful and to add (hardware) transcoding to the MGCP MGW. The NAT changes are already in production, the transcoding will follow soon.

IMSI filtering in the NAT/MUX. There is the use case that in certain areas IMSIs are not allowed that are normally allowed in your network (e.g. on a test cell, only allow test IMSIs). Now in general the first message (GSM0808 Complete Layer3 Information) should carry the T-MSI/IMSI and a NAT/MUX could filter based on this message (which I did). The reality is a bit different, in the case of a Location Updating Request most phones send you the TMSI of a different network, in that case the NAT/MUX will need to inspect every message until the real IMSI and TMSI (TMSI reallocation) is revealed and then apply the filter. I have added this part to the NAT and it appears to be working. While doing this we also encountered a crasher that was hiding and only happens on unreliable networks (e.g. not during unit testing).

For the last days I was working on a funny real life issue. In our case certain connections have an adaptive bandwidth. E.g. when a voice call is setup we will only slowly get the bandwidth we need. This means that on the first couple of RTP packets we will have a huge jitter. Now luckily we are using a proprietary MGW that puts out press releases of their field proven Jitter Buffer technology... which means that money was spend on PR instead of making it work and we had to run the audio through a different hardware based decoder/encoder to really handle the jitter. In the next couple of weeks we are hopefully able to replace this MGW with our own.

Using Video in QML - the hacky way

2010-09-04T03:09:00.003+02:00

For my current QML project I am required to embed video "into" the QDeclarativeView. I think the proper way will be to use the Qt Multimedia package that comes with a QML VideoView but for this project the customer wanted to use a specific proprietary media player (I wish I could call names).

I started with creating a QWidget called VideoWidget and added three Q_INVOKABLE methods, one is setPos(x, y, w, h), setUrl(url) and play(), and bind one instance of that widget to the QML runtime. In the QML code I have one Rectangle {} where the video should be and us onWidthChanged, onHeightChanged, onXChanged, onYChanged to call the setPos on the QWidget. This mostly works but on start the x,y is not set properly but I am lucky as my video is not shown on start.

The next part is to actually host the video. In the first incarnation I was using the QAxContainer to host activeX content. Normally I use the testcon.exe, load the control, figure out which methods to invoke and then convert it to dynamicCalls. The only issue I faced was that ActiveX control just locked up on my copy of Windows7 but thanks to wine I could figure out what to do.

The only thing that was missing is to position the video on top of the QML content that appeared to be more difficult. The natural choice is to have the QDeclarativeView as parent to the VideoWidget, but that didn't work, having a common QWidget as parent to the VideoWidget and QDeclarativeView also didn't work. I had to resort to have two top level windows and trying to make them appear as one, so I had to handle QWidget::event(), use QWidget::setAttribute to have no taskbar for the video, etc.

Now this award winning closed source media player has quality and performance issues and I needed to try something else, sadly loading VLC in testcon.exe does not work properly, as it refuses any dynamic call to it, so in the end I am using Phonon for the video playback. The great thing about that solution is I can remove the hack with two top level windows and have a common QWidget as parent, and in the future I might be able to embed the videowidget into the QGraphicsScene of QML.

The bad part is DirectShow which doesn't handle rtsp/mms and I am failing to find a ASX stream which works over HTTP. The best I can find is the flumotion demo site but that doesn't offer a standalone URL I can throw to my player.

Detecting IMSI Catchers

2010-08-29T17:22:00.003+02:00

I was wondering how and if IMSI Catchers can be detected and how much work it would be to support that in OsmocomBB. The only problem is that I have not seen any IMSI Catchers, have not written one and have read too little about it.

So how does an IMSI Catcher operate? Well, I have no idea and need to speculate. In contrast to a real network it is only meant to be used by a few Mobile Stations (MS), it is supposed to be the most attractive Cell, by nature it should have more SDCCH than TCH.

The next step in thinking is to figure out how to achieve some of the above goals. As it should only work with a few handsets the System Information might/should contain an Access Class allowing only certain IMSIs to attach, one should see a lot of Location Updating Reject messages or unanswered messages. To be the most attractive cell the signal strength should be higher than the others, the channel configuration might be guessable by looking at the RACH and see which kind of channels are requested and assigned (keeping track of them).

The next thing would be to use a database like OpenCellID, or some other database and check if the LAC/CI has been seen in this area, comparing the SI to the other SIs of the same operator...

I plan to start such a thing as it is mostly about statistic and stochastic and I have become too rosty on these topics. The question is how likely (t-test) is that this SIx is coming from the real network, how likely is that this RACH pattern is coming from the real network.

any ideas and comments?

Going from dummy to real data

2010-08-29T15:32:00.003+02:00

I was writing about my current QML project earlier and now was the time to go from dummy data to real ones. For the prototyping phase we were using models created with ListModel and the attributes we need in the UI. For some JavaScript code that is called to execute actions we were using ListModel.get(index) to get the item and then execute code.

The QML Documentation was bringing me very far. I was using the rootContext of the QDeclarativeView to add a controler object and the models to the QML runtime. The controler is called when the models needs to be updated, e.g. I am calling them from ScriptAction on state transitions. For QML a model only needs to set the rolenames and then handle the different roles. This was working quite nicely, and the old modeltest can help to debug the model.

After the above a simple ListView { model: myModel } will work nicely. Now I was writing about that we are using ListModel.get and I assumed that it would just magically work for my model as well, e.g. using the rolenames as well. From what I have seen that is not the case. This means your model needs to implement a Q_INVOKABLE QVariant get(int index); and internally use a QMap and somehow duplicate code that is already there.

Another problem came from updating models. This is something that does not happen with static data, but happens in my case. I was using the big hammer with beginResetModel() and endResetModel() in the model and on the QML side one can use onCountChanged in the ListView to handle a massive update and execute JavaScript. In my case this was used to automatically select the first item and set the currentIndex of another ListView.

More on bad webservices

2010-08-28T04:00:00.003+02:00

I am using postfix on my server and it has the nice feature that it can move mails with +something at the end into a folder I want. This allows me to see who is selling my email address to which company and such. Now the downside is that many forms of webapps claim that my email address is not valid.

So today I was over at finanzen.net to try to register a new account and instead of telling me that they do not like my valid email address, they say the user name was already taken. I tried with many usernames (coming from pwgen) and all the same, it just started to work when removing the '+' from the username...

Now I tried to use their contact form to tell them that I would like to use their service but they have hired a company that has no clue about web programming... well it is ofcourse broken too...

It is 2010, there are so many good web frameworks around, why are proprietary webapps still so bad? Who is writing all that crap?

Tips and Tricks for QML

2010-08-19T13:36:00.003+02:00

I would like to share some more information about how I resolve my QML issues. In my last post I got a friendly reply of hiding states and such information of a Component inside an Item of that component. This was a very good hint.

My application is making heavy use of delegates for the ListView, PathView, Repeater and sometimes qmlviewer stops displaying content with a warning: "QDeclarativeComponent: Component is not ready". In contrast to the many other places this error message is not very helpful. In all of my cases this error came from a syntax error (unbalanced {}), using a duplicate property or having fun with lists. So check your model (if it is static) and your delegate file for syntax errors, e.g. load them in the qmlviewer and see if the error message is changing.

IIRC writing something like Rectangle { Item {} Item{} } is just a short form for writing Rectangle { data: [ Item {}, Item {} ] }. Now with the shortened way one does not need to use "," at all, with the real list way, one may not have a "," after the last item. It is good that the syntax checker is so strong, it just happens to conflict with my C99 usage.

The application requires some kind of table display and in general ListView/PathView do not allow such models. My not so unique idea was to have a ListModel and then another ListModel hanging off as an property. So the first approach might look like the one below.

import Qt 4.7

ListModel {

ListElement {

moreData: ListModel {}

}

But it is not supported by QML and you will get a nice error telling you that this is not possible. The workaround is to use the simple [] way to create a model that is working on the views.

import Qt 4.7

ListModel {

ListElement {

moreData: [

ListElement {}

]

}

With the above I can use a Repeater/ListView inside a Component/Delegate with a model of this row and I have a small table view. With my current approach I have to use a Flickable and two Repeaters increasing my memory usage but that is acceptable for now.

QML and dealing with states

2010-08-10T18:33:00.003+02:00

In the last days I have resumed my QML work (I had a small break to work on the MGCP GW code of OpenBSC to fix some real world issues) and there is one kind of issue I tend to run in and I wonder how others are solving it. Let us imagine we have a QML Component for a Button. The Button itself can hold a text (property alias text: buttonLabel.text) and the button has three states (enabled, focused, pressed) that depend on the MouseArea that inside the button as well. Actually this approach is directly coming from the many nice examples and demos provided by Nokia.

Now the problem is... I'm using the Button in many places and depending on some other external state the label of the Button should change and I keep writing things like this:

UI.Button {

id: text_button

text: 'My Text'

MouseArea {

anchors.fill: parent

onClicked: { console.log('clicked'); }

}

states: [

State {

name: 'some-state'

PropertyChanges {

target: text_button;

text: 'Other text'; }

}

/* more states... */

]

}

And then I am going to wonder why things don't work. The first issue is that my own MouseArea will receive the mouse click and the button will not work... but that is easily fixed. Do not add a custom MouseArea and have a clicked signal inside the button component. The second issue is with the states.. the above code is breaking the focus/pressed logic.

The way I am dealing with this kind of problem is to move the state into a parent and control the text from there. What is the proper way of solving this problem? Creating multiple buttons and control the visibility/opacity from outside? Duplicate the component states inside the custom states (cross product of my states and the component states)?

I know that talking about errors is bad as this will make people remember the wrong solution but I hope that other people stepping into these kind of problems will remember this as a possible problem..

Coscup2010/GNOME.Asia with strong web focus

2010-08-10T18:08:00.006+02:00

On the following weekend the Coscup 2010/GNOME.Asia is taking place in Taipei. The organizers have decided to have a strong focus on the Web as can be seen in the program.

On saturday there are is a keynote and various talks about HTML5, node.js. The Sunday will see three talks touching WebKit/GTK+. There is one about building a tablet OS with WebKit/GTK+, one by Xan Lopez on how to build hybrid applications (a topic I have devoted moiji-mobile.com to) and a talk by me using gdb to explain how WebKit/GTK+ is working and how the porting layer interacts with the rest of the code.

I hope the audience will enjoy the presentations and I am looking forward to attend the conference, there is also a strong presence of the ex-Openmoko Taiwan Engineering team. See you on Saturday/Sunday and drop me an email if you want to talk about WebKit or GSM...

MGCP Road to Stability

2010-08-08T00:17:00.003+02:00

I have spend the last week differently than I had planned. I have been to Iceland, the Hotel still didn't restart their DVB-S receivers or renewed their smartcards. So 13/16 Channels are stuck in different set-top box messages and only RTLII of Switzerland, BBC and Eurosport are fully functional. The sad part is thet one could receive China's CCTV program here...

I have spend the last week working on the MGCP Gateway of OpenBSC to allocate the network and the BSC/BTS port separately, to add a feature to forward the RTP stream from the BTS IN/OUT, NET IN/OUT to another system. On this different system one can use something like GStreamer to decode the stream and listen to it. This can be useful to debug when the voice doesn't arrive where it should.

In a network simulation with Linux's netem we have tried to simulate a bad vsat link and wanted to see how big the latency/jitter can be to still have an acceptable voice call and now I will play a bit with RTP jitterbuffers... This allows me to look at GStreamer once again to see if their jitter buffer is finally working.

Progress in OpenBSC

2010-08-04T10:29:00.004+02:00

This is just a small list of things that have happened in OpenBSC:

Introduce a GSM 08.08 BSC API to separate channel management from the MSC Code. This has also killed reference counting of our logical channels and we release channels a lot faster.
Toying with USSD. I started to play with ASN.1 and generate USSD messages, right now we can send a unstructuredSS-Notify down to the Cellphone and play with the alerting types, sending a notifySS as part of the Setup message is not liked by phones yet.
Splitting out the SCCP code into a new libosmo-sccp library, I have also put my MTP-Level3 code in there, so maybe we should call that library libosmo-ss7.
I was writing my first C Node in Erlang to parse SCCP messages in C and return the result to erlang to Erlang code.
The mgcp code can now dynamically loop/unloop the stream, patching the SSRC, seqno and timestamp.

In the next couple of days/weeks I want to continue working on the BSC API and then completly nuke the current on-waves/bsc-master branch as everything is merged back to master in a way better structure than it was in the branch. But today I am trying to make my code and the nanoBTS crash.... and figure out why it is doing that.

HTC Desire/Android GSM Protocol Issue

2010-07-25T22:09:00.002+02:00

I was playing with ASN1 and Supplementary Services over the Weekend. My goal was to provide extended user information during a call setup. So the first step was searching for information of how it could look like, this involved going through the GSM Spec, the 2nd part was wrestling with asn1c to generate some dummy data (as I couldn't find a trace doing that), the 3rd part was being able to generate that from OpenBSC and send it to the phone.

Well, long story short. It doesn't work, the better phones ignore the data I am sending, all the others refuse to show incoming calls, the Phone failing the most is a HTC Desire with Android. First of all OpenBSC has a pretty simple USSD implementation and it doesn't like that, now when you place a phone call (while Android decides to do a USSD request in the background), OpenBSC will close the channel, not do any Call-Control and on the Android one screen shows the call is terminated, in the decoration it is still active, then the wakeup/sleep logic gets confused, you can not hang-up and then the phone is restarting.

The executive summary: Not implementing USSD in your network (we send an error and such) can make Android phones reboot if someone is placing a call at the time android retries to send the USSD query.

Debugging hints I

2010-07-20T21:19:00.004+02:00

Being an Engineer one need to resolve problems and sending a mail to a mailinglist and asking for help is most of the time not the problem solving skill you should use. So here is a list of some easy hints...

Finding a build error (using make) in a log file. You might use a tool like OpenEmbedded that keeps the log file around and you have a build failure. First of all check the end of the file, but one might have compiled with -j X and the failure was a bit earlier. The best way to find the error is to search for '***'. E.g. inside or less or vi type (/\*\*\*) and it will bring you to the first compile error... GNU make is kind enough to flag the error.
The compiler is bitching about something that might not make any sense. With C languages a program called CPP, the C PreProcessor, is ran over your code and it might include funny headers changing your code. What I normally do is to use the compile line as seen in the terminal and replace the -c with a -E as this will only preprocess and then look at the code. The pre processed code is long, search for includes from places you don't expect and such...
If a program points you to a log file, read it. In the case of autoconf (configure) a config.log is written, now the bad news is that the error is not at the bottom of the file. What I do is to look at the text of the last test and search for that inside the config.log, it brings me to the failing test...
Dealing with remote crashes. You have a segfault but don't know when/where it is happen and it is hard to reproduce? Just use ulimit -c unlimited in a shell and run your script/applications, the kernel will place a core dump in the directory and you can use gdb and your binary to inspect it.
Running a libtool based binary/lib from inside the build directory. You can use libtool --mode=execute gdb tests/sms/sms_test and then libtool will setup the environment and invoke your command..

more hints at a later time...

Anti-Pattern 23 - Make sure to not get feedback

2010-07-09T11:32:00.004+02:00

Create a website to compete with an established one.
Make something bad like handling cursor/space keys in the side and alienating people using keys to navigate.
Create the impression one can give input, place a feedback button.
Use a form, allow the user to enter text.
Make sure the submit is not just a simple submit but make it go through javascript so you can reference variables that do not exist

If you follow these steps, you are guranteed to burn money, lose busines and you will not get feedback from your customers.

Hybrid Application Example with QtWebKit

2010-07-05T16:58:00.010+02:00

In general one of the fascinating aspects of WebKit is the focus on just being a Web Content Engine as it can be seen in the Project Goals. One of the results is that one can easily build a Web Browser around it, or embed it into a Mail Client, a Chat Client, or into your application to handle payment in amazon, display Wikipedia or similiar things.

On the other hand it is possible to embed native widgets into the Web Content using WebKit/GTK+ and QtWebKit, or use the JavaScript Engine API to bind native objects into the JavaScript world. I have started using the moiji-mobile.com name to push WebKit usage for these kind of hybrid applications. I am going to create examples showing how to create applications using QtWebKit, how to deploy them on real hardware, howto handle content development on mobile devices and more.

The first part of this series is a simple example of what could be a SIP based desk telephone using QtWebKit and native technology. In this example I am embedding QObjects to provide information normally not available in the Web, I embed a QWebView inside the content to provide a browser and I am painting on top of the HTML content from the QWebView to provide touch feedback and the code can be found in my git repository.

Adding a route with android's route

2010-06-29T06:43:00.002+02:00

Today I needed to add a route on the android shell but was greated with "Invalid Argument" failures, there is no "-h", no man page. So we will have to use the sourcecode.

First of all the code is in /system/core/toolbox/route.c

and it supports three types of argument..

route add default dev DEVICE
route add default gw IP dev DEVICE
route add -net IP netmask NETMASK gw IP

Qt Quick vs. Android XML Layouts

2010-06-29T04:00:00.003+02:00

In the last month and next couple of weeks we were/are developing a similiar application for Android, iPhone and the Desktop (using Qt and Quick) with native technologies. I will focus on Android and Qt as both are Open Source and we were developing them in a similiar fashion with one designer and one developer interacting with each other.

In both cases the application has a couple of views mainly consisting of a ListView of some sort and I will explain how this has been realized for Android and then with Qt Quick and provide some summary of what has worked well, what Qt Quick could learn from that. I have not worked with Qt Quick or Android prior to this project so I might have used the technology in the wrong way, and at your option you can either hold this against me or the technology.

We have started the project by doing the Android application. The model for my data is based on the abstract class called android.widget.BaseAdapter and I needed to implement a couple of functions. The first is the answer how many items I have, return an Object at the given position, an identifier for a position and finally a function that creates a View.

This function is called getView(int position, View convertView, ViewGroup parent) and the common usage is to check if convertView can be used to display the item at position and then modify (e.g. set the iccon, text, hide things on that view) otherwise we were using the LayoutInflater to create a View from one of the pre-defined layouts and set the items.

In Android one can describe Layouts in a XML Document, or use a still very basic Layout Designer (that will mess with the indenting of the XML), one unique feature is one can have different layouts for different properties, e.g. one can create a layout that will be only displayed if the application is in Landscape, uses Docomo as Operator and more. This is a very powerful system and we have only used it to have different layouts in portrait/landscape so far. The next important thing is one can ask the LayoutInflater to create the View for a given identifier, the next part is to find the items inside that view. Each View has a method called findViewById that is searching the children for that id.

One common pattern is to not call findViewById everytime one needs to access a widget but create a ViewHolder class with references to the Views and use View.setTag(Object tag) on the top View to set that ViewHolder on the View.

So how did the Designer and Developer interaction work here? First of all the Designer was responsible to create the View for the various parts of the UI and I was responsible for creating that BaseAdapter derived class, implement the getView method and set the right text/icon/visibility on the items of the layout based on the item that is to be displayed.

How did that work? In general it worked great, I could create a UI that worked, the Designer could make it look nice, the biggest problem (besides the basic UI editor, fighting with the RelativeLayout class) is that during redesigning a layout id's were dropped from the Widgets and I had to assist the Designer to identify the missing IDs and add them back to the Design to keep things working and display the right thing again.

The Quick project is still work in progress and a lot like the Android project, the Developer creates the code (or states and interaction) and the Designer makes it look good. Our ListView's are backed with classes based on QAbstractListModel (using setItemRoles to export the roles as QString) and the designer is forced to edit the files by hand right now.

The thing that really works better with Qt Quick is that the delegate (equivalent to the getView in Android) is in the hand of the Designer. This means that the Developer only needs to export the right strings for the roles from the model and the designer can use that to create the UI.

From my point of view the Qt Quick approach is really superior to Android's XML Layout system, with Qt Quick we could get started faster by having a full click dummy (create the ListModel and fill it with dummy data), allowing the designer to fill the UI with content from the first day of the project, and by having the delegate in the Designers hand we have no "losing" id's in the diff of an unformatted XML Stream. The only thing that has no proper equivalen in QML/Quick is basing the state on external properties, or with other words, I have not seen any documentation for some basic system/device properties.

First Steps with Qt's Quick

2010-06-23T16:52:00.006+02:00

I am very excited by the Qt Quick technology and I have finally found a reason to use it and gain some experience with it and want to report from my first hours of exploring it. The first time I have seen Declarative UI was with Enlightenment and the Edje framework. I liked it so much that I registered Epicenter five years ago. The plan was to create a handheld framework using declarative UI.

Now Qt Quick is not just a copy cat of Edje but adds some nice improvements over it. The first one is the usage of Javascript instead of the C like language, better integration with Qt's Object Model (properties) and libraries/plugins. From my observation at Openmoko, our developers kept a set of C preprocessor macros around to do coming things with edje with Quick it seems to be better as one can import libraries and such.

The most common mistake I have made so far is dealing with the width/heigh of an Item. In my case I created an Item, placed a MouseArea (to deal with user input) of the size of the parent (anchors.fill: arent) in it and then also add some Text (as sibling). Now it appears possible that the Text is bigger than the parent item. For performance reasons no clipping is done by default so it renders just fine, just clicking doesn't work. Which is bringing me to my debug trick...

I place a Rectangle { anchors.fill: parent; color: 'blue' } inside my item and then I actually see where it is. Another nice thing would be an Xray view showing the border of each item, their id but only in black and white. My solutions for this problem so far (from my first hours of using it) is to either use a Row/Column which seems to get the widh/heigt updated based on its children, or in some cases place a width/height inside the Item itself.

This is bringing me to the biggest issue with the qmlviewer and also an idea on how to resolve it... In the last couple of month's I started to contribute to GNU Smalltalk and looking more into Self, Smalltalk-80 and such. The clever Xerox PARC people invented Model-View-Controller as part of Smalltalk-80, Qt adopted some parts of it with Qt4. In the meantime something called Morphic emerged. Morphich is a direct-manipulation User Interface. This means one is creating the UI inside a running UI by composing it from simple objects (just like Quick). In contrast to it one can inspect each element and interact with it, change it at runtime without restarting. This allows faster changes, and easier debugging in case something goes wrong. E.g. it easily answers the question of what is the width of that?

So for the immediate future I would like to see something like the WebKit inspector emerge for QML. This would allow to inspect the scene graph, change it from the console, has some simple hit testing to inspect one element, has the JavaScript Debugger and Profiler available, the timeline... and I am pretty sure to not be the first one to want it.

How to make sure your online business is failing and ways to fix it.

2010-06-22T16:53:00.004+02:00

This is a more calm rant of a post I have done earlier today and removed. Everytime I have to use a proprietary online shop to buy something the thing is just of very poor quality. One example of last year is Best Western, their bookingsystem was so bad, I had to book the Room through Yahoo Travel.

So here is my top six of how you can make sure your online is failing (and how to fix it):

Make sure your webpage only works with Flash, Internet Explorer 5 with the right locale, a Java plugin to enter the CVC number of your credit card... It is not that JQuery or website testing frameworks like Selenium needs to be invented...
Make sure the user experience is bad, try to nest the content in as many frames as possible, make sure the user needs to scroll with at least two scrollbars. The easy solution is, test your content with 1024x768 and don't use frames...
Provide options on your purchase and then send long, very long boilerplate emails that the user needs to call your expensive hotline to pick another option. Provide fewer options, make sure the payment process is working for them, or use another good company for the payment.
Provide wrong errors codes. E.g. when a flight is fully booked, claim the credit card number or the postal address is wrong. Blame it on the credit card company, it is easy, unless the customer calls the credit card company...
Let the user enter his address multiple times, insist that he must have a landline number...
Have a hobby programmer decide that + is not a valid part of an email address...

GPRS issue resolved

2010-06-21T09:40:00.003+02:00

Hi,

with some more debugging and fun with wireshark scripting and looking a pretty obvious issue has been resolved. Now GPRS for us is actually using IP, UDP, NS (some simple address and type of the messages), BSSGP (protocol between SGSN and BSS) and for actual data there is LLC at the end of the BSSGP. The LLC is part of the BSSGP payload as TLV (Tag, Length, Value).

I created a simple setup that worked. It involved getting the traffic from the BTS, relayed with a simple smalltalk script (I had to do some fixes to GNU Smalltalk), and then send it to another SGSN. With a small variation of sending the data through our proxy I made the nanoBTS crash.

From observations I found that the other SGSN is padding the FLOW-CONTROL-BVC-ACK and FLOW-CONTROL-MS-ACK packets to 28 bytes, but padding/not padding had no effect on the crash.

The next observation was (before I tried doing it manually) that I now have each packet twice, once coming from the SGSN and how it looks after our proxy, apparently the proxy truncated the UDP packets....

So what errors have happened?

The nanoBTS accesses random memory with short LLC frames and crashes, instead of crashing it should send a STATUS (I think BSSGP) returning our
The wireshark BSSGP dissector does not check the size of the LLC frame (I created a bug report with a patch)..
The proxy code was not reading the whole datagram and we had to increase the size, according to the spec the maximum size is 1600 byte for Framerelay... we now have a slightly bigger message...

Fun with GSM...

2010-06-10T12:32:00.003+02:00

In the last couple of weeks I was traveling, searching a flat (a total failure so far), working on the Android Project, something else and of course GSM. I am having fun kicking another blackbox and try to figure out why it is catching fire and starts burning.

We assume this is related to GPRS (that got recently added to OpenBSC thanks to the sponsoring of On Waves) and something within the UDP messages we send. Now the GPRS protocol sandwhich can compete with the Whopper 7. We do have UDP, then NS, then BSSGP, then LLC and as the BTS is crashing it has failures handling UDP, NS or BSSGP. Of course we are using the excellent Wireshark and filter the packet dump for malformed but there is nothing that Wireshark is considering malformed... Now Wireshark allows to write dissectors, taps and more with LUA. So my next step was to write some LUA scripts to analyze a stream that makes the BTS crash and one not. The LUA Script can be found inside the OpenBSC git repository.

Today I tried to continue with the BSC/MSC split but there is quite some stuff to fix inside the code before this will become a reality, e.g. in the SMS code we have no place where we consider the task of sending SMS to a subscriber as "done". Also looking at the many places where we stop a "transaction" makes it highly likely that we do have leaks there...

Android mediaserver segfault

2010-05-19T14:07:00.001+02:00

Hmm... we just want to stream some video... giving android a MPEG TS stream does not work, giving it a HTTP Pantos, does not work, giving it a ms3u playlist segfaults the mediaserver. fun

Android and Java

2010-05-19T11:22:00.004+02:00

I am now playing with the Android UI/Java Code and right now I have something really really simple to do. Over time I have a bunch of URIs to GET or POST and after it is done I want to have the status and the data. On top of that it should not block the UI thread. Android is using the HttpComponents of the apache project. The good thing about it is that Google was not reinventing the wheel, the bad thing is that the code is Java. So I will need to write 200 lines of code to setup a threadsafe ConnectionManager and a HttpClient on top of that and then I need to write my own thread pool... Which is bringing one back to the main complain about Java, it is an overengineered system...

But in general I think the Android UI classes are very promising, yesterday I implemented the preferences for my application and it is really really easy, better than anything i have dealt with before. One needs to create a xml/preferences.xml, one can use "categories" and each category can have different preference items, even custom ones. The system will automatically create the view for that (like with the most xml documents for Android) and it will take care of saving/loading the settings.

RR Channel Release and USSD

2010-05-19T11:15:00.003+02:00

Traditionally we have a struct gsm_lchan representing a logical channel. It has a given type based on the physical allocation (SDCCH, TCH/H, TCH/F), a given mode (traffic or signalling) and the trx and timeslot it is on. We also have a struct gsm_subscr representing a GSM Subscriber in our combined HLR/VLR. Whenever a Channel is used for a subscriber we are taking a reference count and assign the subscriber to the lchan. Whenever releasing the lchan we check the subscriber pointer and if it is set send a proper RR Channel Release and a SACH Deactivate.

For the real OpenBSC BSC we have no HLR/VLR in process, we have no local database as I am paranoid on running out of diskspace on a system that should work without maintainenance. But this means that on a channel release I am not running through the GSM04.08 RR Channel Release and so far this has worked out well. Well until we have figured out that USSD is broken. The workaround for now was to create one dummy GSM Subscriber and assign it to the channels, the proper fix is on the way by a proper split between BSC and MSC functionality but that is taking quite some time.

Programming for Android

2010-05-07T05:39:00.002+02:00

In previous lifes I have done development for Windows Mobile, the iPhone and now it is time to use Android. In general I had a negative bias towards Android because of the fact that they create an island on an island and have shown poor interaction with many upstream projects.

The Poor

The Android SDK GUI is really really poor, it is worse than my Opie SDK attempts I did within a couple of days. It is really schocking how poor the UI design of Google is on such an important tool. The biggest issues for me are the usage of real estate. This starts with the new project dialog not fitting on 768 height (the resolution of my 13,3" notebook screen), it continues with the Android Log in the debugger view. It is a single Tab inside a Tab Widget with a huge amount of padding. With my screen setup one line of log fits, the text of it can not be selected and if the line is bigger than the 1366 width of my laptop I have bad luck, it can not be scrolled.

The other thing fewer people do is using JNI. The native library is placed into the android package and then uploaded to the target, when recompiling and then asking eclipse to launch the application the dependency on the library is not checked, deleting the Android Package is not recreating it as well, one has to make a change in a java file from within the IDE to get the new lib into the package. This is certainly not as poor as the above.

The Good

This is where Android starts to win me. There is plenty of Source Code to look at. It is starting with the Examples from the SDK/NDK, the huge amount of code for the core applications and various other resources. The adb utility is really awesome as well. It allows you to get a shell, upload/download files and to display the whole system log. I really appreciate a well crafted logging system and the Android one seems to fall into this class.

Recipe for desaster

2010-04-22T14:37:00.002+02:00

I really hate closed source systems, every time I have to deal with it, it turns out to be a waste of time dealing with it. Right now I'm dealing with a closed source BSS (I wish I could call names), I can make it fail by sending a very simple message, a message that is crucial for everything (SMS, Call) to terminate on the MS. Of course an expensive support contract was signed but apparently it does not buy more than the right to create a ticket in a bug tracker and let it rot there. Never ever make your business depend on a black box!

Thoughts on Fedora 12 and 13

2010-04-19T03:59:00.002+02:00

I was a bit unhappy with the performance of Ubuntu Karmic and Ubuntu Lucid (mostly the start scripts to handling filesystem errors at all...) and decided it is time to try a RPM based Distribution again. I really don't like to wait for a download of a DVD and then to upgrade the whole system with new software after the installation. Coming from Debian I totally love the slick network installer. This means I have to install my RPM based distro like this or I will stay with Ubuntu or go back to Debian. I have tried doing with OpenSUSE some time ago and it failed miserable, so now it was time for Fedora. So here is a random list of notes I hate about Fedora.

Netinstaller needing a 200MB download. One can build a whole GNOME image with that...

The guided partitioner is nice and fails. It picks a 500MB boot partition which is not enough if you want to use pre-upgrade later to upgrade your system (which silently fails when going out of disk space...), so I want to increase the boot partitions size, but I need to delete the LG and PV of LVM before.. So at the end my guided partitioning was all manual as I want to change the size of one partition... this could be handled way better.

YUM is awfully slow and one is not encouraged to do the equivalent of a "apt-get dist-upgrade" with it, one needs to use a installer CD to do the upgrade. This seems to be really backward.

I had to learn the wonders of rpm --rebuilddb during a failed upgrade...Something upgraded the berkeley db version or such.. and the package database was corrupted.

Removing pulseaudio (it takes like 50% of CPU time on my netbook) removes bluez..

Besides that Fedora seems to be a robust and well maintained distribution which contains really recent versions and sometimes even the future... like systemtap with a utrace enabled kernel to trace user applications.

Got a new MSI X340 Laptop

2010-04-19T03:52:00.003+02:00

After the break down of my beloved Macbook I decided it is time for a new notebook. So I went out to the various markets and searched around. I was in search for something slim but still powerful. So it could be a Macbook AIR? Looking at the specs revealed it contains NVIDIA GFX, so that is a no go. Then I saw a Sony X-Series... very very slim... then my math failed and i thought it is cheap... well Sony's are never cheap and the repair is always outrages... I stumbled across the MSI X-Series and i really liked it...

So my X-340 has a Intel Chipset and is a Centrino2, it is one of these ultra low voltage CPUs... and the Linux supports rocks. Wireless, Bluetooth, Camera (not that I need it), Sound, Touchpad, Graphics (intel gma4500) all work out of the box. The CPU seems to be powerful enough for what I want to do.

So far I seem to have made a good pick.

GSM RACH Bursts and Paging Requests

2010-04-18T11:17:00.003+02:00

Yesterday I had the pleasure of trying OpenBSC on a real network and the result was desaster, but honestly what else to expect when trying it the first time. It is not that OpenBSC was crashing, leaking memory, or not recovering from failure it is just the load of the network was differrent than what I assumed and that leads to problems.

What happens is one is seeing a lot of location updating requests, which will load the SDCCH but that is really fine and we have seen such things at the Chaos Congress, what is different is the result of location updating requests, the network will flood us with paging requests... Right now we are sending up to 20 paging requests every two seconds... The first thing to notice is that this too much for the nanoBTS, it is sending us a nice CCCH/ACCH/BCCH overload warning which we do not handle (we should start two timers and throttle the amount of messages we send) the other part is... if we are out of SDCCHs and ask 20 more phones a second to get one... We have created the RACH DoS that Dieter Spaar has done with a Mobile Station.

The Random Access Request contains the channel type and one to IIRC four bits of random numbers, so even if we have a free channel... it can happen that two phones believe that we have assigned a channel to it... and then we see RF Failures, which in turn will trigger the phone to try again (or we page it again)... and then nothing will work....

The other observation is that if our cell is really busy we should start to assign TCHs to fullfill location updating requests....

So the changes I need to make is to change the paging to not page as much as we physically can stuff into the PACCH but as to how much of the responses we can handle (pretty obvious?) and the other is to allocate a "bigger" channel in case we have no smaller channel... E.g. use number of free channels divided by X for paging requests...

Traveling woes..

2010-04-13T07:21:00.003+02:00

I'm just back to the flat. On this trip (in order) my luggage didn't arrive witht the same airplane, my cellphone broke during the flight (the headset speaker is dead), my laptop backlight broke and my luggage didn't arrive at the final destination... So after doing a backup of the data and removing the disk I will bring my laptop to an Apple dealer, and for the weekend I try to get my phone replaced... any my luggage should be home later today as well..

Besides that the traveling was fine...

Hacking on OpenBSC

2010-04-11T21:57:00.010+02:00

I was invited to visit the On-Waves (they have a shiny new website) office in Paris this week and I was quite busy hacking away on the OpenBSC codebase. On-Waves allows me to play a bit with their MSC and learn more about GSM and in exchange OpenBSC gains a more and more complete and stable GSM A-Interface.

When developing code for OpenBSC we are mostly sitting very close to the BTS, only have one active subscriber, test one thing, restart, test another thing, restart but with any piece of software I'm writing, I want OpenBSC to be rock solid, run unattended for years, have no memory leaks, deal with the nanoBTS going away and coming back, the MSC going away and coming, all this at any point in time. So far events like Hacking at Random and the Congress are the ideal testing ground as many different handsets, subscribers, etc are the ideal playground.

My testing was limited to a small set of handsets connected via USB and executing AT commands for call handling and sending SMS. I'm addressing subscribers on the same cell. That means whenever I do a call I have mobile originated and mobile terminated testing covered and this is done by funny chat scripts that work most of the time. The next thing is to simulate failure, for some stuff where a specific layer3 message would be send, we have to wait for a more complete OsmocomBB, so what I can easily do is to cut off TCP connections. I have done this with another piece of weird shell magic. I use the output of $RANDOM and treat it as seconds and then use a kill -SUGUSR2 `pidof bsc_msc_ip` to close the MSC connection at a random time. And then I let it running and wait for failures.

I have fixed a bug/issue in the way we do release a channel. There are multiple things involved. First of all is instructing the BTS that a given channel on a timeslot is open or closing it (RF Channel Release of RSL), the other part is that on the channel one can have logical applications running (SAPI), this can be call control (SAPI=0) and SMS (SAPI=3). When opening a connection to a Mobile Station (MS) the SAPI=0 is always established, when attempting to deliver a SMS we need to open SAPI=3 first. Now our issue with bringing this down was that whenever we got a SAPI release confirm (we asked for the release and it was released) or release indication (the MS closed it) and we used to respond with a RF Channel Release. Now when trying to bringdown a connection were we delivered a SMS we would issue a RF Channel Release twice and the nanoBTS ACKed it twice! To make matter worse, whenever we get a RF Channel Release ACK we mark it as free. We had this small window when we got the first RF Channel Release ACK, allocated the channel again, and then get the second RF Channel Release ACK. I have fixed this issue in multiple ways. The first is to use the T3111 timer to wait until we issue the RF Channel Release, the second is to handle (RF) failures by "blocking" the lchan for a short second to receive multiple errors and release acks and the last bit is to properly bring down the channel. When we have SAPI!=0 we bring that down first, then we send SACH deactivate, followed by SAPI=0 release and then finally we send the RF Channel Release. This makes things more reliable on our side but we need to fix some more things. There is a FIXME inside the gsm_04_08_utils.c that mentions the start of a T3109 timer. In any case when sending a SAPI release the BTS will answer with success or a timeout and we handle both.

Today I addressed losing the RSL or OML connection to the nanoBTS and making sure we are reconnecting and not leaking any memory. This took me most of the day to get stable and I have found a bug or such inside the osmocore/select.c when releasing a bsc_fd that is the last one of the list. The difficulty here is making sure we do not leak memory, close all file descriptors, close all channels that take place on the RSL connection and make sure that when the BTS is up again we can use the channels that were allocated during the failure. To help with testing I added two commands to our vty interface to drop the OML or the RSL connection on a given BTS. The other part that was helpful is to use Linux's Netfilter and drop packets on a TCP connection and to wait for a failure. Now I can simulate most of the network failures easily and could build some trust.

And my final wishlist item would be to have like 16 GTA02 boards, use FS0 on each and run a simple script to dial, send SMS, pickup phonecalls this would allow me to heavily test the networking in an automated way. On top of that would be to have a OsmocoreBB enabled Calypso or C123 and then I could even send messages that are normally not send at all. And thanks to FreeSoftware development I'm sure we are going to reach that goal.

GSM Fail...

2010-04-08T15:46:00.002+02:00

Hi,

okay... I have a simple task... dial some numbers, drop some SMS. Right now I'm using a whacky mix of shell and chat script to do it but it is not as nice and reliable as it should be. So I was trying ofono and fso to use my serial device to do some calls for me... Or at least I tried to. FSO was relatively easy to install on Debian, having a nice readme, picking a GSM driver... well and then... how do I specify the config value... looking at the source, finding stuff... taking an old example from Charlie... nothing... hmm.. Okay, on to ofono...

So ofono is written to make the life of designers more easy, it must be so easy that there is no documentation required... after finding some examples in the test directory of the code... I send one MSG to enable the Power of my modem and hope the VoiceCallManager comes up... instead I do see a segfault in the ofonod... *sigh*

I think I will write a custom application to send and parse some simple AT commands for now... how frustrating.

Using oeaudit.py

2010-03-23T10:34:00.003+01:00

In the last days I have cleaned up my OE Audit tool and it should be usable by everyone now. The tool requires two inputs, one is the list of packages to be build by OE for the given configuration (distro, machine) and the other is the FreeBSD auditfile. This FreeBSD auditfile can be automatically downloaded.

Without much more overhad, here we go.

$ bitbake -s > available
$ export PYTHONPATH=/place/bitbake/lib
$ /OE/contrib/oeaudit/oe_audit.py -f
This will fetch the auditfile from the FreeBSD project for you
$ /OE/contrib/oeaudit/oe_audit.py -a auditfile -p available
Now you will see a list of vulnerabilities in the packages OE is going to use.

Dealing with security issues in the context of OpenEmbedded

2010-03-21T03:36:00.002+01:00

One thing that has bothered me while being at Openmoko is the lack of Security Response by the OpenEmbedded Crew. In one way a security issue is just like any other bug and distros don't upgrade each package for each bug fixed upstream but it is getting worse when the security issues exists in the default installation, in a daemon listening to network traffic and such with ready to get exploits on the network.

I think it is really unethically to go around and claim how great OpenEmbedded is and then companies like Openmoko, Palm, etc. ship vulnerable software to their users and it is easy to pass the black pit to companies actually using OpenEmbedded, let me say it is too easy.

There are various things one can do to address these problems. One option is to downgrade and use the classic Buildroot as their maintainers seem to address vulnerabilities in time. I use the word downgrade as these systems provide less functionality, flexibility than OpenEmbedded, e.g. they lack the creating of SDKs, chosing the libc (glibc, uclibc, eglibc) but then again they do their homework and provide people with security updates in time, the other option is to go to a distribution like Debian or Fedora with a proven track record of handling security issues.

But I'm going to talk about the third option that includes improving OpenEmbedded. I had the idea while being at Openmoko but the guy who was assigned to do this was laid off shortly afterwards so it never happened. In general for every package we ship in OpenEmbedded there is an established distribution (e.g. FreeBSD, Debian, Fedora) that is shipping it as well. Or in the seldom cases where OpenEmbedded is the first adopter, the software is kept current and there is not much security research anyway. This means that to provide security upgrades to our users we only need to monitor the big and established guys and that sounds like something that can be partially scripted.

I'm using FreeBSD on my servers and in the FreeBSD world there is an application called portaudit which is looking at your built/installed packages and is comparing the name, package version and patch release to a list of known security issues in the ports tree and then asks you to upgrade, Gentoo has a similiar XML file for each security incident, Debian has a security feed as well.

A long story short, on a flight to iceland I was hacking a python script called oe_audit.py that is using the FreeBSD auditfile and the output of "bitbake -s" (the list of providers and their versions) and then starts comparing these lists. Right now the script is inside the OE tree, it is still a gross hack but I will improve it to be a proper python script. In its first version it has found issues with plenty of packages in OpenEmbedded and thanks to the help of some we are down to only a couple of issues in our tree. In general addressing security issues is not that hard, follow a couple of mailinglists, look at websites, when a CVE is published search for the patch, apply it to our version, be done. Specially given the amount of OE developers we could nominate a security sherif each week that has to do the upgrades... It is not that we see more than three upgrade a week anyway... So this week it would have been Pango, Php, Pulseaudio...

Using setitimer of your favorite posix kernel

2010-03-13T07:05:00.002+01:00

As this was requested in a comment of a previous post and knowing your kernel helps to write better performing systems here is a small information of how to use the interval timers provided by your posix kernel.

What is the interval itime?r

The interval timer is managed and provided by your kernel. Everytime the interval of the timer expires the kernel wil send a signal to your application. The kernel is providing three different interval timers for every application. The different timers are for measuring the real time passed on the system, the time your application is actually executed and finally the profiling timer which tmes the time when your application is executed and when the system is executing on behalf of your application. More information can be found in the manpage with the name setitimer.

Why is it useful?

In the QtWebKit Performance Measurement Utilities we are using the interval itimer as the timing implementation for our Benchmark Macros. To be more precise we are using the ITIMER_PROF to measure the time we spend executing in the system and in the application, we are using the smallest possible precision of this timer with one microsecond. The big benefit os using this instead of elapsed real time, e.g. with QTime::elapsed, is that we are not depending so much on system scheduling. This can be really nice as even with a lightly crouded system we can generate stable times, the only thing influecing the timing is the MHZ of the CPU.

How is it implemented?

It is a kernel timer, this means that it is implemented in your kernel. In case of Linux you should be able to find a file called kernel/itimer.c, it defines the syscall setitimer at the bottom of the file. In our case the SIGPROF seems to be generated in kernel/posix-cpu-timers.c in the check_cpu_itimer routine. Of course the timer needs to be accounted by things like kernel/sched.c when scheduling tasks to run...

How to make use of it?

We want to use ITIMER_PROF, according to the manpage this will generate the SIGPROF. This means we need to have a signal handler for that, then we need to have a way to start the timer. So let us start with the SIGPROF handling.


Elapsed time handling
static unsigned int sig_prof = 0;
static void sig_profiling()
{
    ++sig_prof; 
}


The signal handler
    struct sigaction sa; 
    sa.sa_handler = sig_profiling;
    sigemptyset(&sa.sa_mask);
    sa.sa_flags = SA_RESTART;
    if (sigaction(SIGPROF, &sa, 0) != 0) {
        fprintf(stderr, "Failed to register signal handler.\n");
        exit(-1);
    }


Start the timer
tatic void startTimer() 
{
    sig_prof = 0;
    struct itimerval tim;
    tim.it_interval.tv_sec = 0;
    tim.it_interval.tv_usec = 1;
    tim.it_value.tv_sec = 0;
    tim.it_value.tv_usec = 1;
    setitimer(ITIMER_PROF, &tim, 0); 
}

Discussion of the implementation

What is missing? We are using the sigaction API... we should make use of the siginfo_t passed inside the signal handler.

What if we need a higher precision or need to handle overflows?
There is the POSIX.1b timer API which provides timers in the nanosec region and also providers information about overflows (e.g. when the signal could not be delivered in timer). More information can be found when looking at the timer_create functions.

When is the interval timer not useufl?

Imagine you want to measure time it takes to complete a download and someone wrote code like this:


QTimer::singleShot(this, SLOT(finishDownload())), 300000);

In this case to finish the download a lot of real time will pass and the app might be considered very slow, but it in terms of the itimer only little will be executed as the time we just sleep is not accounted on us. This means the itimer can be the wrong thing to use when you want to measure real time, e.g. latency or time to complete network operations.

Things I didn't know about RTP and AMR

2010-02-28T12:43:00.003+01:00

Oh my god... Looking at an AMR payload wrapped in RTP, wrapped in UDP, wrapped in IP... one will recognize that there is a +70% overhead on payload vs. header... The other thing is... the AMR payload can have a CRC or not but it is not indicated in the RTP header and must be signalized out of band... it took someone a while to figure it out. Hurray on having clever colleagues.

Dealing with Performance Improvements

2010-02-28T12:03:00.004+01:00

I hope this post is educational and help the ones among us doing performance optimisations without any kind of measurement. If you do these things without a benchmark you are either a genuis or very likely your application is going to run slower. I'm not going to talk about performance analysis right now but tools like OProfile, callgrind, sysprof and speedprof are very handy utilities. The reason I'm writing this up is that I saw a performance regression in one of my testcase reductions and this is something which I don't appreciate and in general I see a lot of claims about performance tuning but little bit in regard to measurements and this part is very worrying.

For QtWebKit we have the performance repository with utilities, high level tests and something I labeled reductions. In detail we do have the following things:

Macros for benchmarking. I started with the QBENCHMARK macros but they didn't really provide what I needed and changing them turned out to be a task I didn't have time for. I create WEB_BENCHMARK macros that work the same as the QBENCHMARK macros. One of the benefits is to provide better statistics, it prints the mean, std deviation and these things at the end of the run. And it has a different metric for measuring time. I'm using the setitimer(2) syscall to measure the CPU time we are executing in userspace and kernelspace on behalf of the application. This metric is a robust way to avoid issues like CPU scheduling and such. It would be the wrong metric to measure latency and such though, as we are not executing anything when waiting.

Pick the area you want to optimize. With the QtWebKit performance repository we do have a set of reductions. These reductions consist of real code, a test pattern and test data. The real code is coming from WebCore and is driving Qt, the test pattern comes from loading real webpages. It is created by adding printf and such to the code and the test data is the data that was used when creating the test pattern. We do have these reductions for all image decoding operations we are doing on the webpages, for our font usage, for QTextLayout usage.
The really awesome bit about these reductions is that they generate stable timings, are/should be fully deterministic. This allows to really measure any change I'm doing to let's say QImageReader and the decoders.

Using the setitimer(2) syscall we will have pretty accurate CPU usage of the benchmark, using the /lib/libmemusage.so of GLIBC we should have an accurate graph of the memory usage of the application. It is simple to create a benchmark, it is simple to run the benchmark, it is simple to run the benchmark with memory profiling. By looking both at CPU and Memory usage it will become pretty clear if and where you have tradeoffs between memory and CPU.

And I think that is the key of a benchmark. It must be simple so people can understand what is going on and it must be simple to execute so everyone can do their own measurements and verify your claims. And specially having a benchmark and having people verify your measurements is keeping you honest.

Finally the commit message should state that you have measured the change, it should show the result of the measurement and it should contain some interpretation. e.g. you are optimizing for memory usage and then a small CPU usage hit is acceptable...

Explorations in the field of GSM

2010-02-26T14:04:00.004+01:00

Something like 14 months ago I had no idea about GSM protocols, 12 months ago I was implementing paging for OpenBSC, beginning from last summer I explored SS7 and SCCP, wrote a simple SCCP stack for On-Waves. Started to implement the GSM A Interface for OpenBSC, the last week I saw myself learning more about MTP Level3. With the Osmocom I start to explore GSM Layer 1 (TDMA, bursts, syncing), GSM Layer 2 (LAPDm) and on GSM Layer3 we mostly see the counterpart of OpenBSC.

I feel like I am back to school (in the positive way) and I have learned a lot in the recent year and looking forward I will learn more about protocols used at the MSC side and such. I'm very excited about what the future is going to be like. Will we have a complete GSM Network (BTS, BSC, MSC, MS, SMSC, GPRS gateway(s)) with GPL software by the end of the year?

Conclusions of my QtWebKit performance work

2010-02-11T11:07:00.002+01:00

My work on QtWebKit performance came to a surprising end late last month. It might be interesting for others how QtWebKit compares to the various other WebKit ports, where we have some strong points and where we have some homework left todo and where to pickup from where I had to leave it.

Memory consumption

Before I started our ImageDecoderQt was decoding every image as soon as the data was complete. The biggest problem with that is that the ImageSource we are embedded into does not tell the WebCore::Cache about the size of the images we already have decoded.

In this case there was no need to decode the whole image as soon as the date comes in but wait for the ImageSource to request the image size and the image data. This makes a noticable difference on memory benchmarks and allows us to have the WebCore::Cache control the lifetime of decoded image data.

We still have one case where we have more image data allocated than the WebCore::Cache thinks. This is the case for GIF images as we are decoding every frame to figure out how many images we have there.

To fix that we should patch the ImageSource to ask the ImageDecoder for "extra" allocated data, and we should fix/verify the GIF Image Reader so we can jump to a given GIF frame and decode it. This means we should remember where certain frames begin...

Performance

Networking

Markus Götz and Peter Hartmann are busy working on the QNetworkAccessManager stack. Their work includes improving the parsing speed of HTTP headers, making sure to start HTTP connections after the first iteration of the mainloop instead of the third.

In one of my tests wget is still twice as fast as the Qt stack to download the same set of files. And wget is using one connection at a time, no pipelining... and Qt is attempting to have up to 6 connections in parallel. This means there is still some work to do in reducing latency and improving scheduling of requests. I'm pretty confident that Markus and Peter will work on this!

Images

The biggest limitation of the Qt Image decoders is that in general progressive loading is not possible and unless I have messed up my reduction the Qt Image decoders are faster than the ones we have in WebCore.

With some of my reductions I can make some stuff twice as fast for the pattern QtWebKit is having on QImageReader. Currently when asking the QImageReader for the size, the GIF decoder will decode the full frame (size + image data). For the GIF decoder we start the JPEG decompression separately for getting the size, the image and the image format.

A proof of concept patch for the JPEGReader to reuse the decompression handler showed that I can cut the runtime of the image_cycling reduction by 50%.

Misc

One misc. performance goal is to remove temporary allocations. E.g. remove QString::detach() calls from the paint path, to not copy data when moving from QString to WebCore::String, QByteArray to WebCore::String. Some of these include not using WebCore::String::utf8(), but have a zero cost conversion of WebCore::String to QString and use Qt's utf8()...

Text

But the biggest problem of QtWebKit performance is text and I statzed to work on this. For Qt we always have to go through the complex text path of WebCore which means we will end at QTextLayout, which will ask harfbuzz to shape the text.

There are two things to consider here. For QtWebKit we are using Lars's QTextBoundaryFinder instead of ICU. I'm not sure if we have ever compared how ICU and QTextBoundaryFinder split text. We might do more work than is necessary, at least it would be good to know. Specially for Japanese and Korean we might split words too early creating more work for our complex text layout path.

The second part is to look at our QTextLayout usage pattern and start to optimize for it... the quick solutions of asking QFont to not do kerning, and not to do font merging (to not use the QFontEngingeMulti) didn't really make a noticable difference... To get an idea of the size of the problem, on loading pages like the Wikipedia Article of the Maxwell Equations we are spending so much time in WebCore::Font::floatWidthForComplexText that other ports like WebKit/GTK+ takes to load the entire page. This also seems to be the case for sites like google news.

And this is exactly where I would have loved to continue to work on it, but that is now pushed back to my spare time where it needs to compete with the other hobby projects.

On getting Free Support

2010-01-31T04:17:00.004+01:00

Most of the things I know about Software and Hardware I have from reading books, looking at sourcecode but most importantly people willing to answer my questions on IRC and giving me a direction I could look for answers.

Now it seems to be my part to give back and help others to gain knowledge, but some things have changed. Free Software has made it to the mainstream, so besides hackers that want to understand things, we do have paid programmers that don't want to understand but still need to make it work.

So the other day I was finding me in a IRC query. His target board was a Freescale i.MX27 ADS and his mission was to make it boot from NAND. He was using the OpenEmbedded "mx27ads" machine and had successfully build a kernel and rootfs. Now the problem is that Freescale is not particulary liked in the Free and Open Source Software Community and that Freescale prefers to be on an isolated island. After about 8 hours of spending my time on this, i decided to get back to paid work and carry on.

If you are searching for support on mailinglists, irc channels and irc queries be prepared to think, giving Free Support means that you will be helped to understand the problem and have to pick a solution yourself. If you don't like that, don't want to think, don't have the time to think, you should consider getting someone from the irc channel as consultant.

So here is a list of things that work when paying a consultant but not when you are paid to do your job and you need someone else to do it for you:

Pasting your log somewhere and then ask what is wrong with it. In case of compile failures with GNU make you have to search for '***' in the error log, in case of an early failure of a bitbake run it tells you what software to install, in case of configure failures read the config.log, in case of other failures search the log for 'Error', 'rror', 'Failure', 'ailure'. And most importantly think before you post it.

At OpenEmbedded we check for Software being installed on the system. This includes checking for GIT, CVS, SVN and other projects needed to bootstrap. If you don't have it installed OpenEmbedded will tell you "Error... you don't have installed:" and the output will be finished by "You will have to install....". If you paste such an error and ask what the problem is you are really embarassing yourself, your parents, your teachers and your country. The Error message tells you which binaries it searched for and couldn't find, and it points you to common package names for them. I don't think it can be any easier for a Software Engineer.

Knowing your hardware. Remember, you have the hardware in front of you, you have it connected somewhere, you have something compiled on your system. So if you are asked about the system specification you will have to point to a PDF, website that names the SoC, the used flash chip (NAND, NOR, erase size, name of it), the SDRAM used and whatever else is used on the system. If you paste a FAQ of something like Buildroot you have clearly failed. Know the stuff that is on your desk, if you don't nobody can help you.

Pasting unrelated information. When you are asked to paste the output of something. Do not just write that oneline by hand. In many cases we humans apply interpretation to things. When you are asking for help it is an indication that you are not able to interpret the result in a way that lead to a result. Paste the full log, it is saving everyone a lot of time.

Listen/Read to what people tell you. If you have a kernel without NAND support, but want to boot from NAND. You will have to enable NAND and the MXC NAND driver in your kernel. You will have to compile the kernel again, and you will need to boot that kernel. Now we are all humans and have done mistakes before.

One of the most common mistakes is to not change the config, or not to boot the kernel you have built.

For the config, check .config after you have built if it is the one you expected it to be, in case of OpenEmbedded copy the .config back to the defconfig (recipes/linux/${PN}-${PV}/${MACHINE}/defconfig). You should copy it back to have a backup in the case you are rebuilding the kernel or such.

For the second thing it is rather easy. The kernel does contain the TIME it was built and it contains the number of times you have built it from the same directory. So if you rebuild the kernel the TIME and the NUMBER will go up. The kernel does print this information at bootup. If you are asked to check that, don't paste the log, but check it for yourself. Last but not least, if the NUMBER did not go up or the TIME did not change you have either not rebuilt it, or not from the same directory... And if you didn't rebuild from the same directory then you are likely to not use the right config...
Don't do crazy things while people spend their time to help you. Do not remove your build directories and just start over. No, even after your next rebuild the kernel will lack NAND support. That is because it is not enabled in the defconfig for your machine, and the build process is deterministic... You have someone on the other side that decided he wants to help you, if you are not focused, why should he?

And for all of you, that have read until the end. If you decide to seek help in an open forum. First do your homework by having some idea about the problem, have information ready (hardware spec, build logs, whatever), be prepared to think, formulate a hypothesis and try it.

With Free Software you are in the fortunate situation that you can talk to the guys who build the stuff you are using, all you need to do is to be focused on receiving help and be prepared to think.

Reverse engineering with okteta

2010-01-01T07:28:00.002+01:00

In the last week I was hacking on OpenBSC to make GSM 12.21 Software Load usable for the ip.access nanoBTS. The difficulta was not within GSM 12.21 as Harald had it implemented for the Siemens BS11 BTS. The difficulty was that some messages need to contain paramaters and these come directly from the firmware file which ultimately means that one needs to understand the firmware file format to extract these. okteta came to rescue me and it was extremely good at doing this.

Okteta has not only the hex view one expects but also some useful utilities. Selecting a couple of bytes and the "Decoding Table" can tell you the different values in different endinanesses. So whenever I thought this is a file length, I would look into the "Decoding table", select bytes and see how many I selected and if it could make sense, it can calculate various checksums over a selection.

Thanks a lot for Okteta, it safed my day!

Looking back to 2009

2009-12-24T14:02:00.003+01:00

The second year as part time freelancer has passed.

Looking back the most significant things are:

Signing the contribution agreement for gdb and glibc with the Free Software Foundation and trying to contribute to both projects. So picking future work will always have to be compatible with this.

Hacked on OpenBSC. At first just simple stuff like a telnet interface, paging and later doing paid work for On Waves to add SCCP over IP, GSM 08.08 and other things for "toy" integration of OpenBSC into a real network.

Mid this year I asked Nokia if they have work for me in Asia, later I started focusing on QtWebKit performance. Allowing me to improve QtWebKit and Qt (which will benefit a lot more users), but also to look into various tools like OProfile, memprof, memusagestat and just know netfilter queue's... more on this later.

I have done my usual things on OpenEmbedded, working on landing patches through the patchwork queue, finally redoing the Bitbake parser and working on the Qt recipes.

I didn't manage to make a Linux Kernel contribution. I wanted to write a i2c driver for a fm radio chip but I fried my hardware with a broken power supply, my MIPS patches are not yet done. So if you know of any Kernel work where stuff can be released/upstreamed please let me know!

Kafka's Briefe an den Vater

2009-12-24T13:53:00.002+01:00

I used to read books of Franz Kafka a lot, I loved his struggle with his father as I could relate to it. I loved the idea if writing letters in form of liberating myself. But in recent years as I got older and parents became less important.

So the last thing I will ever say about my "father" (and in case he reads it, as even old people can use computers...). I have no memory left of you, I have no picture left from my youth that displays you, I have no gift from you left in my posessings, I will get rid of your last name, You are a lonely old man. Good bye, I don't care for you.

(Qt)WebKit Sprint in Wiesbaden

2009-12-23T10:20:00.002+01:00

The sprint is over for some time. You can see summaries of the different sessions and some slides in the wiki. Besides talking about QtWebKit and how to improve it (API, features, speed, make people aware that they can contribute, influence the release schedule, policies.. *hint*) the thing that has impressed me the most is unrelated to coding.

We all hear when someone from our Community is leaving the Qt department, and we always wonder how life will continue, who will fill the gap. In the last year a couple of new people got hired at Oslo and I'm really impressed how they find such capable people that are technically skilled and willing to move to Oslo! kudos!

hawkboard.org

2009-12-07T20:59:00.003+01:00

Apparently TI wants to continue its' success with the beagleboard and apply it to the OMAP-L138 product in terms of hawkboard according to a presentation at FOSS.IN the coolest thing about the hawkboard and the OMAP-L138 is the floating point DSP.

Talking about performance measurements at foss.in

2009-12-04T08:50:00.002+01:00

It is the second time I'm at foss.in and this time I was talking about the current work I'm doing on QtWebKit. Nokia is kind enough to give me enough time to explore the performance of QtWebKit (mostly on Qt Embedded Linux and ARM) and do fixes across the stack in WebKit, Qt or whereever we think it will be necessary.

Performance for me comes in memory footprint and runtime speed (how long does it take?) and for this I have experimented using OProfile, Memprof/Memusage, QBENCHMARK but also wrote some WebKit specific tools. E.g. a tool that allows me to mirror webpages to turn them into a benchmark (which still has quite some problems), a simple http server to serve the content, some test case reductions in order to look into specific areas like networking, image decoding, painting, fonts.

The slides and links can be found here and they link back to the WebKIt wiki where you can find an introduction to the (Qt)WebKit specific tools, a set of bugs and pending patches, and a set of issues that are known but not yet handled.

The main message of the talk is to not do optimisation by myth, but to use a stable environment and one of the existing tools to see what is going. It is really easy.

Attending foss.in

2009-12-01T20:26:00.002+01:00

Thanks to generous sponsoring I managed to make it to Bangalore for FOSS.IN and Girish kindly agreed to provide accomodation. It is really great to be in India again, to see the streets, the local market, catch up with friends from India and Europe.

Girish is currently struggling with plugin code in QtWebKit for Mac, Win and X11 in both window and windowless mode, in the good ole wold and in QGraphicsView... I'm analyzing the loading behavior of a particular website and now I need to find out why we take a whole second to do layout and sometimes just actively do nothing... It is really awesome to have some clever company here in India.

GSM AMR (Speech Version 3) with OpenBSC

2009-11-19T11:26:00.002+01:00

This week I had to make parts of OpenBSC work with TCH/H and use AMR. This work is needed for On Waves and when I say parts I mean the strict BSC subset of OpenBSC (in contrast to make the MSC code we have work as well).

The first part is to make TCH/H work and that was easy as LaF0rge did almost everything to make it work. You have to change the OpenBSC configuration to use TCH/H instead of TCH/F for the given timeslots. The next thing was to make channel assignment work. The Mobile Station (MS) comes on the Random Access Channel (RACH) and is asking for a channel and gives a random number (so it can identify the response). Now depending on a global indicator (NECI) the MS will ask for different channels.

So the next step was to add a NECI configuration to our VTY configuration code and then change the code that decodes the channel request to know about the NECI and pick the right channel. On top of that a small hack to assign a TCH/H in case of a MS requesting "any" channel as part of paging.

Now that TCH/H should work one has to focus on the speech. GSM 08.08 and GSM 04.08 have different enums for speech. GSM 08.08 is differenting speech version 1,2,3 for full and half rate totalling in six different values, for GSM 04.08 there is a TCH mode that includes speech version 1,2,3, various data modes and signalling (but no differentation full/half rate channel). After getting this right and selecting speech version 3 it still didn't work. It turned out that one has to fillout the optional Multirate Configuration when one is using speech version 3. This multi rate configuration needs to be present in the GSM 04.08 RR Assignment Command, Modify Channel but also in the RSL messages for Modify Request and Channel Assignment.

After this AMR on a TCH/H should work (when the BTS is supporting it too). The next step for someone else is to make the MSC code in OpenBSC work with TCH/H and other audio codecs. This would require to stop to ask for a TCH/F, change the channel requested decoding again..

Visiting On Waves in Iceland

2009-11-18T10:25:00.002+01:00

Currently I'm sitting in the nice offices of On Waves and when not trying to convince the embassy of India to give me a visa I'm working on OpenBSC. For this week I try to make call handling with the MSC rock solid.
So far I have fixed some bugs, added features to OpenBSC, enabled A5/1 encryption, started using TCH/H, started using AMR, fixed bringup on nanoBTS coldstart and now I'm working on the MGCP to verify that I can hear audio for my calls.

The benefit of using GCC NEON intrinsics

2009-11-07T04:26:00.002+01:00

I'm currently writing NEON code for the Qt PorterDuff SourceOver implementation. At the beginning one has to make the decision to use inline assembly, a seperate .S file or the ARM NEON Intrinsics.

I have chosen to go with the ARM NEON Intrinsics embedded into C++ code for a couple of simple reasons. At first it is portable across gcc and RVCT doing a .S or inline assembly would not work for RVCT that is used by the Symbian people. The second reason is that I get type safety. The NEON registers can be seen as 8bit, 16bit, 32bit, 64bit signed/unsigned registers when doing low level assembly you might pick the wrong operation and it is hard to see, with using the intrinsics you get a compiler warning about your mistake. One downside is that with some easy things I can make my compiler abort with an internal compiler error... but this will change over time.

Next is the myth that GCC is crap and that the instrinsics are badly "scheduled". From my looking at the assembly code it is mostly arranged like I wanted it to be. On a simple operation GCC was putting a LDR in the code right inbetween neon load and stores operations, with a simple change in the code this LDR was gone and I should not see any of the described hazards.

Now my ARM NEON code is slower than the C code (that is using tricks) but that is entirely my fault and I have some things I can try to make it faster. And to be more specific the ARM NEON code is four frames faster than the old C code (that was not using tricks).

Collecting hints to increase performance in Qt (and apps)

2009-11-07T04:10:00.004+01:00

I'm working part time on improving the performance of QtWebKit (memory usage and raw speed) and I have created some tools to create an offline copy of a number of webpages (gmail, yaho mail, google, news sites...).

Using these sites I have created special purpose benchmark reductions. E.g. only do the image operations we do while loading, while loading an painting, load all network resources. One thing I have noticed is that with a couple of small things one can achieve a stable and noticable speedup. These include not calling QImage::scanLine from within a loop, avoid using QByteArray::toLower or not use QByteArray::append(char) from a loop without a QByteArray::reserve.

I have created a small guide to Qt Performance, I will keep it updated and would like to hear more small hints that can be used to improve things. If it makes sense I can migrate it to the techbase as well.

Painting on ARM

2009-11-03T09:26:00.002+01:00

I'm currently work on making QtWebKit faster on ARM (hopefully later MIPS hardware) and in my current sprint I'm focused on the painting speed. Thanks to Samuel Rødal my work is more easy than before. He added a new paintengine and graphicssystem that allows to trace the painting done with QPainter and then later replay that. Some of you might feel reminded of Carl Worth's post that mostly did the same for cairo.

How to make painting faster? The Setup

Record a paint trace of your favorite app with tst_cycler -graphicssystem trace, do the rendering and on exit the trace will be generated

Use qttracereplay to replay the trace on your hardware (I had some issues on my target hardware though)

Use OProfile to look where the time is spent and do something about it...

Change code go back to qttracereplay..

What did I do so far?
Most samples are recorded in the comp_func_SourceOver routine. With some searching in the MMX optimized routines and talking to the rasterman I'm doing the following things to improve things on the const_alpha=255 path. In the qttracereplay I go from about 17.4 fps to around 26 fps on my beagleboard with Qt Embedded Linux on the plain OMAP3 fb but I still need to do a more careful visual inspection of the result.

Handle alpha=0x00 on the source special by not doing anything

Handle alpha=0xff on the source special by simply copying it to the dest

Unroll the above block eight times interleaved with preloads...

I will have to clean all this up, merge it with the symbian optimized copies (which sometimes require armv6 or later)... I will probably look at BYTE_MUL now and see if I can make it faster without taking a armv6 or later instruction... or honestly first understand how the current BYTE_MUL is working...

DTrace for GNU/Linux -- SystemTap

2009-10-30T07:41:00.002+01:00

One idea is in my mind. How can libmemusage.so, libmemintercept of memprof, memprof be merged into one and provide more and better results. E.g. sometimes you want to trace and get the profile when the heap maxes, or you want to have a histogram of allocations. It is possible to write C code for that and integrate it into one. On the other hand with technologies like DTrace one can easily write the histogram generation, profiling, etc with a trace script.

So what can we do on Linux? The thing coming closest to it is Systemtap. You write a trace script and the script gets compiled into a kernel module that is then loaded and is doing the probing. In theory one is able to even trace userspace with it.

The only problem is that SystemTap is not ready yet. To do really useful stuff one has to patch the Kernel with the utrace patch, Ubuntu/Debian are not featuring recent enough elfutils. So it will probably take another year until memprof can be a simple gui around SystemTap and such and probably two years until most distros come with a ready to use SystemTap.

memprof 0.6.2 release

2009-10-26T14:42:00.002+01:00

Today I have released memprof 0.6.2. The most prominent change is merging a merge of raster's timegraph for memory allocations and fixing various stability bugs introduced post 0.6.0. The code is currently located on gitorious and the release tarball is here and the shortlog can be seen below:

Cristi Magherusan (2):
some other minor changes, mostly guint -> gsize's
fixed a typo, bug #51556 in the gnome bugzilla

Holger Hans Peter Freyther (10):
mi-perfctr.c: Remove the O_CREAT (from the openSUSE buildservice)
memprof.glade: Open and save the file
Provide a GtkFileChooseButton to select the executable.
merge rasterman's extra window
.gitignore: Ignore generated files
process_find_line: Clarify who is owning the returned pointer
detailwin.c: Fix possible crash when opening the maps file fails
process_locate_symbol: Make sure a valid string is always returned
add_leaf_to_tree: Avoid running into a crash
memprof release 0.6.2

Stefan Schmidt (2):
configure.in: Use AM_SILENT_RULES if available
stack-frame: Introduce macros for stack pointer regs and use them.

Tomasz Mon (2):
configure.in: Search for bfd.h provided by binutils development package
Integrate the detailwin into the main GtkNotebook

William Pitcock (1):
use elf_demangle() in more places

What is the size of a QList::Data, RenderObject?

2009-10-12T05:45:00.003+02:00

We tend to write classes without really caring about what the compile will do to create the binary file. When looking into performance and specially memory usage and you create certain objects thousands of times it becomes interesting of how much memory one is wasting for padding/no good reason.

The Linux kernel hackers wrote a tool called pahole that will analyze the DWARF2 symbols and then spit out friendly messages like the one below.


        struct Data {
                class QBasicAtomicInt     ref;                   /*     0     4 */
                int                        alloc;                /*     4     4 */
                int                        begin;                /*     8     4 */
                int                        end;                  /*    12     4 */
                uint                       sharable:1;           /*    16:31  4 */

                /* XXX 31 bits hole, try to pack */

                void *                     array[1];             /*    20     4 */

                /* size: 24, cachelines: 1, members: 6 */
                /* bit holes: 1, sum bit holes: 31 bits */
                /* last cacheline: 24 bytes */

In this case QList::Data could have used at least three bytes less memory and changing the definition of sharable and array would have removed a whole in the struct. Maybe that is something for Qt5 to keep in mind.

The research question. Can QtWebKit memory usage be reduced by shrinking some of the Qt structs without losing functionality?

Memory profiling on GNU systems

2009-10-02T05:44:00.002+02:00

This is a small guide on how to observe memory allocations of a process. When carrying out a change it is not only of interest if all test cases still pass, if the benchmarks are faster but it is also important to figure out if there was a change in storage (stack and RAM) requirement.

If you are using the GNU libc it is likely you have a /lib/libmemusage.so installed on your system. This library can be preloaded using LD_PRELOAD and will intercept calls to malloc,free,realloc and various other calls. In short it will trace memory allocations for you. The limitation of that tool is that it will not tell you how much memory the kernel actually mapped, anything about memory fragmentation, etc.

To use libmemusage all you have to do is to prepend MEMUSAGE_OUTPUT=mytrace and LD_PRELOAD=/lib/libmemusage.so to your application. This will instruct the library to write out a trace to the mytrace file.

This trace file can be converted to a graph using the memusagestat utility. It is not installed by most GNU distributions and can be either build from the glibc sources or from the QtWebKit performance measurement utilities. Using memusagestat -o output.png mytrace an image with memory allocations and stack usage like the one at the end of this post will be created. The redline is the heap usage, the green one is the stack usage of the application. The x-scale is the number of allocations.

NOTES: As of today the graph creation with the x-Axis being the time is broken as the generated trace file has some issues, I'm looking into the problem but it will take some more days.

Standing on the shoulders of giants - Fixing OpenOCD for the Beagleboard

2009-08-27T05:18:00.003+02:00

Ever since fixing a little bug in the breakpoint code for the s3c2442 I'm subscribed to the OpenOCD mailinglist. One mail catched my eye, the mail was from a former Openmoko colleague Matt Hsu. He is working with most of the Taiwan Openmoko Team at the newly founded 0xlab and he was trying OpenOCD on the OMAP3 beagleboard.

Yesterday I decided to go to their lab to hack with Matt on the Cortex-A8 support in OpenOCD. After downloading the datasheets, we started to poke the code and tried to get a picture of how things are supposed to work. We have fixed various candidates for failures, like not waiting for the previous instruction to be finished when executing, not waiting for data to be there before reading.

Our symptoms were that after typing "halt" a lot of "invalid mode" messages got printed. After some tracing we ended at the execution of one instruction. The Cortex-A8 supports a mode were an ARM instruction can be written to a register and this instruction will be executed when being in debug mode. Now for a debugger the instruction is copying registers and other data to a special channel that can be read from the debugger. After looking further every register dumped liked this had the value 0x0...After this we were sure to have found the place were things go wrong. All we needed to find out why this instruction is not properly executed.

After studying the datasheet some more, going through the bits of DSCR and DCCR, and looking at the value of DSCR we have, it was clear that we just need to enable execution of instructions from the ITR (Instruction Transfer Register).

After this debugging session I still have little knowledge about ADI and the related things and this is why Free Software is so great, I can spend one afternoon, and can make a big change, and I can do it because I build on support from previous developers!

People should have stayed with Typewriters... or why DHL sucks

2009-08-24T06:47:00.003+02:00

Todays epic fail of DHL. When shipping a package you will have to leave a phone number to DHL. My dear reader, don't worry, this phone number will never be used to call the receiver of the package in case of an error.

So today I was checking the website where the package is and I'm supposed to call the DHL customer service. And in all glory of the internet and HTML they are not able to embed a link to the customer service package, in case there is no useful link on the tracking page.

Using my geek power I made a google query for "MyCountry dhl customer service". After loading of a java applet, and then selecting my country again, I'm on a page to show me the phone number of my local customer service. Oh my dear reader, if you think the odysee stops here you are wrong... The number on this page triggers a voice message that I can call this other number, this other number gives me another number, and this others number number gives me a number to a private person...

DHL... how hard can it be to have a working and DHL Customer Service phone number...

Embedded syntax highlighted sourcecode into OpenOffice.org Write

2009-08-20T04:22:00.002+02:00

The other day I was in need to embed some sourcecode into an OpenOffice.org Document. Normally I'm using LaTex and then can use the lgrind tool to prepare my sourcecode and use \lgrindfile{file} to embed it. Well but I'm somehow required to use OpenOffice.org Writer to generate a MS Office .doc file.

After some googling the GNU Source-highlight project came to rescue. I can syntax highlight my source code and patches, then I can use a HTML renderer to render these pages, select and copy the text and finally paste it into OpenOffice.org writer.

It works great!

WebKit on non mainstream platforms

2009-08-17T16:48:00.003+02:00

We have seen some interest in MIPS on the mailinglist and little in the bugtracker but no contribution from these folks. So the other week I decided to attempt to run some of our regression tests on a MIPS platform. The only problem is that I don't have any MIPS hardware... can you change that?

I had this idea of implementing enough of MacroAssembler to run/compile yarr with the RegexJIT.cpp. In the absence of MIPS hardware I used qemu-mipsel useremulation, enabled yarr/RegexInterpreter.cpp and ran into an infinite loop. I have fixed the bug and landed the patch upstream and now I'm able to complete run-javascriptcore-tests using yarr. The next step will be to work on the MacroAssembler and learn about MIPS assembly.... maybe I progress a bit more next weekend.

Searching for hardware donations

2009-08-14T11:57:00.002+02:00

In the last couple of weeks I have increased my involvement with OpenEmbedded again. I have worked on ARM/x86, keeping uclibc building, getting more things build with uclibc, worked on Qt, started merging patches, specially the MIPS side of things have catched my eye. So everything is fine... if there wouldn't be a little twist.

I don't have any ARM or MIPSEL targets at home. For my work on Openembedded and WebKit I'm searching for one MIPS and one ARM board, with network connectivitiy and working oprofile support. Anyone has a board spare and is able to ship it to Taiwan? Having a board would actually allow me to run the code I write for ARM and MIPS.

thanks

OpenEmbedded gains ability to create Qt Embedded Linux SDK

2009-08-12T08:13:00.002+02:00

Generating a Qt Embedded Linux SDK is as easy as typing bitbake meta-toolchain-qte. The generated SDK will contains the cross compiler, depedencies needed to recompile Qt Embedded Linux and Qt Emebedded Linux itself.

Building Qt applications for the software is as easy as sourcing a script and invoking qmake2. More documentation can be found in a new section of the OpenEmbedded User manaul.

Using SDKs produced by OpenEmbedded can greatly improve the productivity of development teams as they can start developing their applications right away and in contrast to classic SDKs if a new dependency is needed it can be easily installed using the opkg-target utility.

Fun on the tech-bsd mailinglist

2009-08-11T16:11:00.002+02:00

For various reasons I'm subscribed to the tech bsd mailinglist and this mail made me smile. Let me quote some parts of it.

"I don't know who you are, but you do nothing. What do you do?
I don't see your name on any the source code.
The source code does define the behaviour. Exactly. Perhaps the
source code is wrong, but it *EXACTLY DEFINES THE BEHAVIOUR*.
So you shut up, loser. Just go away."
Theo de Raadt

Tale of a day/morning

2009-07-29T12:55:00.002+02:00

My debug build of libQtWebKit finished and I thought the horror of linking a huge library is over and that I could start to debug. Turned out that gdb was segfaulting when launching my application. Eeeek., google... well googling for gdb and crash is not really finding gdb things. Okay so I was using cvs to get the latest version of gdb, compiling... *crash*. It seemed to crash in demangling, so just in case I was updating binutils and rebuilding gdb. same crash. Hmmm, let us try a debug build... hmmm. Okay one of these 300 char symbols, using the libiberty/testsuite it worked fine... hmmm. using valgrind... using it again and reading the output... oh stack overflow... Turns out alloca is a nice api... there is no error checking and no way to get a breakpoint set... even with -fno-builtin...I was kicking out alloca on my stracktrace from the crash and made it go away, filed a bug and attached a patch, let us see how long this takes to end up in a version. Fixed my QtWebKit within 30 seconds after seeing the stack trace...

I spent the other half of the day/morning with copying the Q.713 data structure into a headerfile and sketching out my header file for a SCCP lite GSM A-Interface implementation. I hope I can publish this soon. Before starting to implement this I will go back to do some WebKit performance analysis...

And with all the GNU bashing going on, I wonder if GNOME 3.0 will be called NOME 3.0... This reminds me of Goethe and the "Die Geister, die ich ruf werd ich nicht mehr los".