Tuesday, June 17, 2014

Playing with QV4

QtDeclarative is where the fun is. Starting with Qt5.2 a JavaScript Engine written by Digia is used. Compared to JavaScriptCore and V8 this engine is very basic but tightly integrated with the QML and Quick code. Motivated by attending the Qt Developer Summit and my work on GNU Smalltalk I started to look at the VM.

There are some environment variables that help to see what the JavaScript VM is doing and also how it is doing things. The below table gives a quick overview of the available flags and what they do. Specially the usage of QV4_SHOW_IR and QV4_SHOW_ASM helps to understand what is going on.

Name Function
QV4_NO_SSA Do not convert the IR::Function to SSA representation. This disables optimizations as well.
QV4_NO_OPT Do not run the optimizer. This disables dead-code-elimination, constant propagation, copy propagation
QV4_SHOW_IR Show the Intermediate Representation at the various stages of the compilation/optimization.
QV4_SHOW_ASM Show the disassembled code. This requires QtDeclrative to be compiled with CONFIG+=disassembler and without PCH
QV4_NO_REGALLOC Do not use the linear register allocator
QV4_FORCE_INTERPRETER Do not use the JIT but force the interpreter
QV4_MM_AGGRESSIVE_GC Run the GC on every allocation
QV4_MM_STATS Print the time it took to mark and sweep

Thursday, March 27, 2014

Long Time no See (好久不見)

Almost a year ago I took my Qt/DirectFB Jenkins down. With that CI for MIPS/uclibc and DirectFB stopped being done (as far as I know). It was a difficult decision but it reflected my situation. I didn't do a Qt project for a long time and didn't have any Qt related work on the horizon.

My main work involves using Smalltalk (GNU Smalltalk and Pharo) and writing a lot of a C code for the various Osmocom/GSM related projects and the joy of integrating software to build  HW products. I didn't think I would use/write/develop using Qt anytime soon. For sentimental reasons I stayed on the qt-devel mailinglist (I still read it day to day) and followed the work done by Lars, Simon and all the others with great joy.

For an internal project of sysmocom I needed to parse, generate and stream (through HTTP) JSON content. I built a first prototype using GNU Smalltalk and the Iliad Framework. The system was quickly built and we were able to gain some experience with it. Given the amount of data we intended to pipe through the system we wanted to move to a fully compiled version (instead of spending the time tuning the VM). I decided to use the Json support of QtCore, QtNetwork for the HTTP client and planned to use libsoup as the HTTP Server. After some research I stumbled across Tufao and used that one instead and don't regret it. Thanks to QThread and queued signal and slots I was able to make use of more than one CPU core and the application was fun to develop.

I am a Unix Dinosaur so for the buildsystem I shortly considered using qmake but then ended up using autotools. Thanks to the autotroll m4 macros building Qt applications is not that bad. I decided against using cmake as it still feels backward (e.g. no config.log, most scripts don't use pkg-config but some hand rolled magic like the FindBoost thing).

We are building Debian packages using an internal OBS appliance and this way can easily update/deploy our software thanks to that. After deploying every couple of hours the application/thread would crash. The backtrace pointed to QNAM, deleteLater and QObject.. David Faure had documented and fixed various race conditions and after we upgraded our application to use Qt5.2 the crashes stopped occurring. The application was last re-started on the 5th of February and works quite reliable.

This month we started a simple REST server using Qt and Tufao. Maybe I should search for a nice Qt consumer related project too? Qt is definitely here to stay.

Wednesday, December 18, 2013

Ported DBI and DBD-PostgreSQL from GNU Smalltalk to Pharo

We are building a gsmSCF in Pharo. Normally I would use MongoDB and Voyage to store my objects but as a gsmSCF deals with the balance of users I wanted to use something that supports transactions.

Looking at the SQL support in Pharo. I found SqueakDBX, DBXTalk, native Postgres drivers but all of them lacked prepared statement support and one really doesn't want to format queries by hand.

This is when I decided to port GNU Smalltalks DBI framework and the DBD-PostgreSQL driver. It is not the greatest database framework (QtSql is really close to that) but it does support prepared statements and I have running systems that use this framework.

I have used the gst-convert utility to convert the syntax from GST to FileOut syntax as understood by Pharo and converted the use of DateTime to DateAndTime, Dictionary>>#from: to Dictionary>>#newFrom: and some other incompatibilities. I have fixed some other incompatibilities by hand, e.g. >>#subString: aChar changed to >>#subString: aString and >>#nl to >>#cr. This gave me a port of ROE, DBI and DBD-PostgreSQL relatively quickly. I had some issues with Monticello not properly detecting certain extensions and I had to re-save the methodSource to fix that.

I spent the most time porting the DBD-PostgreSQL FFI code from GNU Smalltalk to NativeBoost of Pharo. The easy part could be converted easily. The literal array in Pharo is interesting and the usage in the >>#nbCallOut: is quite funny. The most difficult part was to create a char** for one call-out and type conversation. I had a de-tour using the NBExternalArrayType and then went to use NativeBoost class>>#allocate: and >>#asNBExternalString. It didn't help that when NativeBoost can't convert a parameter and there will be use-less error message. This is specially annoying when a function takes like 6 arguments or more. Anyway, I had some success yesterday night.

In GNU Smalltalk we are using namespaces so we have DBI.Connection as the entry point. In Pharo this is Connection right now but I should call it DBIConnection in the future. A small example is below and with simple tests it looks like there are no memory leaks in the code. I need to make the code work reliable across image save/restore.

 | connection statement result |
connection := Connection connect: 'dbi:PostgreSQL:dbname=aDb' user: nil password: nil.
statement := connection prepare: 'SELECT * from table WHERE col > $1 AND type = $3'.
result := statement executeWithAll: #(1 'abc').
[result atEnd] whileFalse: [
   | row |
   row := result next.
   row at: 'columnName'.

Tuesday, June 25, 2013

Using GNU autotest for running unit tests

This is part of a series of blog posts about testing inside the OpenBSC/Osmocom project. In this post I am focusing on our usage of GNU autotest.

The GNU autoconf ships with a not well known piece of software. It is called GNU autotest and we will focus about it in this blog post.

GNU autotest is a very simple framework/test runner. One needs to define a testsuite and this testsuite will launch test applications and record the exit code, stdout and stderr of the test application. It can diff the output with expected one and fail if it is not matching. Like any of the GNU autotools a log file is kept about the execution of each test. This tool can be nicely integrated with automake's make check and make distcheck. This will execute the testsuite and in case of a test failure fail the build.

The way we use it is also quite simple as well. We create a simple application inside the test/testname directory and most of the time just capture the output on stdout. Currently no unit-testing framework is used, instead a simple application is built that is mostly using OSMO_ASSERT to assert the expectations. In case of a failure the application will abort and print a backtrace. This means that in case of a failure the stdout will not not be as expected and the exit code will be wrong as well and the testcase will be marked as FAILED.

The following will go through the details of enabling autotest in a project.

Enabling GNU autotest

The file needs to get a line like this: AC_CONFIG_TESTDIR(tests). It needs to be put after the AC_INIT and AM_INIT_AUTOMAKE directives and make sure AC_OUTPUT lists tests/atlocal

Integrating with the automake

The next thing is to define a testsuite inside the tests/ This is some boilerplate code that creates the testsuite and makes sure it is invoked as part of the build process.

 # The `:;' works around a Bash 3.2 bug when the output is not writeable.  
 $(srcdir)/package.m4: $(top_srcdir)/  
  :;{ \  
         echo '# Signature of the current package.' && \  
         echo 'm4_define([AT_PACKAGE_NAME],' && \  
         echo ' [$(PACKAGE_NAME)])' &&; \  
         echo 'm4_define([AT_PACKAGE_TARNAME],' && \  
         echo ' [$(PACKAGE_TARNAME)])' && \  
         echo 'm4_define([AT_PACKAGE_VERSION],' && \  
         echo ' [$(PACKAGE_VERSION)])' && \  
         echo 'm4_define([AT_PACKAGE_STRING],' && \  
         echo ' [$(PACKAGE_STRING)])' && \  
         echo 'm4_define([AT_PACKAGE_BUGREPORT],' && \  
         echo ' [$(PACKAGE_BUGREPORT)])'; \  
         echo 'm4_define([AT_PACKAGE_URL],' && \  
         echo ' [$(PACKAGE_URL)])'; \  
        } &>'$(srcdir)/package.m4'  
 EXTRA_DIST = $(srcdir)/package.m4 $(TESTSUITE)  
 TESTSUITE = $(srcdir)/testsuite  
 DISTCLEANFILES = atconfig  
 check-local: atconfig $(TESTSUITE)  
 installcheck-local: atconfig $(TESTSUITE)  
  $(SHELL) '$(TESTSUITE)' AUTOTEST_PATH='$(bindir)' \  
  test ! -f '$(TESTSUITE)' || \  
  $(SHELL) '$(TESTSUITE)' --clean  
 AUTOM4TE = $(SHELL) $(top_srcdir)/missing --run autom4te  
 AUTOTEST = $(AUTOM4TE) --language=autotest  
 $(TESTSUITE): $(srcdir)/ $(srcdir)/package.m4  
  $(AUTOTEST) -I '$(srcdir)' -o $@.tmp $  
  mv $@.tmp $@  

Defining a testsuite

The next part is to define which tests will be executed. One needs to create a file with content like the one below:

 AT_BANNER([Regression tests.])  
 cat $abs_srcdir/gsm0408/gsm0408_test.ok > expout  
 AT_CHECK([$abs_top_builddir/tests/gsm0408/gsm0408_test], [], [expout], [ignore])  

This will initialize the testsuite, create a banner. The lines between AT_SETUP and AT_CLEANUP represent one testcase. In there we are copying the expected output from the source directory into a file called expout and then inside the AT_CHECK directive we specify what to execute and what to do with the output.

Executing a testsuite and dealing with failure

The testsuite will be automatically executed as part of make check and make distcheck. It can also be manually executed by entering the test directory and executing the following.

 $ make testsuite  
 make: `testsuite' is up to date.  
 $ ./testsuite  
 ## ---------------------------------- ##  
 ## openbsc test suite. ##  
 ## ---------------------------------- ##  
 Regression tests.  
  1: gsm0408                     ok  
  2: db                       ok  
  3: channel                     ok  
  4: mgcp                      ok  
  5: gprs                      ok  
  6: bsc-nat                     ok  
  7: bsc-nat-trie                  ok  
  8: si                       ok  
  9: abis                      ok  
 ## ------------- ##  
 ## Test results. ##  
 ## ------------- ##  
 All 9 tests were successful.  

In case of a failure the following information will be printed and can be inspected to understand why things went wrong.

  2: db                       FAILED (  
 ## ------------- ##  
 ## Test results. ##  
 ## ------------- ##  
 ERROR: All 9 tests were run,  
 1 failed unexpectedly.  
 ## -------------------------- ##  
 ## testsuite.log was created. ##  
 ## -------------------------- ##  
 Please send `tests/testsuite.log' and all information you think might help:  
   Subject: [openbsc] testsuite: 2 failed  
 You may investigate any problem if you feel able to do so, in which  
 case the test suite provides a good starting point. Its output may  
 be found below `tests/testsuite.dir'.  

You can go to tests/testsuite.dir and have a look at the failing tests. For each failing test there will be one directory that contains a log file about the run and the output of the application. We are using GNU autotest in libosmocore, libosmo-abis, libosmo-sccp, OpenBSC, osmo-bts and cellmgr_ng.

Friday, May 31, 2013

Don't use Hubstaff, if not for the ethical issues, for the security issues

I was recently pointed to a software called Hubstaff. It is meant for virtual companies that do not trust their employees and want to see what their staff is doing. Two central features are to measure the activity and to regularly send screenshots.

Hubstaff does not respect copyright! Their software is installing various GNU GPL and GNU LGPL licensed libraries without respecting the license of these libraries. The sourcecode was not offered at the same place and I didn't see a written offer.

Hubstaff is using HTTP (not encrypted) for all the traffic! The application is sending the login and password in clear text. If you use Hubstaff at the airport, at a local coffee place, in a shared office, everyone can see your password and take over your account. The activities, notes and screenshots are transferred in clear as well. This means that everybody can look at potential confidential information sent from your employee to the Hubstaff server.

To make it worse, it appears trivial that your employees will send you wrong activity information and screenshots. In general this is a game your employees will win but Hubstaff gives a huge head start to your employees.

In short Hubstaff does not respect copyright law, they don't value your data and they have not the slightest clue about security/privacy. No sane business will trust them.

Update: Hubstaff claims that starting from version 0.8.0 SSL is used for the connection to their server, they also claim that their GPL violations have been addressed. I have not verified those claims.

Monday, May 06, 2013

Spring cleaning of the

About a week ago I asked for help/support on the Continuous Integration of Qt for the combination of Linux/MIPS/UCLIBC/DirectFB and due the lack of feedback I have removed the DNS entry and cleaned up my system.

This means that currently there is no (public) build testing for any of DirectFB, MIPS, UCLIBC and the bitrot will increase over time. I have asked on the mailinglist about the removal of the Broadcom 97425 device support as it likely to be unused now.

Tuesday, April 30, 2013

Interested in MIPS/UCLIBC/DirectFB becoming a Tier1 platform?

Are you running Qt on a MIPS based system? Is your toolchain using UCLIBC? Do plan to use Qt with DirectFB? If not you can probably stop reading.

During the Qt5 development the above was my primary development platform and I spent hours improving the platform and the Qt support. I descended down to the kernel and implemented (and later moved) userspace callchain support for MIPS [1][2] in perf. This allows to get stacktraces/callchains for userspace binaries even when there is no framepointer. I stress-tested the DirectFB platform plugin and found various issues in DirectFB, e.g. this memleak. I modified the V8 MIPS JIT to provide the necessary routines for QML. While doing this I noticed that the ARM implementation is broken and helped to fix it.

At the time Nokia was still using Puls. This meant that getting an external build to integrate with their infrastructure was not possible. So I started to setup a Jenkins for DirectFB and Qt myself. The Qt Jenkins is compiling QtBase, QtJsBackend, QtXmlPatterns, QtDeclarative and QtWebKit for MIPS/Linux/UCLIBC. On top of these there a daily builds for the various QtBase configurations (dist, large, full, medium, small, minimal) and running the V8 unit tests using the built-in simulator for ARM and MIPS. The goal was to extend this to run the all the Qt tests on real hardware. The unit that supported my work was shut-down before I could implement it and the platform work has mostly been in maintenance mode since then.

This has all worked nicely for the release up to Qt 5.0 but when Qt5.1 got merged into the stable branch and received some updates the build started to break and I don't have enough spare time to fix that.

If anyone is interested in either taking over the CI or helping to make this part of my work again I would be very happy.