Author Archives: alex

Troubleshooting: Firefox Hangs when Connecting to Embedded Devices with Self Signed Certificates

As you might know my day job is about embedded Linux devices and as for all modern network connected devices you want TLS encrypted connections to those – which leads to all kinds of problems, especially with certificates. What we do here: let each devices create its own self signed certificate and teach the users to ignore the warnings in the webbrowsers.1

So almost all of our devices share the same CommonName (CN) and Issuer, and when developing I connect to numerous of them over time. So far no big deal, however Mozilla Firefox does not like all those similar but not equal certificates. It gets slow over time, really slow, so slow connections to the devices run into timeouts. Other sites are not affected however.

My colleague ran into this last year, my browser was not affected until today. What he did: start over with a new user profile in Firefox. Works, but all your addons and customizations are gone. You could of course also use another browser. Well, I did not want that, so I searched the web for a cause or even a solution.

I found that blog post Troubleshoot Firefox’s “Performing TLS Handshake” Message and it gave me a hint for a working workaround. What I did, deleted those two files from my profile (while Firefox was not running):

  • cert8.db
  • cert9.db

That did it, connections to my devices are nice and fast again. Have to search for the bugreport at the Mozilla Firefox project next …

Update: I tried to find some matching bug reports, and found some.

  1. Yes, we are aware of the problems, it’s complicated … []

Running EAGLE 9.6 on Debian 10 (buster)

For some side project I wanted to look at the schematics of the Adafruit PowerBoost 500C, which happened to be made with Autodesk EAGLE.

Having run EAGLE on Debian 9 (stretch) for a while now without great hassle, I did not expect much difficulties, I was wrong. First I downloaded the tarball from their download site. Don’t worry, there’s still the “free” version for hobbyists, however it’s not Free Software, but precompiled binaries for amd64 architecture, better than nothing.

After extracting, I tried to start it like this and got the first error:

There’s no verbose or debug option. And according to the comments on the blog post “How to Install Autodesk EAGLE On Windows, Mac and Linux” the problem also affects other users. I vaguely remembered somewhere deep in the back of my head, I already had this problem some time ago on another machine, and tried something not obvious at all. My system locale is German, looked like this before:

As you can see, no English locale, so I added one. On Debian you do it like this. Result below:

This seems like a typical “works on my machine” problem from an US developer, huh? Next try starting eagle, you’ll see the locale problem is gone, the next error appears:

That’s the problem if you don’t build from source against the libraries of the system. In that case EAGLE ships a shared object, which itself is linked against the from my host system, and those don’t play well together. I found a solution to that in a forum thread “Can’t run EAGLE on Debian 10 (testing)” and it is using the ld_preload trick:

This works. Hallelujah.

Performance Analysis on Embedded Linux with perf and hotspot

This is about profiling your applications on your embedded Linux target or let’s say finding the spots of high CPU usage, which is a common concern in practice. For an extensive overview see Linux Performance by Brendan Gregg. We will focus on viewing flame graphs with a tool called hotspot here, based on performance data recorded with perf. This proved to be helpful enough to solve most of the performance issues I had lately.

Installing the Tools

We have two sides here: the embedded Linux target and your Linux workstation host. For your computer you need to install hotspot. In Debian it is available from version 10 (buster). You can build from source of course, I did that with Debian 9 (stretch) a while ago. IIRC there are instructions for that upstream. Or you build it from the deb-src package from Debian unstable (sid) by following this BuildingTutorial.1

The embedded target part needs basically two parts. You have to set some options in the kernel config and you need the userland tool perf. For ptxdist here’s what I did:

  • Add -fno-omit-frame-pointer to global CFLAGS and CXXFLAGS

Note: I had to update my kernel from v4.9 to v4.14, otherwise I got build errors when building perf.

Configuring the Kernel

I won’t quote the whole kernel config here, but I have a diff on what I had to set to make perf record useful things. These options are probably important, at least I had those on in my debug sessions (others might also be needed):


Using it

For embedded use, I basically followed the instructions of upstream hotspot. You might however want to dive a little into the options of perf, because it is a very powerful tool. What I did to record on the target was basically this to get samples from my daemon application mydaemon for 30 seconds:

This can produce quite a lot of data, so use it with short times first to not fill your filesystem. Luckily I had enough space on the flash memory of the embedded target available. Then just follow what the hotspot README says: copy the file and your kernel symbols to your host and call hotspot with the right options to your sysroot. This was the call I used (from a subfolder of my ptxdist BSP, where I copied those files to):

Happy performance analysing!

  1. I did not test that with hotspot []

Troubleshooting nextcloud running on lighttpd

For historic reasons my personal nextcloud runs on lighttpd. (In fact I started in 2014 with ownCloud on Debian 8 (Jessie) and never changed the webserver.) This works, although it is not a first citizen supported platform, but after all nextcloud just needs some PHP/MySQL, so the webserver should not matter, should it?

After upgrading to nextcloud 15 and installing the social app/plugin, I got some warnings about missing rewrites/redirects. So it seems there are some new rewrites for service discovery in town, you can learn about those on the troubleshooting page of the administration manual.

You get config snippets for Apache and nginx (which are by far the most used webservers, I don’t blame nextcloud for not giving advise on every exotic webserver out there). For lighttpd you have to do it on your own and this is what I came up with. Load mod_redirect and mod_rewrite in your main lighttpd.conf and for the nextcloud config add this:

If you installed in a subfolder (the subfolder is still called owncloud here, adapt if needed):

For the future I consider changing the webserver, because lighttpd is not recommended by SabreDAV, which is used by nextcloud. But that’s not decided yet, for today this is it.

Wireshark USB capture setup with groups and udev

Wireshark does not only capture network traffic, but also different things like USB traffic. I needed that today and it needs some additional setup on Linux. There’s something in the Wireshark wiki on that topic, but I consider that not an elegant solution: USB capture setup.

The solution I use is basically one proposed on stackoverflow and uses a separate Linux system group and udev: usbmon (wireshark, tshark) for regular user.

On Debian you can do this:

You have to log off and on again, check if you are in that group with the command id.

Now create a new file /etc/udev/rules.d/75-usbmon.rules and put this into it:

After doing modprobe usbmon your devices /dev/usbmon* should belong to the new usbmon group and you can start capturing things with Wireshark.

SSH Remote Capture mit Wireshark

Manchmal will man ja mal auf einem entfernten Gerät oder einem Embedded Board den Netzwerktraffic beobachten. Auf meinem PC würde ich dazu Wireshark nehmen, das läuft so natürlich nicht auf einem Gerät ohne grafische Oberfläche. Üblicherweise kommt dort tcpdump zum Einsatz, dann allerdings mit sehr viel weniger Komfort als man von Wireshark gewohnt ist.

Einen Tipp für eine komfortablere Variante gab es im RFC-Podcast Folge 12: RFCE012: IP Routing III + in eigener Sache.

Man kann nämlich den Traffic auf dem fraglichen Gerät aufzeichnen und live zu Wireshark auf dem PC rüber schubsen, bspw. mit SSH.

Eine übliche Variante, die schon seit langem funktioniert, ist folgende. Man startet auf dem PC in der Konsole dies hier:

Was bedeutet das? Es wird auf dem PC der SSH-Client aufgerufen und angewiesen sich als Nutzer ‘root’ mit dem Rechner auf der IP zu verbinden. Dort soll er dann das Programm tcpdump mit gewissen Optionen starten. Ich rufe hier tcpdump mit vollem Pfad auf, weil es sein kann, dass es sonst nicht gefunden wird. Den Output davon bekomme ich im PC auf stdout und pipe den dann nach Wireshark, die entsprechenden Optionen bei dessen Aufruf bewirken, dass er die Daten auch verarbeiten kann und damit sofort loslegt.

Bisschen knifflig sind die Optionen von tcpdump, daher im einzelnen:

-i interface

Netzwerk-Schnittstelle, auf der tcpdump lauschen soll

“Packet buffered” Output, d.h. tcpdump sammelt nicht, sondern schickt den Output pro Paket raus
-w file
tcpdump schreibt keinen lesbaren Output auf die Konsole, sondern ein maschinenlesbares Format in eine Datei, in diesem Fall nach stdout
filter expression
Hier will man den Traffic der SSH-Verbindung selbst natürlich ausfiltern, vorsichtig sein mit den Klammern, die müssen entweder escapet werden oder man schließt den Filterausdruck in Hochkommata ein.

Soweit so gut. In neueren Versionen von Wireshark gibt es noch die Variante, das direkt aus dem GUI heraus aufzurufen. Die entsprechenden Optionen sind leider nicht so super dokumentiert. Was für mich funktioniert, sieht man im folgenden Screenshot:

Screenshot Wireshark SSH Remote Capture Setup

Einige leere Optionsfelder sehen so aus wie Argumente für tcpdump, die scheint Wireshark zumindest in der hier gerade eingesetzten Version v2.6.3 nicht zu berücksichtigen, daher auch Remote Interface und Remote Capture Filter mit in der Zeile Remote Capture Command.

Getting CDash to work on Debian 9 (stretch) with lighttpd and MariaDB

After reading It’s Time To Do CMake Right and The Ultimate Guide to Modern CMake I stumbled about the slides of Effective CMake by Daniel Pfeifer. (I did not watch the related video C++Now 2017: Daniel Pfeifer “Effective CMake” though.)

What attracted my attention where the commands ctest_coverage() and ctest_memcheck() from the slide about CTest which comes with CMake and which I already use. In libcgi and some non free projects I create additional tests to be run with valgrind if that tool is found on the build host, but when using CTest/CDash I don’t need to do that and also get coverage tests on top, so I set up a local CDash server on my workstation, which was painful in multiple ways.

After extracting the CDash archive and configuring lighttpd to server its PHP files the install.php came back with the following error message:

Specified key was too long; max key length is 767 bytes

It was not easy to find the cause. The web says this is fixed in MariaDB 10.2.x while my Debian stable still has 10.1.x … and I found only some workarounds on that problem for other projects than CDash. I could “solve” that by changing the database collation from utf8mb4_unicode_ci to utf8_unicode_ci in phpMyAdmin on the still empty database cdash before running install.php.

The more challenging problem was to actually submit results to the CDash server when calling CTest. The webserver always responded with HTTP Status Code 417. That was only partly fault of CDash, which seems to call curl with some strange (?) headers for submission. That turned out to trigger some (from my side) unexpected behavior in lighttpd, for which several tickets exist, I found #1017 eventually, which led me to the lighttpd 1.4.21 release info giving the hint I needed. I added this somewhere in my lighttpd config files:

In the end I have a local CDash instance now and could already submit some helpful coverage and memcheck results. Best thing: This way I can remove the error prone additional tests from my CMakeLists.txt and still run the tests with valgrind, even more flexible than ever.


So, what is this all about? Every now and then I want to try if I can solve some programming problem and make a tiny new project for that. For convenience I publish them on GitHub so I can clone them and play with them from multiple computers. Maybe someone else finds it and it helps someone, who knows.

I could name those tiny (not so) throw away projects like “testthis” or “evaluatethat” or come up with some more or less matching name, but GitHub has a nice feature when creating new repositories, GitHub makes a naming suggestion and I just pick that. See:

Screenshot GitHub new repository creation

Create a new repository on GitHub

Over time the number of those projects grew and to not lose track of them, here is a list:

Bridges with meaningful error messages

Just wanted to create a network bridge interface on an embedded Linux system. First try was with the well known brctl and I got this:

Searching the web for this, led me to an old blog post which comes to this conclusion:

So that’s the most silly way to say: you forgot to compile in bridge support into your kernel.

ip from busybox is a little more helpful on that:

And yes, the “real” ip from iproute2 is also as helpful as that:

And yes, somehow they are all right, it is actually my fault:

I’m going to change my kernel config now …

HowTo: Debug core dumps with ptxdist 2017.07.0

Debugging for embedded projects is a little harder than for your own computer. In many cases you can not run gdb on the target, and even if you can use gdbserver1 this does not cover all use cases. For post mortem analysis (e.g. after a segmentation fault of your program) you want to examine so called core dumps. Given you successfully found out how to let your target create those2, copied it to your workstation and unpacked it, you still need to know how to analyze it.

With the release of ptxdist 2017.07.0 the handling of debug information changed. Quoting from the release announcement:

The debug symbol handling was reworked. The debug files are now named based
on build-ids and (optional) debug IPKGs can be created. They are not
installed by default but can be installed manually as needed. This is
useful to gdb on the target or with valgrind and perf.

In my BSP those debug info is put to /usr/lib/debug in the root folder from which the target files are copied. This looks like this now:

You can imagine it is not possible anymore to load this manually, the debugger will have to find out by itself. Getting this to work caused me some headaches, but this it how I got it work: Create a file ‘gdb-config’ with the following content:3

Note: the order of the commands is important, it does not work the other way round!

Then load your core dump:

So you run the gdb from your toolchain, load the previously crafted file with gdb commands with -x, give the path to your executable with -e and finally the core dump file with -c and that’s it. You may now have a look at a backtrace and find out what caused the segfault …

Update: I was pointed to an easier possibility to invoke the right gdb with the necessary options in #ptxdist IRC channel (on freenode). The previous call would be like this:

No need to pick the correct gdb and find and configure the right directories, just add the path to your tool and your core file, ptxdist handles everything else. I bet this would also work with older ptxdist versions, where the debug symbols where placed somewhere else, but didn’t try it. This however was also just added with ptxdist 2017.07.0:

There is also a new ptxdist command ‘gdb’ for remote debugging that sets up
the sysroot correctly and a wrapper script that can be used by graphical
development environments.

  1. we already had this topic here: KDevelop: Debuggen von Programmen, die root-Rechte benötigen (German) []
  2. this would be content for another post []
  3. of course you adapt the paths to the ones you use on your machine []