Using KVM with Qemu on ARM

This is part two of my blog post about Kernel-Virtual Machine (KVM) on a 32-Bit ARM architecture. The post is meant as a starting point for those who want to play with KVM and provide a useful collection of Qemu commands for virtualization.

Virtualization host setup

The Kernel configuration I used for my platforms Host kernel can be found here. Since I run my experiments on a Toradex Colibri iMX7D module, I started with the v4.1 configuration of the BSP kernel and updated that to v4.8 plus enabled KVM as well as KSM (Kernel same-page merging).

As root file system I use a slightly modified version of the Ångström distributions “development-image”, version 2015.12 (built from scratch with OpenEmbedded). Any recent ARM root file system should do it. I let Qemu v2.6.0 preinstall (by just adding “qemu” to the image and specifying ANGSTROM_QEMU_VERSION = “2.6.0” in conf/distro/angstrom-v2015.12.conf).

Virtualization guest setup

For the virtualization guest setup I was looking for something minimalistic. I uploaded the compiled binary of the Kernel (as tared zImage) and initramfs (as cpio.gz).

I built a custom kernel directly using v4.7 sources and a modified/stripped down version of the vexpress_defconfig (virt_guest_defconfig). I found it useful to look into Qemu’s “virt” machine setup code (hw/arm/virt.c) to understand what peripherals are actually emulated (and hence what drivers are actually required). Read more »

U-Boot/Linux and HYP mode on ARMv7

The newer ARMv7 Cortex-A class cores such Cortex-A7, A15 and A17 come with a virtualization extensions which allow to use KVM (kernel virtual machine). The NXP i.MX 7Dual SoC which I worked with lately includes the ARM Cortex-A7 CPU. I went ahead and tried to bring up KVM on i.MX 7. I was not really familiar with the ARMv7 virtualization architecture, so I had to read up on some concepts. This post summarizes what I learned and gives a big picture of software support.

The Hypervisor mode

To provide hardware support for full CPU virtualization an additional privilege level is required. User-space (PL0) uses the SVC (Supervisor) instruction to switch to kernel-space (PL1, SVC mode). A similar separation between Kernel and hypervisor is required. The ARMv7 architecture with virtualization extension calls this privilege level PL2 or HYP mode.

Linux with KVM for ARM uses this mode to provide CPU virtualization. The CPU needs to be in HYP mode when Linux is booting so KVM can make use of the extension. How KVM uses the HYP mode in detail is explained in this excellent LWN article. After building a kernel with KVM support, I encountered this problem first: By default, the system did boot in SVC mode.

Brought up 2 CPUs
CPU: All CPU(s) started in SVC mode.
kvm [1]: HYP mode not available

Secure and Non-Secure world

ARMv7 privilege levels

ARMv7 privilege levels

To understand how to switch into Hypervisor mode, one needs to understand the whole privilege level architecture first. Notable here is that on ARMv7 CPU’s the HYP mode is only available in non-secure mode, by design. Any hypervisor needs to operate in non-secure mode, there is no virtualization extension in secure mode. Read more »

Using the perf utility on ARM

Given two systems, both with a Cortex-A5 CPU, one clocked at 396MHz without L2 cache and one clocked at 500 MHz with 512kB L2 cache. How big is the impact of the L2 cache? Since the clock frequency is different, a simple CPU time comparison of a given program does not answer the question… I tried to answer this question using perf. perf is often used to profile software, but in this case it also proved to be useful to compare two different hardware implementations.

Most CPU’s nowadays have internal counters which count various events (e.g. executed instructions, cache misses, executed branches and branch misses etc…). Other hardware, e.g. cache controllers, might expose performance counters too, but this article focuses on the hardware counters exposed by the CPU. Read more »

Linux earlyprintk/earlycon support on ARM

Tux with some earlycon bootlogThe serial console is a very helpful debugging tool for kernel development. However, when a crash occurs early in the boot process, one is left in the dark. There are two early console implementations available with different merits. This post will throw some lights on them.


The kernel supports earlyprintk since… probably ever. At least 2.6.12, where the new age (git) started. After enabling “Kernel low-level debugging”, “Early printk” under “Kernel hacking” and selecting an appropriate low-level debugging port, you are ready to get early serial console output. Read more »

Debug IPv6 issues on OpenWrt

IPv6 logo

My new provider IPv6 connectivity, however after connecting my OpenWrt based WNDR4300 (using Chaos Calmer 15.05-rc3) things unfortunately did not “just work”… This post summarize some hints how to debug IPv6 on OpenWrt.

First, make sure that the upstream (WAN) interface is configured according to the documentation in the OpenWrt Wiki. Using LuCI, you have to create a new interface (WAN6) which should c

If your ISP provides IPv6 support, it should “just work” at this point. However, what to do if not? Read more »

DNSSEC support in OpenWrt 15.05 Chaos Calmer

DNSSEC does not require any special support on the router, since the validation is typically done by the client itself. However, caching the DNSSEC records makes validation for clients faster, and a router in a trusted network can provide DNS replies which carry the ad flag (authenticated data).

Chaos Calmer comes with dnsmasq without DNSSEC validation support by default (DNSSEC is not enabled at compile time). However, one can use the dnsmasq-full package, which provides the DNSSEC features. Without DNSSEC compiled in or enabled, dnsmasq forwards the DNSSEC records properly, however it does not validate them and therefor the DNS replies do not contain the ad flag:

# dig SOA +dnssec

; <<>> DiG 9.9.6-P1 <<>> SOA +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62577
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 6, ADDITIONAL: 11
   ^^^^^^^^^^^^^^^ (no ad flag, dnsmasq did not authenticated the DNS response)

Read more »

Vybrid Cortex-M4 Linux support – Status update

Vybrid Cortex-M4 Linux boot log

Vybrid Cortex-M4 Linux boot log

This is a status update about my work on Linux for Cortex-M4 inside Freescale Vybrid SoC’s. While I managed to boot Linux on the M4 core since fall last year, several improvements have been made since then.

MSCM interrupt router driver

Peripheral interrupts need to be routed to the core requiring them. The routing is done by the interrupt router inside MSCM (Miscellaneous System Control Module). So far, the boot loader (U-Boot) configured all interrupts to the primary core. A proper driver for the interrupt router was missing, hence to get Linux running on the secondary core, manual fiddling of the interrupt router registers was needed. Read more »

Cross compile Linux for ARM using LLVM/clang on Arch Linux

LLVMLinux project logo

LLVMLinux project logo

The LLVMLinux team made quite some progress in enabling LLVM/clang to build the Linux kernel. There has also been a talk at the Collaboration Summit 2015 in Santa Rosa (slides). So I thought, give it a try and compile a kernel for my ARM based board of choice. The upstream Linux kernel is not yet ready to be built using LLVM/clang, some patches are still required. Therefor I used the kernel from the LLVMLinux git server.

The nice thing about LLVM/clang is that it supports multiple targets. The binaries provided by the Arch repositories support several ARM targets:

$ pacman -S clang
$ llc --version
 LLVM version 3.5.1
 Optimized build.
 Built Jan 14 2015 (03:18:15).
 Default target: x86_64-unknown-linux-gnu
 Host CPU: core-avx-i

 Registered Targets:
 aarch64 - AArch64 (little endian)
 aarch64_be - AArch64 (big endian)
 arm - ARM
 arm64 - AArch64 (little endian)
 arm64_be - AArch64 (big endian)
 armeb - ARM (big endian)

Read more »

Starting BuildBot using systemd Service unit

This post shows how to create Service unit files for systemd which let start and restart BuildBot on Linux distributions using systemd. This unit files depend on a working VirtualEnv installation of BuildBot in the /srv/buildbot directory. It also runs BuildBut with the user/group “buildbot”. Two unit files are necessary to start the master and slave instance respectively. Instead of using the buildbot/buildslave commands, the twistd binary is executed directly in foreground. twistd is an application/service framework for Python, which BuildBot makes use of.


Description=BuildBot master service

Environment="VIRTUAL_ENV=/srv/buildbot/sandbox" "PATH=/srv/buildbot/sandbox/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
ExecStart=/srv/buildbot/sandbox/bin/twistd \
          --nodaemon --no_save --pidfile= --python=master/buildbot.tac


buildbot-slave.service Read more »

Fixing a broken Git repository

After having several hard kernel crashes, some of my files ended up empty on my disk. I then got errors like:

error: object file .git/objects/81/b43db0dadcafe18eef75c60c57cc496d523d83 is empty
error: object file .git/objects/fb/9b5b3daa56c7833be66f0eaac2b5a906474627 is empty

I can recommend this Wiki page on which tells how recovery basically should be done. This post shows some of those steps a bit more “hands-on”. I started with moving this corrupt (empty) object away, and then got this message when doing some git operations:

fatal: unable to read 81b43db0dadcafe18eef75c60c57cc496d523d83

Read more »