Crosscompiling with Cgo

After succeeding in crosscompiling Gitea I wanted to setup a test installation of Gitea with support for sqlite so I didn't have to setup a database before being able to fiddle around with the installation's settings. As mentioned in the source install instructions for Gitea you can specify an environment variable to include sqlite support in the Gitea build:

$ TAGS="bindata sqlite sqlite_unlock_notify" make generate build

Trying to crosscompile this results in the Gitea installation wizard to complain about being built without CGO_ENABLED=1:

Issue #384 and its discussion at the go-sqlite3 repo sheds some light on what's supposed to happen. Apparently sqlite requires to call C code from within Go, so Cgo (documentation) needs to be enabled via CGO_ENABLED=1

$ make generate
$ env CGO_ENABLED=1 GOOS=linux GOARCH=arm GOARM=7 TAGS="bindata sqlite sqlite_unlock_notify" make build
GO111MODULE=on go build -mod=vendor -v  -tags 'bindata sqlite sqlite_unlock_notify' -ldflags '-s -w  -X "main.MakeVersion=GNU Make 4.1" -X "main.Version=1.10.0" -X "main.Tags=bindata sqlite sqlite_unlock_notify"' -o gitea
runtime/cgo
# runtime/cgo
gcc: error: unrecognized command line option '-marm'; did you mean '-mabm'?
Makefile:357: recipe for target 'gitea' failed
make: *** [gitea] Error 2

Duh, this isn't as easy as I thought it would be. The error thrown by gcc unrecognized command line option '-marm' points out that it doesn't know about ARM. Looking at the GCC documentation we can read up that -marm instructs GCC to compile our code for the ARM instruction set. We need a version of gcc that knows how to do that. More specifically we need a compiler that targets ARM platform. I'm going to stay on my amd64 machine for compile speed so we need a crosscompiler that runs on amd64 and spits out arm. We can achieve that by building a crosscompiling toolchain which is kind of complex. However there are tools like crosstool-ng that can take care of that!

After downloading and compiling crosstool-ng we can get a list of available templates via

$ ct-ng list-samples

crosstool-ng offers the armv7-rpi2-linux-gnueabihf template which looks like a perfect fit for me. We can enable that template (line 1), and tweak it if we want to (line 2). After that we can run it (line 3).

$ ct-ng armv7-rpi2-linux-gnueabihf
$ ct-ng menuconfig
$ CT_SAVE_TARBALLS=y CT_LOCAL_TARBALLS_DIR=tarballs CT_DEBUG_CT_SAVE_STEPS=1 ct-ng build

I've chosen some variables to run ct-ng with:

• CT_SAVE_TARBALLS: this enables saving the downloaded tarballs for tools which are part of the toolchain to build
• CT_LOCAL_TARBALLS_DIR: specifies where to keep the tarballs
• CT_DEBUG_CT_SAVE_STEPS=1: enable the debug setting of saving intermediate results of build steps, you can see the steps with $ ct-ng list-steps. This is really useful since building a toolchain takes a lot of time. Building the armv7-rpi2-linux-gnueabihf toolchain took about 60 minutes on a i5-6600K at 3.5 GHz. If the build fails crosstool-ng will save the results of the previous successful steps and will allow you to retry again starting with the failed step:

$ ct-ng build
...
Saving state to restart at step 'debug'...
...
[ct-ng fails and provides a stacktrace with info about what to fix]
...
$ ct-ng debug+
[ct-ng continues at step 'debug']

After the toolchain building finished you'll have a compiler for your target platform available.

I continued to try to crosscompile Gitea with Cgo enabled and specified which compiler to use via the CC and CC_FOR_TARGET environment variables:

$ make clean
$ make generate
$ env \
CC=/home/rnorden/raspi-toolchain/.build/armv7-rpi2-linux-gnueabihf/buildtools/bin/armv7-rpi2-linux-gnueabihf-gcc \
CC_FOR_TARGET=/home/rnorden/raspi-toolchain/.build/armv7-rpi2-linux-gnueabihf/buildtools/bin/armv7-rpi2-linux-gnueabihf-gcc \
CGO_ENABLED=1 GOOS=linux GOARCH=arm GOARM=7 TAGS="bindata sqlite sqlite_unlock_notify" make build
$ file ./gitea
./gitea: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-, for GNU/Linux 4.20.8, stripped

Compare that to the previous build without Cgo:

./gitea: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, Go BuildID=h9RD9VL6olgt11cIx9Zu/p6AnFAzcl0jeK_YVt595/wp57ui5pSk9xMq4SLCiY/dWlUM83_sURlHvHilueb, stripped

The important difference is that we've build a dynamically linked binary instead of a statically linked one. I won't go into details here about linking but the output of ldd will show us what that's about:

$ ldd ./gitea_statically_linked
	not a dynamic executable
$ ldd ./gitea_dynamically_linked
	linux-vdso.so.1 (0x7eff0000)
	/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76f56000)
	libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0x76f2a000)
	libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x76f00000)
	libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76db2000)
	/lib/ld-linux-armhf.so.3 (0x76f6b000)

The output of ldd will only be correct when executed for the binaries' target platform. We can see that the go compiler set up the binary in a way that it specifies it's dynamic dependencies. That way the dependencies don't get included in the binary and will be resolved on the OS running the binary. I'm no expert on C and compiling so I don't know specifics about the CC* variables used. I'll try to get some more info and update the post in the future.

With all that working correctly Gitea's install wizard didn't complain anymore and happily finished setting up my Gitea install :-)