Have you ever had cross-compilation dreams ?

Because I have, and they always were shattered by the pesky C tooling. Look, I respect C, I love C for what it has permitted to develop. I like the simplicity of it and the way you are not disturbed by anything while coding. I also understand why every OS kernel is written with it and why every major system tools are coded with it.

BUT

The tool chain is unbearable. That’s it. It evolved for far too long without having to do any form of cross-compiling and now if you want to use any C dependencies in a program you plan to cross compile it is a complete ordeal.

You need the standard library for the target system, you need the compiler for the target system and it, itself, must be compiled for the system you want to use to cross-compile. Some OS have multiple standard libraries and there might be several compilers for your target. By simply adding a C dependency you are completely jeopardizing your entire Go project, if you plan on cross -compiling.

Because that’s right you can add C dependencies in your Go programm. Using CGO, wich is packaged with Go, you can link C libraries with the compiler installed on your system. While CGO is an incredibly useful tool that I am happy to have if needed be I can’t stop building all of my go projects with CGO_ENABLED=0. I just want to be SURE that I will still be able to cross-compile easily. I am not alone in this case because this well known post also talk about it. One of the main reasons for me to use Go is to not have any problem when I want to distribute my software. I want a statically compiled single native binary and all this from the same codebase. This must be true for every major OS and every major instructions set. Even wasm. That’s what was advertised to me and that’s what I want.

The case of SQLITE

Of course, and I said it at the start of this post, nobody is going to pretend to rewrite every C software in other languages and it would be a lot of work to do it in Go. For more information you can read the post from the official SQLITE website talking about it.

So what do we do ? SQLITE is so useful when you need to organize relational data and then everyone and their dog is using it to write small tool. You want to create a little tool that has to store pesistent data ? you need SQLITE. You have to interface with another program using it, you have to use it. It’s at the same time an incredible piece of software available everywhere and one of the most unpractical code base to cross compile. Interpreter like Node or Python bundle it but if you want to stay in the Devops mindset you want to ship one binary and that’s it. Meaning using a native compiled binary. Meaning compiling the C code at some point.

The best pure Go implementation

When I said noone was rewriting it, I lied. There is one very good pure go reimplementation of SQLITE. Performant and clean, but not feature complete. Ladies and Gentleman: SQLITE in pure Go. The only C dependencies are the tests so they are not compiled in the release. While this is impressive you still want to be 100 percent compatible with SQLITE implementation. Feature wise and bug wise. Because what would be more terrible than using a library to conform to a file standard, for it to fail you after years of use in your codebase because some feature is not exactly working the same ? What would be the best way to still have the same behaviour as the C code without having to compile it using CGO and without having to recode it in another language ?

Circumventing this problem in a peculiar way

Ok so let’s sum everything up. You want to use Go and SQLITE but don’t want to depend on CGO because cross-compiling C code is horrendous. You don’t want to use a Go implementation too because this is a big project with lots of features and you want to be sure you are on par with the original lib. What could be a way to run SQLITE withtout rewriting everything and still having all the core functionnalities ?

WASM

BOOM

You didn’t expect it, did you ?

Wasm, the compiled language of the web, is gonna save us all from this cross-compiling madness…by introducing more complexity. Ok, are you ready ?

We are gonna use the wasm runtime Wazero. It allows us to have a complete Wasm runtime running in our binary and we can now ship wasm code to any platform because Wazero itself is coded in pure Go. But wait, we are using a C library and you are talking about wasm. Yes but the project is using a transpiler from C to WASM named emscripten. With this we can have a complete wasm build of our library and embed it in our Go executable. When importing the module, the WASM file will be used for any call we make to the library. The project is hosted here and this is a lot of thoughts that went into this.

This is still not a perfect solution because some changes had to be made and it comes with some modifications in behaviour. Though this is mainly if you are using the VFS so for a regular usage or for compatibility you are good to go.

Thoughts for later

With this method it seems quite “easy” to add C dependencies in Go binaries and still have the full power of cross compilation. I wonder if other libraries will later be ported to go in the same way. After all, with minimum work you can achieve good performance and port a number of C libraries easily.

Seeing the ingenuity of the people who thought of this I can only wonder: will we find a good way for Go to have a proper cross-platform GUI library without CGO ? At the time it seems impossible but some project like Wails give me hope. Could it be that one day every major OS will provide a way to run WASM or HTML content and allow us to remove C wrapper from our code ? Will the web standard infuse to the system layer ?