Thoughts on Go

Tom Parkin - 08 June 2020

We recently decided to take a look at the Go programming language when developing our open source L2TP library.

In this blog post we explore the experience of using Go.

Our starting point

At Katalix we specialise in development for Linux systems, generally working on what might be described as "system software" -- that is, software which provides services to other software rather than interfacing directly with the user.

As you might expect, the languages we have tended to use for these applications have been C and C++. Alongside these we also make heavy use of scripting languages (mainly shell and Python) for tools and integration purposes.

Of course, C and C++ are the foundations of any Linux system. The kernel is written in C, as are large swathes of lower-level libraries and userspace components; and that's not likely to change any time soon.

This said, it's good to take a look at alternatives every so often; and a new open-source project seemed like an ideal opportunity to do just that.

New "systems" languages

There is no shortage of programming languages. Wikipedia's rather intimidating list of notable languages must stretch into the hundreds.

Choosing one in particular, then, is an exercise in winnowing the pool of possibilities down to some manageable subset that can be reasonably compared in order to arrive at "a winner".

We had some starting criteria to consider:

• It had to improve on C and C++ for implementing low-level system components. Important properties included: memory safety, a lack of undefined behaviour, high performance, and first-class support for concurrency.
• It had to be mature enough to be worth using for a new project. Immature languages change quickly, making it difficult to keep code up to date. We wanted something production-ready.
• It had to have sufficient popularity to be worth using for a new project. We wanted a language which had momentum behind it, and which looked to be growing in popularity rather than shrinking.

Two newer languages particularly stood out: Go and Rust.

Of the two, we found Rust the more interesting from a design perspective, but also the more alien coming from C/C++. We'll definitely keep an eye on Rust for the future, but for this project we decided to give Go a try.

Getting going

The first port of call was Go's documentation to get the toolchain installed, and to follow the Tour of Go. The latter is a great introduction to the language. Although minimal, it covers all the key points a newcomer needs to know in short order.

The jewel in its crown, however, is the embedded "playground" feature which lets you complete exercises right there alongside the tutorial notes. If you're the sort of person who learns more through doing something than reading about it you'll appreciate the playground!

Once we'd worked through the Tour, we were off to the races, armed with references to the effective Go guide as well as the language specification and memory model.

Songs of experience

At this point, we've completed a small project, go-l2tp, and made a small contribution to the Go standard library (I will talk more about this later).

Although we are by no means expert Go coders, we'd be comfortable working on further Go code bases and designing systems using Go.

So the time seems ripe to reflect on what we've learned, and to sum up our experience of using Go for systems component coding in 2020.

Go: the good bits

There is a lot to like about Go:

• Familiar-looking. If you're coming from a C or C++ background, Go syntax is comfortingly similar; and it's generally not hard to read code and figure out what it is doing.
• Good coding environment. We used neovim with the faith vim-go plugin which made for a great experience.
• Consistent and automated formatting (go fmt). This is something that you don't really appreciate until you've used it for a while! But it is a massive time-saver to have one consistent, automated way of laying out code; time which can be better spent elsewhere.
• Integrated documentation tooling (go doc). Similar to go fmt, the benefits of automated and integrated documentation tooling are non-obvious until you've become used to them. With appropriate editor macros, it's possible to pull up the documentation for any library I'm using, or even for functions in my own code base. Its like integrated manpage lookups, but for the whole code base, and is very, very useful.
• Integrated test environment (go test). By including a test framework as a core part of the language toolchain, the Go devs have made testing a first-class citizen. And it's great! Having the test framework right there from day one in your project makes it practically friction-free to add tests as you work.
• A helpful and supportive community. There are multiple different ways you can get help with Go. We made heavy use of the Go Slack workspace, as well as the golang-nuts mailing list. Without exception, the interactions we had were positive across a wide range of topics. We got help with newbie mistakes. We got design advice and suggestions as our project grew in scope and sophistication. We even got a generous response from one of the language designers when we thought we'd spotted a bug in the standard library (no bug as it transpired, you'll be pleased to learn). A good community makes such a difference to a newcomer.
• Fast builds. There's not too much to say about this, other than: fast builds are a good thing!
• Easy deployment. We must add a bit of a caveat here: we've not really used this in anger so far. However, the concept of a single static binary which can be installed by simply copying it into place is quite attractive!
• Memory safety. Explicit memory management is both one of the great benefits and one of the great drawbacks of coding in C. Go's approach is very refreshing for situations where fine-grained control of memory just isn't necessary (or even desirable!). During development of go-l2tp we never struggled with memory lifetime issues, and only rarely hit data race problems. The elimination of a whole class of potential bugs is wonderful.
• Interfaces and embedding. Simplifying somewhat, these are Go's answers to polymorphism and inheritance. As is often the case with Go, they're simple mechanisms, and you can get surprisingly far with them.
• Error values. Go makes error handling a primary concern with explicit error values. This does tend to encourage code which checks and handles errors as a matter of course rather than as an afterthought. There is a downside to this approach, which is that it does tend to obscure the intent of a given bit of code; to the extent that there are proposals to refactor how errors are dealt with for Go 2. On balance, though, I think error values are a win.
• Concurrency. Go's killer feature is perhaps its inbuilt support for concurrency. Having this as a core part of the language is a total game-changer. As an example, during the development of go-l2tp I needed to serialise access to a resource as I scaled the library's capabilities. I decided to do so using a goroutine (effectively, although not literally, an OS thread) pulling requests from a queue. If I were doing this in C, I'd need to do a bunch of pthreads coding, possibly implement my own queue, and worry about mutexes and locking. In Go it took a few tens of lines to wrap the required data in a structure, add some channels for communication, and implement a simple select statement to read from the channels. It's hard to overstate how much easier concurrency is in Go than it is in C or C++.

Go: the bad bits

As we all know, nothing is perfect, and Go is no different to anything else in this respect. Some of these niggles are more about learning the tooling and environment as much as anything else, but still, these are things we struggled with to some extent while writing go-l2tp:

• The move from $GOPATH to modules. The Go ecosystem as a whole is in a period of flux in terms of how dependencies are managed in a project. The problem is that lots of documentation on the wider internet refers to the old approach. The Go documentation itself covers both old and new approaches (as both are supported). The net result is that it's challenging for a newcomer to figure out what they should be doing.
• Some confusing tooling behaviours. Perhaps as a consequence of the move from $GOPATH to modules, some of the toolchain behaves in a confusing manner. Frustratingly, the most confusing behaviour is around how to go from a newbie-friendly "single file in a directory" type repository layout to multiple subdirectories with multiple files. The documentation really doesn't do a good job of holding your hand here: we ended up asking on Slack.
• Debugging. I get the impression that logging is how Go debugging is supposed to work. GDB is next to useless for debugging Go processes, in part thanks to the runtime. There is a Go symbolic debugger called delve which I tried to use, but never got further than an opaque error. I asked on the #delve channel on the gophers Slack workspace but never got a response.
• Distance from the C API. As we touched upon earlier, Linux is mostly built on C. As a C programmer, I'm well versed in talking to the OS using the APIs exposed by the C standard library, and when we started out on the go-l2tp development work I had assumed that these APIs would be easily accessible from Go. This wasn't entirely the case. The main problem we had was that the socket address structures used by the Linux kernel L2TP subsystem were not available from Go code. There are two approaches to accessing them. The first is to patch the Go unix package (effectively part of the standard library). This wasn't so very painful to do, as it turned out, but still, it was quite an overhead just to access a socket address type. The second approach is to use cgo to access the structures directly. I ended up using this technique later for another address structure unknown to Go in order to avoid patching the standard library again, but the results are not pretty.
• Runtime magic. Generally, the Go runtime is great, and does a lot of work so the programmer doesn't have to. But in a way, that's a problem. For go-l2tp, we work a lot with sockets, making reference to file descriptor numbers in kernel netlink messages, and making system calls on sockets to establish the L2TP data plane connection. This turned out to be quite painful to deal with in Go, because file descriptors per-se are something of a second class citizen: the runtime and the underlying magic that makes efficient IO possible depend on Go types, not file descriptors. We ended up being able to do what we needed using some slightly obscure interfaces hidden away in package syscall, but figuring this out wasn't easy.
• Interoperability. One nice thing about writing a library in C is the reasonably certain knowledge that it'll be usable by just about everyone: most mainstream languages are able to call C in some form or other. That's not so much the case with a Go library, due to the fact that Go depends on the Go runtime. At the time of writing the runtime cannot be embedded. So our go-l2tp library is only going to be useful to people wanting to write Go code.
• Concurrency. As I said earlier, Go's concurrency support is fantastic. But once you've started using it, I can guarantee you'll end up debugging dining philosophers deadlocks within short order. Experienced gophers will nod wisely and say things along the lines of "never start a goroutine without a very clear idea of when and how it will stop". There is of course no substitute for experience and a robust design, but to some extent by making concurrency easy, Go makes it easy to write concurrency bugs too.

Conclusions

Going into go-l2tp development, we'd chosen to use Go as it seemed like a decent "systems programming" language, with a strong focus on networking. In short, it sounded perfect for an L2TP library!

That impression turned out to be slightly misguided. Yes, we did manage to build the library in the end, but it was a surprise just how many hoops we had to jump through in order to do so.

Over the course of the go-l2tp development work I hung out a lot on the gophers Slack workspace and have come to the conclusion that go-l2tp is perhaps the wrong type of networking code to be a natural fit for Go right now. Perhaps unsurprisingly given Go's origins at Google, Go is very good at the sort of jobs you're going to end up doing if you're writing an internet service of some description. Talking TCP is its bread and butter! Talking L2TP, somewhat less so.

As an adjunct to the previous point; Go is a very opinionated language. You can infer as much from the fact that it bundles a source code formatter and a test harness. I think this is often a good thing: as I mentioned before, these tools are huge time savers, and a massive win on the whole. So often having a strong opinion is a good design decision.

Where it can fall down slightly is if your particular usecase doesn't overlap neatly with the position Go has taken.

All this said, these are relatively minor niggles. Some of the issues we had can of course be attributed to the natural learning curve you'd have when learning to use any new language.

On the whole, I've come away from go-l2tp having had a very positive experience using Go. It's a small, relatively simple language, which I love -- at times it feels like a nicer C, which I mean as a compliment. The Go community generally was fantastic in our interactions with them. And Go as a "systems software" language in 2020 does make a lot of sense for the memory safety and concurrency implementation alone.