Getting started with Go kit

Go kit is often accused of being too complicated, at least compared to other libraries used for building applications. This post ought to explain the differences between them and demonstrate the thought process of building an application with Go kit.

What is Go kit?

Go kit is advertised as a “standard library/toolkit for microservices”, deliberately avoiding the term framework. While frameworks often require you to write code following a certain style (eg. put everything in an HTTP handler), Go kit provides an integration layer between your business logic and various transports (like HTTP or gRPC), allowing you to focus on what’s important: your business.

Obviously, similar architecture can be achieved with traditional frameworks as well, but they hardly ever provide any tooling or documentation for that. In fact, they often encourage you to hard wire your business logic into their constructs (think about MVC frameworks). Not because they want to do harm, but because it makes keeping their number one promise (rapid development) much easier. In contrast, Go kit is purpose built for integrating your business logic.

Although Go kit’s motto says “for microservices”, it’s a perfectly reasonable choice for modularized monolithic applications (elegant monoliths) as well. It even helps in setting boundaries between modules which is quite useful when you want to break a large application into microservices.

Architecture

The fundamental architecture behind Go kit consists of three layers:

Go kit architecture

Your business logic is at the core of the application. Finding the right algorithmic solution to a business problem is not easy, so it’s important to be able to focus on that. Go kit makes very little assumptions on how your business logic should work, so you have quite a bit of freedom in how you implement it.

(In Go kit terminology this layer is actually called service layer, so that’s how I will refer to it from now on.)

That freedom comes at a price though. In order to ensure interoperability between your business logic and the transport layer, Go kit generalizes your services in the endpoint layer by wrapping them with a request-response based (RPC-style) interface.

Every business call in your service layer becomes an endpoint, a generic controller/action handler. This is a very powerful abstraction, because various transports in the transport layer can talk to the same interface without any additional mapping or conversion. When migrating from a legacy HTTP API to a gRPC service this is incredibly useful, but having to support multiple transports in a microservice architecture is not without precedent either.

In addition to providing an interoperability layer, endpoints are also great for abstracting common application problems away from transports. For example, after loading credentials from the request in the transport layer (eg. Authorization HTTP header) into the context, a middleware in the endpoint layer can actually verify those credentials to authenticate a user. Once again, this is quite useful when dealing with multiple transports in the same application.

The role of the transport layer is probably obvious by now: it provides concrete transport implementations (eg. HTTP, gRPC, Thrift, AMQP, etc). With the endpoint layer in between services and transports the only thing you have to take care of in this layer is decoding/encoding requests/responses.

If the above diagram seems familiar that’s not a coincidence. Go kit’s lightly opinionated architecture fits many software architectural patterns. For example, the above three layers can easily be matched to the layers of Clean architecture.

The infamous dependency rule applies to Go kit applications as well: dependencies always point inward. The service layer knows nothing about endpoints, the endpoint layer knows nothing about transports.

Clean architecture

Building an application

For many people the most effective way to learn is studying examples. Go kit comes with several examples from simple to really complex ones, but they don’t really demonstrate how the process of building an application using Go kit works.

That’s what I’m trying to do in this post. In the following sections you can read about building a simplified version of addsvc, one of the official Go kit examples.

Keep in mind that the example is oversimplified: it’s supposed to demonstrate the process, not the best practices. Checkout the original example if you want to play with the code.

Service layer

Go kit puts your business logic in focus, so it makes sense to start from there and move up in the stack following a bottom-up approach.

Designing an “idiomatic” Go kit service starts with an interface:

package addsvc

import "context"

type Service interface {
    // Sum accepts two numbers and returns the sum of them.
    Sum(ctx context.Context, a int, b int) (int, error)
}

Consumers will interact with this interface to make calls to the service. Although that’s not really the Go way of using interfaces, Go kit promotes this pattern to provide a contract that the package makes with its consumers. More about that later.

Let’s get back to our service and implement it:

type service struct{}

func (service) Sum(ctx context.Context, a int, b int) (int, error) {
    return a + b, nil
}

Ideally, your business logic should be the most important part of your application. It should be well-covered with unit tests.

Finally, we need a factory function (often called a constructor in Go):

func NewService() Service {
    return service{}
}

Wait, what? Return an interface? That’s not idiomatic Go. Well, this is the exception to the rule for two reasons:

I mentioned above that the Service interface is the contract that the package provides to it’s consumers. Consumers can still define their own interface (like the Go practice says), but in a Go kit style application the service interface also serves as a definition for a developer explaining what the service does. Since the business logic is transport agnostic, definition languages like Protobuf or OpenAPI are less ideal for this purpose.

There is also a more practical reason: Go kit encourages the use of service middleware. A service middleware is something that implements the service interface and adds some behavior to it using composition. Typical examples are logging, service level monitoring and tracing, authorization, etc:

import "log"

type serviceLogging struct{
    service Service
}

func (s serviceLogging) Sum(ctx context.Context, a int, b int) (int, error) {
    log.Print("Called sum")

    return s.service.Sum(ctx, a, b)
}

The service constructor is supposed to return a fully configured service, that means it should also initialize all middleware. In that case, a concrete return type cannot be reliably determined, because it depends on middleware and would break consumer code when that changes, so an interface must be returned:

func NewService() Service {
    return serviceLogging{service{}}
}

Endpoint layer

The endpoint layer provides an abstraction between services and transports. It’s sort of an adapter from the transport point of view.

A single endpoint acts as a controller for a single service method, so each business call in a service must be wrapped in an endpoint:

type sumRequest struct {
	A int `json:"a"`
	B int `json:"b"`
}

func MakeSumEndpoint(service Service) endpoint.Endpoint {
	return func(ctx context.Context, req interface{}) (interface{}, error) {
		request := req.(sumRequest)

		return service.Sum(ctx, request.A, request.B)
	}
}

The idea is basically a simplified RPC idiom: the endpoint takes a request (interface{} because Go lacks generics), extracts some values from it, calls the service and then returns the response (interface{} again, same reason) from the service.

Since the endpoint interface accepts a single request (and returns a single response), we need to encode the input/output parameters into structs. In this case the response is just returned as is (since there is only one), but the service accepts two arguments, therefore the endpoint input will be a struct encoding those arguments (called sumRequest).

At first, it might not be obvious what the added value of this layer is, because it’s just a thin layer of glue code between transports and services. What makes it really powerful is the interface itself:

// Endpoint is the fundamental building block of servers and clients.
type Endpoint func(ctx context.Context, request interface{}) (interface{}, error)

It allows extending this layer with middleware, similar to what we saw in the service layer:

func authorized(ctx context.Context) bool {
    // authorize user here
}

func Authentication() endpoint.Middleware {
    return func(next endpoint.Endpoint) endpoint.Endpoint {
        return func(ctx context.Context, req interface{}) (response interface{}, err error) {
            if !authorized(ctx) {
                return nil, errors.New("user is not authorized")
            }

            return next(ctx, request)
        }
    }
}

func MakeSumEndpoint(service Service) endpoint.Endpoint {
	return Authentication()(makeSumEndpoint(service))
}

Middleware is more formalized in this case, but the basic idea is to separate certain business indepentent concerns from the transport, which makes this layer much more powerful than just a way of code deduplication. Rate limiting, circuit breaking, distributed tracing are all examples of what you can do in this layer.

Transport layer

The transport layer is responsible for decoding requests, calling endpoints and encoding the response for the client.

Similarly to endpoints, a “transport instance” translates to a single business call. This ensures that you can easily integrate Go kit specific transports into the transport’s own framework (eg. HTTP mux or gRPC server stubs):

Generally, a transport consists of four things:

Endpoint
Request decoder
Response encoder
Transport specific options

which looks like this in case of HTTP:

transport := http.NewServer(
    endpoint,
    requestDecoderFunction,
    responseEncoderFunction,
    options...
)

The returned transport can then be registered in an HTTP mux (eg. Gorilla).

Request encoder/response decoder functions are responsible for converting between transport and endpoint specific formats. For example, requestDecoderFunction would look something like this:

func requestDecoderFunction(ctx context.Context, r *http.Request) (interface{}, error) {
    var req sumRequest

    err := json.NewDecoder(r.Body).Decode(&req)
    if err != nil {
        return nil, err
    }

    return req, nil
}

You probably recognize the sumRequest type from the previous section.

Transport options usually provide a way to hook into requests (with before/after hooks) and allow registering custom error handlers (for transport errors).

Summary

Go kit is a toolkit that helps you write maintainable and readable applications. It comes with a set of architectural patterns and practices that makes achieving that goal possible.

I encourage you to check out the official examples and the FAQ to learn more about Go kit.

You can also check out my Go kit example application.

Fair warning

I believe Go kit is a great toolkit for building services and applications and I hope I explained the basic concepts well enough, so that you can start experimenting with it (if you haven’t already), but you know the saying:

With great power comes great responsibility

So it’s only fair if I warn you about a few things as well.

Go kit is not for prototyping

Go’s and Go kit’s philosophy is to optimize for reading code, not writing it. It means that writing the initial code probably takes more time than you are used to in a generic purpose framework for example, unless you are like a veteran Go kit user or have the tooling to generate most of the code.

This trait of Go kit often drives people away from it, because developing a service or an application occasionally starts with prototyping, which requires really fast first results and quick feedback loops. Go kit just isn’t built for that.

Go kit is not for beginners

This is purely my opinion, but I usually don’t recommend Go beginners to start learning with Go kit, because it hides a lot of basic concepts (like how HTTP or gRPC work in Go) and adds a certain level of complexity to applications that can easily distract someone who is trying to learn the language.

So if you are a beginner who is looking for a toolkit for the next project: just use pure HTTP or gRPC and move to Go kit later. If you have the chance to work on an existing Go kit application and get help from fellow developers, that’s an entirely different story though. Use that opportunity to learn Go kit as well.

Go kit is not a magic tool

Go kit’s architecture is quite simple, yet really powerful which makes it appealing for all sorts of use cases, but it’s not a one size fits all kind of toolkit. Go kit has certain opinions baked into it, like the RPC paradigm which makes it impractical for use cases that require a streaming connection for example.

Don’t try to use it for everyting, try to apply the patterns and ideas behind instead.

Watch these too

Go + Microservices = Go Kit

Peter Bourgon, the author of Go Kit talks about Go in the Cloud Native ecosystem and how Go Kit fits into it.

packagemain #12: Microservices with go-kit

A series of videos about creating services with Go Kit using various code generation tools.

Building Microservices with the Go Kit Toolkit

Another getting started video with some database examples.

go go kit application architecture getting started