Make and I were good friends for years, building software together, and while I value our friendship a lot, it’s time we move on.
Make is a build automation tool well-known in the software development industry.
It’s installed on most developer’s machines (even if they don’t use it daily)
and everyone who ever attempted to build software from source knows the infamous
make && make install command.
Make has a lot going for it. It is a simple, well understood, and powerful tool with a straightforward syntax. While you might be forgiven for thinking it’s quite a basic tool, most projects only scratch the surface of what Make is capable of. In it’s nearly 50 years, Make has acquired quite a sophisticated feature set (the documentation is 224 pages long for a reason) which makes it suitable even for projects like the Linux kernel.
So what’s wrong with Make?
Make is great…
Let me start by saying: Make is absolutely great. It automates the build workflows of thousands of projects and I’m incredibly grateful for it. But…in 2021 it’s not necessarily the best tool in the box for all projects.
Let’s take a look at a “typical” development workflow with Make. Although I primarily code in Go these days, I’ll try to keep the scenario language agnostic.
- Note: it’s impossible to cover every possible workflow here, so I picked one that I use daily.
Let’s say a new task is assigned to you: add a new feature to an existing software. You write some code and tests then run the test suite:
make test
If you’ve done everything right, tests should be green. Now let’s manually test the feature as well. For that, you need to start the application:
make run
Everything works well. Before committing the changes, let’s run the linter to make sure the code properly formatted:
make lint
Then commit and push the changes. Job is done, you can grab a beer and relax. Or can you?
Ideally, the workflow above works well for small projects with minimal or zero external dependencies on the environment. As the project gets bigger and bigger, the above workflow becomes less and less effective, leading to a bad user/developer experience.
For large projects make test can potentially run thousand of tests which takes time
and often leads to false-negative results (eg. a database is not running for an integration test).
You can use the native test tools for the given language/ecosystem,
but then you might lose some environment settings required for running the tests
(configured in Makefile). The situation gets worse if you follow TDD or test first practices.
make run suffers from similar issues in large projects.
Although most ecosystems have some sort of cache to avoid rebuilding unchanged components,
that’s not always the case. Furthermore, if a project has more than one executable component
which one do you start with make run ?
If you look closely at these issues, you will find that the root cause is that Make’s build language doesn’t lend itself to building up complexity. Modern build systems have been moving towards defining their build metadata using fully fledged programming languages, a la Gradle, which have almost fully superseded traditional build systems.
An obvious limitation of Make’s build language is the lack of modules.
Most languages have modules/packages/namespaces/etc allowing developers
to group parts of the source code into a unit that can be the target of tests,
compilation, or other development-related activities. Make doesn’t have that natively.
Technically, you can implement targets mimicking that behavior (eg. make MODULE=path/to/module test ),
you can even add a Makefile to each directory and use the -C option to build targets for specific directories,
but from a developer experience perspective, these are hard to (re)use solutions.
On the other hand, modules fall naturally out of higher level build languages.
Ultimately, Make is a generic tool with a simple interface and as such, it lacks the support and integrations for a lot of features that so-called “modern” build systems have. That doesn’t mean there is anything wrong with it, it’s simply not (always) the best tool for the job anymore.
…but Please is modern
Please (according to its website) is a “cross-language build system with an emphasis on high performance, extensibility and correctness”.
Similarly to Make, Please is a file-based build system,
with a declarative set of build rules for defining build targets.
Build targets are defined in packages,
which is simply a directory with a BUILD file in it describing the build targets for the package.
Programming languages often use the directory structure for organizing source code into packages/modules
as well so Please usually integrates with language tooling well.
The trick is that these build rules are actually written in the same Python-esqu build language used in the BUILD files.
This means that you can drop into 3GL programming whenever you need,
or define your own build rules to build up complexity without pouring through 50 years of features to find the one you need.
Please is also a so-called static build system which means that every build target
and information for building your software must be represented in BUILD files
and no (or only minimal) information is collected “runtime” during the actual building process.
These four traits of Please (static, file-based, has a notion of packages, advanced DSL) makes it a powerful build tool that lets you represent build artifacts (object files, archives, binaries, generated code, etc) and third-party modules (utilizing the respective package management tools), and handle dependencies between them. Since everything is represented in the build graph, the output of each build target can be cached, speeding up builds significantly. Caching also makes incremental builds possible: Please detects changes in source files and only rebuilds targets in the graph that have changes in their downstream dependencies.
Why Please?
When choosing a tool or solution, the first question is often “Why this one and not the other?”. You should definitely ask the question in this case as well because there are lots of alternatives to Make and Please out there. While I think Please is in a good position, it might not be the best tool for you.
In this section, I’d like to offer some guidance on how you can compare build systems and what priorities I had when I chose Please, but setting your priorities and choosing a build system is up to you.
The first step to choosing a build system is understanding the different categories of build systems. These are totally arbitrary categories made up by me, but (hopefully) they help you in your selection process.
Build systems bundled with language tooling belong to the first category.
Believe it or not, Go’s go build tool is actually a build system:
it coordinates the dependencies between packages, invokes the compiler for each package,
caches the intermediate artifacts, and finally links them together.
Sounds familiar? The advantage of these build systems is that they come with the language tooling and are well-known by the community,
so chances are new hires will already know these tools.
The downside of these tools is that they are very rarely extensible with custom targets,
so they can’t manage your entire build graph.
The second category consists of language ecosystem bound build systems that are not part of the official language tooling (for example Gradle, sbt). Both are very capable build systems and if you work on a Java or Scala project they can both serve you very well. (To be fair: they can both be used for other languages, but that’s not their primary use case) Gradle in particular is one of the most mature build systems out there.
Please belongs to the third category along with Bazel, Buck, and Pants. These build systems are language agnostic and use a python-esque DSL to describe build targets. They are very extensible, support hermetic and remote builds, and have a ton of other features to support large projects, monorepos and polyglot environments.
The fourth (and last) category contains everything else. Make belongs here together with a bunch of build systems I never heard of.
Since I usually work on Go projects, often using generated code and embedding static template files,
I decided to pick the third category. For a Java project,
I’d probably choose Gradle, while for a simple, single-package Go library I’d stick to go build.
Choose the one that fits your needs.
The next step is comparing the actual tools within the same category. I actually played with all four of them (Bazel, Buck, Please, Pants in this order) and settled on Please in the end.
I started with Bazel, because it was the first one, the other three are actually based on Bazel. It has a rich ecosystem with users like Google (they actually created Bazel) and Kubernetes. While Bazel is quite mature, it requires Java (which is still a pain to install in 2021), and writing custom rules is quite complex at first sight.
Buck is just a Bazel clone maintained by Facebook, but it’s not that mature and from what I heard, it’s quite specific to Facebook’s needs, so I’m not even sure why they open-sourced it.
I briefly looked at Pants, but to me, it didn’t seem any better than Please. There is also this.
So I decided to go with Please. Compared to the other three, Please has a number of advantages that appealed to me from the very beginning:
- It’s written in Go, so I can easily contribute to it and the executable is a static linked binary (no JVM required)
- It’s fast (faster than Bazel)
- It’s very easily extensible
- You can literally ask it to do things, like
$ please build(Seriously, how cool is that name?)
Plus as I spent more time with Please I got in touch with the developers and they are really awesome people, helping the community whenever they can.
But….
When talking about technology, especially new and exciting technology, people often forget a very important rule: Everything comes at a price. Introducing Please to an existing project is no exception from that rule.
To prove that, here is the timeline of introducing Please to one of the largest projects I’m working on these days: I started to look for an alternative to Make about a year ago. Deciding to use Please took two months and I finished migrating the whole build process to Please last week.
During that time I’ve sent numerous patches to Please and worked closely with the developers to add Go modules support. It’s still quite new, and it will take some time until it becomes mature enough, but it supports the most common use cases.
Another thing that I needed to adjust to is the build language and writing BUILD files in particular.
I was used to the fact that go build can just collect everything it needs from the source code.
Fortunately, there is a third-party tool, called Wollemi
that can automatically generate BUILD files for Go projects.
(It would be nice to see in Please itself)
The point I’m trying to make: introducing a new build system requires a lot of time and patience. Fortunately, the team behind Please is awesome and they support the community whenever they can.
Conclusion
I’m pretty sure our friendship with Make is far from over. It’s still a great tool, but Please has a lot of features that make the software development experience better, especially for large projects.
This post is only a short introduction to Please, explaining some of the advantages and differences it has compared to Make and other build systems. I didn’t want to go into too many details, because that would have made this post much longer. However, I’m going to write a few follow-up posts demonstrating development workflows with Please for Go and Kubernetes, so stay tuned.
Make sure to check out the Please website to learn more or jump straight to the codelabs to get started. If you are interested in an example Go project, check out this one.
Links
Build Systems à la Carte by Simon Peyton-Jones: A wonderful presentation about build systems with a much more scientific classification than mine. (Read the paper as well.)