TwirPHP: A modern RPC framework for PHP

In the last couple years RPC started to become popular again as a communication mechanism for web-based APIs. It’s not a new concept at all, but it changed a lot over the years: the technology evolved, new IDLs and frameworks (like protobuf and gRPC) appeared. TwirPHP tries to bring this new tech to PHP by porting Twirp, Twitch’s “simple RPC framework built on protobuf”.

What is RPC?

RPC (Remote Procedure Call) is a design paradigm based on request-response message-passing where two different parties (client-server) communicate over a channel (usually network). It started off as a synchronous request-response communication pattern, but it changed a lot over the years. Nowadays modern RPC frameworks are language-agnostic, provide asynchronous, bidirectional streaming mechanisms and a bunch of other features (like load-balancing, authentication and authorization, etc).

Despite the changes in technology and the concept, the basic idea is simple:

Let’s take a function (procedure) which has some input parameters (request) and return values (response)
Expose the function over some protocol (eg. HTTP) to the outside world
Implement the function again (with the same input/output parameters) calling into the remote one (remote procedure)
Call the reimplemented function which will call the remote one (remote procedure call)

RPC Diagram

Why not REST?

Undoubtedly REST is the dominant solution for web-based APIs these days. Before it became popular though, APIs were mostly built using (primarily XML based) RPC frameworks, like XML-RPC or SOAP. However, there was a bit of a problem with XML: lack of data types. In XML most of the data is string, so in order to support types, RPC frameworks added an extra layer of meta data which made the protocol much more complex.

Compared to this, REST offered a loosely typed payload which meant a way out for many.

But the truth is that many web-based APIs labeled as REST are actually RPC-style APIs with RESTish characteristics. While REST’s concepts of data representation and communication mechanism certainly fit many scenarios (and are sometimes easier to understand), it’s not superior to RPC, it’s just different. More importantly, it solves a different problem than RPC. REST is great for modeling your domain and exposing it under a CRUD-style interface, but RPC is the way to go if your API primarily consist of executable actions (procedures).

Let’s take a look at an example: imagine a bookshelf application for e-books.

Adding a book might look something like this in a REST API:

POST /books HTTP/1.1
Host: api.example.com
Content-Type: application/json

{
    "title": "Harry Potter and the Philosopher's Stone",
    "author": "J.K. Rowling",
    "formats": {
        "mobi": "https://example.com/books/hpps.mobi"
    }
}

It’s pretty simple. Now lets add support for converting a book from one format to another. Using REST this is quite problematic, but there are a number of solutions we could choose. We could try to change the book resource itself:

PATCH /books/1 HTTP/1.1
Host: api.example.com
Content-Type: application/json

{
    "formats": {
        "epub": "from://mobi"
    }
}

The server should have to notice the special URL format and start a conversion in the background. This is a little bit problematic though. Where is the state stored in the meantime? How to track the progress?

Alternatively, we could do something like this:

POST /books/1/formats/epub HTTP/1.1
Host: api.example.com
Content-Type: application/json

{
    "fromFormat": "mobi"
}

This is somewhat better, because we can treat the new format as a separate resource, track it’s progress without leaving the book resource itself in an inconsistent state.

Now let’s see an RPC solution:

POST /books/convert HTTP/1.1
Host: rpc.example.com
Content-Type: application/json

{
    "id": 1,
    "from": "mobi",
    "to": "epub"
}

Most of the time we can find a solution that somewhat fits into REST, but it’s not always possible and not always perfect. In any case, using RPC for actions expresses the intent better and makes the API easier to understand. Furthermore, the above action can directly be mapped to a function call:

<?php

function convert(int $id, string $from, string $to): void {
    // real implementation or an HTTP call
}

This example not only shows the difference between REST and RPC style APIs, but also shows that the two are not mutually exclusive.

Modern RPC frameworks

RPC frameworks came a long way and adapted to technology needs over time. First and foremost: new IDLs (Interface Definition Language) were created with binary format for serialization (like Protocol Buffers or Apache Thrift). This makes them not only faster, but thanks to their language independent protocol specifications they are also language agnostic which is a real advantage in polyglot systems.

In addition, it’s also common these days that RPC frameworks have builtin support for secure communication, authentication, load balancing and a bunch of other things, often based on the underlying protocols (eg. HTTP).

One of the most popular RPC frameworks today is gRPC. It’s an open source RPC framework, initially developed by Google. It uses HTTP/2 for transport, Protocol Buffers as IDL and provides features such as authentication, bidirectional streaming, cancellation, timeouts, etc. Well-known projects, like Kubernetes, etcd and many more use it for their APIs.

Another popular framework is Apache Thrift which is actually both an IDL and an RPC framework. It was developped by Facebook, but it’s now an open source project in the Apache Software Foundation.

Introducing Twirp

Twitch released it’s own RPC framework Twirp a little over a year ago. It was designed to be an alternative to gRPC, primarily for internal usage, but seeing how quickly their teams adopted it, they decided to open source it. In their introductory post (which I really recommend you to read) they explain in detail why they decided to implement their own RPC framework instead of just using gRPC.

The most important differences between Twirp and gRPC are HTTP 1.1 support and JSON serialization (in addition to protobuf) which really proves to be useful during development and makes the framework easier to understand. Twirp also comes with a thinner runtime library and relies on the generated stubs instead. This was a design decision based on past bad experience with incompatibilities and breaking changes between different gRPC versions.

Twirp’s simplicity and lightweight nature comes at a price though: it lacks features, like client side load balancing and bidirectional streaming (at the time of writing, but it’s work in progress) so for those who rely on these advanced features Twirp might not be the right tool, but it should be enough for most of the use cases of web-based APIs.

TwirPHP: Twirp ported to PHP

Given gRPC’s popularity, Twirp might not be the first candidate for porting. Unfortunately, gRPC’s HTTP/2 requirement makes implementing the server side non-trivial. However, the more time I spent porting Twirp the more I became confident that it’s probably a better choice after all.

I already glorified Twirp for its JSON and HTTP 1.1 support. Given the primary use cases of PHP and that those use cases usually involved RESTish APIs so far, it’s much more easier to transition to a new API with similar characteristics. Also, PHP might not be the first choice for applications using bidirectional streaming, so the lack of it is probably not a problem. Ultimately, I believe that porting Twirp to PHP was much easier than porting gRPC would have been, thanks to its simplicity.

The latest version of TwirPHP is 0.5.1 at the time of this writing. It received a major update recently, switching to PSR-15, PSR-17 and PSR-18 from HTTPlug and dropped support for PHP 5.x.

Example

Talk is cheap, let’s see some code.

(TL;DR: You can find the end result here.)

As a first step, please make sure the following components are installed in your environment:

Next you will have to decide how you want to install the Protobuf library:

By installing the C extension
Via Composer, installing the PHP implementation

The extension is obviously faster, but at the time of writing it hasn’t been tested on PHP 7 and Windows/Mac, so for better compatibility, I will use the PHP implementation in the example.

Note: The PHP protobuf implementation needs the bcmath extension to be installed to work with JSON.

Let’s create the project and install the necessary dependencies:

mkdir example
cd example
composer init --name twirphp/demo --type project --description "TwirPHP demo project" --no-interaction
composer require twirp/twirp google/protobuf php-http/guzzle6-adapter http-interop/http-factory-guzzle

The next step is creating the protobuf definition for the API. For now I will just use the one from the original Twirp documentation:

syntax = "proto3";

package twirp.example.haberdasher;
option go_package = "haberdasher";

// Haberdasher service makes hats for clients.
service Haberdasher {
  // MakeHat produces a hat of mysterious, randomly-selected color!
  rpc MakeHat(Size) returns (Hat);
}

// Size of a Hat, in inches.
message Size {
  int32 inches = 1; // must be > 0
}

// A Hat is a piece of headwear made by a Haberdasher.
message Hat {
  int32 inches = 1;
  string color = 2; // anything but "invisible"
  string name = 3; // i.e. "bowler"
}

Let’s create it as proto/haberdasher.proto. Use the Protobuf compiler to generate the necessary code from the definition:

mkdir -p generated
protoc -I . --twirp_php_out=generated --php_out=generated ./proto/haberdasher.proto

The generated code includes a service interface (generated/Twirp/Example/Haberdasher/Haberdasher.php) which should look something like this:

<?php
# Generated by the protocol buffer compiler (protoc-gen-twirp_php 0.5.1).  DO NOT EDIT!
# source: proto/haberdasher.proto

declare(strict_types=1);

namespace Twirp\Example\Haberdasher;

/**
 * Haberdasher service makes hats for clients.
 *
 * Generated from protobuf service <code>twirp.example.haberdasher.Haberdasher</code>
 */
interface Haberdasher
{
    /**
     * MakeHat produces a hat of mysterious, randomly-selected color!
     *
     * Generated from protobuf method <code>twirp.example.haberdasher.Haberdasher/MakeHat</code>
     *
     * @throws \Twirp\Error
     */
    public function MakeHat(array $ctx, \Twirp\Example\Haberdasher\Size $req): \Twirp\Example\Haberdasher\Hat;
}

This is the interface which we will have to implement (src/Haberdasher.php) and pass to the server stub later:

<?php

namespace Twirp\Demo;

use Twirp\Example\Haberdasher\Hat;
use Twirp\Example\Haberdasher\Size;

final class Haberdasher implements \Twirp\Example\Haberdasher\Haberdasher
{
    private $colors = ['golden', 'black', 'brown', 'blue', 'white', 'red'];

    private $hats = ['crown', 'baseball cap', 'fedora', 'flat cap', 'panama', 'helmet'];

    public function MakeHat(array $ctx, Size $size): Hat
    {
        $hat = new Hat();
        $hat->setInches($size->getInches());
        $hat->setColor($this->colors[array_rand($this->colors, 1)]);
        $hat->setName($this->hats[array_rand($this->hats, 1)]);

        return $hat;
    }
}

In order for the generated code and our implementation to work, we have to add them to the Composer autoloader:

{
    "autoload": {
        "psr-4": {
            "Twirp\\Demo\\": "src/",
            "": ["generated/"]
        }
    }
}

Make sure to dump the autoloader:

composer dump-autoload

The last step is wiring everything together. Since this is highly application dependent, I will just use the simplest possible example here (server.php), but you could use whatever router, message implementation, application environment you want:

<?php

require __DIR__.'/vendor/autoload.php';

$request = \GuzzleHttp\Psr7\ServerRequest::fromGlobals();

$server = new \Twirp\Server();
$handler = new \Twirp\Example\Haberdasher\HaberdasherServer(new \Twirp\Demo\Haberdasher());
$server->registerServer(\Twirp\Example\Haberdasher\HaberdasherServer::PATH_PREFIX, $handler);

$response = $server->handle($request);

if (!headers_sent()) {
	// status
	header(sprintf('HTTP/%s %s %s', $response->getProtocolVersion(), $response->getStatusCode(), $response->getReasonPhrase()), true, $response->getStatusCode());
	// headers
	foreach ($response->getHeaders() as $header => $values) {
		foreach ($values as $value) {
			header($header.': '.$value, false, $response->getStatusCode());
		}
	}
}
echo $response->getBody();

The generated client will already implement the service interface, we just have too use it (client.php):

<?php

require __DIR__.'/vendor/autoload.php';

$client = new \Twirp\Example\Haberdasher\HaberdasherClient($argv[1]);

while (true) {
    $size = new \Twirp\Example\Haberdasher\Size();
    $size->setInches(10);

    try {
        $hat = $client->MakeHat([], $size);

        printf("I received a %s %s\n", $hat->getColor(), $hat->getName());
    } catch (\Twirp\Error $e) {
        if ($cause = $e->getMeta('cause') !== null) {
            printf("%s: %s (%s)\n", strtoupper($e->getErrorCode()), $e->getMessage(), $cause);
        } else {
            printf("%s: %s\n", strtoupper($e->getErrorCode()), $e->getMessage());
        }
    }

    sleep(1);
}

All we have to do now is start the server and run our client:

php -S 0.0.0.0:8080 server.php
# in a different shell
php client.php http://localhost:8080

You should see a client output like this:

I received a black flat cap
I received a brown helmet
I received a golden fedora
I received a brown fedora
I received a black flat cap
I received a brown fedora
I received a white baseball cap

To sum up:

We defined our API using Protocol Buffers
Generated most of the code
Implemented a single interface
Wired the implementation into a simple application using standard interfaces

It’s that simple.

You can find the code for this demo here and more information about getting started in the documentation.

Conclusion

Thanks to it’s simplicity Twirp is a perfect fit for most web-based use cases. It’s easy to transition to a Twirp-based API from any previous RESTish solution. By using open HTTP standards, TwirPHP is easy to integrate and use in any kind of projects.

Stability note

As of 0.5.0 TwirPHP is considered to be stable. The API will not change, unless serious flaws are found. The structure of the generated code might change but it shouldn’t affect existing applications. After a few months of beta period the first stable version (1.0.0) will be tagged.