The Golden Path and the Branching Path

This is the first of a five-part series on the Branchstack, the approach behind Platter's latest Branching Postgres product. Read part two here.

Platter was born during a breakfast of hot-sauce-drizzled chorizo and eggs 🤤. Idle banter turned to rapid-fire complaints and proposed solutions as we wondered aloud why getting software to users reliably was so difficult—much more so than writing that software in the first place. We soon realized that we were ideal partners to take on this problem together, and we've been dreaming up and working hard on solutions ever since. But our approach has changed drastically based on things we've learned along the way.

In this post, we'll talk about the problem of software delivery in general, how we built a Golden Path to solve that problem, and how Platter now instead supports any number of shortened development and deployment paths, with a set of software development principles and practices that we call the Branchstack. In following posts, we'll dive into the details with deeper analysis and real-world examples we and and our customers have encountered.

The Problem

The Problem was clear from that first conversation: getting software projects from a dev's computer to their intended audience is too difficult. Whether those projects are command-line tools, static sites, desktop applications, stand-alone APIs, or anything in between, too much time is spent deploying and instrumenting applications instead of creating and iterating on them.

"Glue code" hardens quickly: ad-hoc tool adoption and platform engineering fixes environments and deployment processes in place too early, before teams even know if they are solving the right problems for the right users. Meanwhile, bugs dependent on conditions in an increasingly jury-rigged deployment pipeline and production environment surface far too late, reported by confused users.

We saw the Problem affect developers of all skill levels and teams of every size and shape. And we had a few ideas on how to solve it.

The Golden Path

If you're looking to solve the Problem, then it makes a lot of sense to start focusing on the path that application code takes from a developer's machine to the public cloud to end-users. Platter's first approach was to enumerate those steps and their possible variants, pick a few variants that represented the best-practices of development at the time, then integrate those steps into a single, blessed workflow. The thought was that developers would happily use our Golden Path if it were convenient enough.

This is the same kind of bargain you see with frameworks like Ruby on Rails: if you do things the Rails Way™, then you get a set of tools that abstract over the Hard Parts™ of web development. The success of Rails and Rails-esque frameworks is a testament to the popularity of this approach. In a way, Platter's first approach was to build the Rails of Infrastructure.

When thinking about the Golden Path for a "standard" web application at the time, we identified a few common and critical steps:

Learn

The first step of any project is to learn about the tools needed to build anything useful in the first place. We constantly thought about how we could either provide tools to help users learn what they needed to know about building "real world" applications or automate deeper concepts well enough that users never had to learn those concepts at all. An example of this approach was our use of containers through every environment while making sure that our users never had to learn about Docker or containerization at all.

Create

Once you know what tools to use and what you want to build, it's time to create a new project. For the applications on the Golden Path, that meant creating a single monorepo with a React-based frontend, an Express-based backend, and a Postgres database paired with a built-in migration framework. All of this was done through a dedicated CLI. That's right, Platter was templating projects before it was cool.

Build

The monorepo format meant that we could spin up a containerized development environment for every project. This allowed a quick feedback loop for local development from day one, with pre-configured file watchers to make sure that the frontend, backend, and database were always hot-reloaded in lockstep.

Deploy

Like many other deployment platforms, we leveraged git and GitHub to trigger builds and rollouts for these application bundles. Because we were working with databases, we did our best to provide coordinated rollouts for database migrations, then backends, then frontends. Here's what the Create, Build, and Deploy phases looked like in practice:

Bonus feature: Patrick's feverish 3AM optimism, mentioning all the other Golden Paths we were also going to pave 😅.

Operate

Once a project was deployed, it was Platter's job to provide an interface for making sure that the pieces that made up that application were operating like they were supposed to. We had grand ambitions (mostly unrealized) of providing Grafana-powered dashboards to all of our users and exposing the configuration knobs necessary to make sure that their services were handling their workloads appropriately.

Did it Work?

Not really. While this Path was truly Golden for a small subset of users, it was both too narrow and too wide for most.

It was too "narrow" in the sense that the entire development process needed to stay on the Path. If someone wanted to use serverless functions instead of hosted Express, or if they wanted to use a different template or layout for their project, or if they had trouble enabling a container runtime on their local machine, they were out of luck until we could bake that option into the Path.

It was too "wide" in the sense that simple and complex steps alike were included in the Path, and no distinction was made between the two in pursuit of a single, all-encompassing workflow for every Platter-driven application. We automated simple things that had better solutions elsewhere, like repo creation, making incorporation into existing projects and workflows difficult.

The Golden Path approach meant that the cost of starting down the Path was high, and the cost of deviating from the proscribed Path later was even higher.

What We Learned

While paving the Golden Path did not solve the Problem to the degree that we had hoped, it also meant that we had spent more time than just about anybody else thinking about the development process holistically and gathering data on where that process tended to fail. Rarely could one walk the Golden Path without a guide close-by, so we spent hours working directly with users of all skill-levels trying to take their ideas from inception to production. We also wrote thousands of lines of code solving different problems at each step in the general case, which makes it easier in hindsight to categorize parts of these steps according to the difficulty users had progressing to the next step, the likelihood of something going wrong during that step, and the frequency with which each step needed to be repeated during the development process.

We had spent a lot of time automating parts of the process that were sometimes tedious, but often so infrequent that they weren't worth the time it took to incoporate them into The Golden Path. This includes repository setup, project structuring, dependency management, and local development watch loops. While these steps can sometimes be frustrating for new users, they're a comfortable and reliable part of development for experienced users and are tackled by templates, generators, and even copy-and-pastes of basic configurations everywhere. Even our efforts to lower the learning barrier proved to be less fruitful than we had hoped: there was no escaping learning about git, basic language proficiency, dependency management, command-line tooling, or even the containers that we tried so hard to abstract from users, and that learning usually only had to happen once to unblock that user forever.

Where most applications got into real trouble (the exact kind that the Golden Path was built to prevent) was in the difficult, variable, and frequent steps. Those included:

1. migrating production database schema (and handling rollouts and rollbacks in case of failure)
2. coordinating rollouts between databases, APIs, and frontends
3. operating stateful cloud-based infrastructure in production over time (whether that "state" was data or resource configuration)
4. replicating "production" between environments (usually development and some kind of staging environment)

As it turns out, #4 isn't just hard, but impossible when talking about local environments: you cannot replicate your production environment locally. And while that may seem obvious (your laptop is hardly a GCP or AWS datacenter), as the gulf between what we could reproduce locally and what was running in production grew wider for our users, so did the incidence of critical bugs.

There was simply no way to pave a path with enough gold to make it infallible when we include all of the steps from developer machine to staging environments to production. Which led us to our next approach: instead of paving one blessed path to production, why don't we build tools that can shorten all of them?

Branchstack

One of the reasons we have to have all of these discrete steps in the first place is because there is a fundamental mismatch between the way that developers work on a codebase and the way that they work with data and stateful infrastructure. Code is version-controlled, quickly copied, compiled, and discarded as the needs of the project change. In contrast, databases and other stateful infrastructure are treated as precious "pets" that live in their own one-off environments, rarely to be copied or configured directly.

Platter Postgres (the first Branchstack-focused product built by the Platter team) uses a different model. Instead of paving over each step, Platter Postgres eliminates the hardest steps thanks to its cloud-only, serverless, branching model. We're already excited with the traction this service has been getting, as it was explicitly designed to sidestep the most painful parts of the Golden Path. We think that branching databases will fundamentally change the way we build software, and we're excited to start with Postgres.

But we also want to look past Postgres towards an entire ecosystem built on Branchstack principles. Over the next week, we'll be releasing more in-depth articles on the key Branchstack tenets of Environmentless, Serverless, and Branching.

If you have any questions about this introduction to the Branchstack or our experiences building software without it, please send us an email at [email protected] or join us on the #kitchen Slack channel.