Ephemeral Environments

Until this year, every Axle engineer had their own permanent development environment. We replaced them with on-demand environments built on ECS, Neon, and infrastructure we already had. Engineers can now work on several features in parallel, product can test before merge, and our non-production AWS bill dropped about 15%.

Here's how we did it, and what we got wrong along the way.

One environment per engineer stopped working

Since Axle's earliest days, every engineer had their own deployed development environment. That made sense when the team was small, but as we grew, the limitations became harder to ignore.

Each environment was effectively a complete copy of our platform: its own ECS cluster, frontend deployment, database, and dozens of SQS queues, Lambdas, and S3 buckets. Spread across three Pulumi stacks, that came to more than 200 AWS resources per engineer.

The infrastructure overhead was significant, but the bigger issue was that our environments had started to dictate how engineers worked, rather than support the way they needed to work.

• One feature at a time. If your environment held a feature waiting on review, you couldn't deploy the next one. This was the most common complaint on the team.
• Drift. No two environments behaved quite the same. Product couldn't tell a real bug from a stale stack, so they stopped testing in them and waited for changes to reach dev.
• Upkeep that grew with headcount. Every new engineer meant another stack to back-merge after each release and repair when it broke.

What blocked us from switching earlier

We knew our existing model wasn't going to scale, and we had a fairly clear picture of what should replace it: environments provisioned on demand, fully isolated databases, and as little ongoing maintenance as possible.

For a long time, we assumed Kubernetes was the natural answer. Its orchestration and service discovery seemed purpose-built for exactly this problem. But migrating our ECS infrastructure to Kubernetes would have added a level of complexity that was hard to justify at our scale.

When we broke the problem down, it came to two questions:

1. How does a new backend become reachable without someone creating DNS records?
2. How does each environment get its own database without a slow dump-and-restore?

We spent months on these questions before realizing we didn't need Kubernetes to solve either one. We needed a handful of capabilities we could assemble from infrastructure we already had.

Key breakthroughs

The wildcard DNS record

A wildcard DNS record isn't new, but pairing one with hostname-based routing on a single shared Application Load Balancer (ALB) gave us dynamic routing without a service mesh or an orchestrator.

For each environment, we add a target group pointing at its ECS task and a listener rule on the shared ALB that matches its hostname. The wildcard DNS record does the rest, sending every environment's subdomain to that same ALB.

The result: isolated environments, shared infrastructure, and zero DNS changes as environments come and go.

Copy-on-write database branching

Giving every environment its own database sounds great in theory. On Aurora, the economics fall apart: a new cluster takes minutes to provision, incurs compute costs, and leaves you paying for infrastructure that might only exist for a few hours. For environments that spin up and tear down automatically, that isn't practical.

Enter Neon and its copy-on-write database branching.

Neon is built on Postgres, but it separates compute from storage, which makes near-instant branching possible. Rather than physically copying a database, a new branch references the same underlying data as its parent. It only stores its own version of a page once that page changes.

That means we can fork a database containing gigabytes of data in about a second, without duplicating the dataset or provisioning a dedicated database server up front. Each branch gets its own isolated state, while the underlying storage stays shared until something changes.

The next challenge was keeping those branches populated with realistic, up-to-date data. We maintain a permanent Neon parent branch called seed-dev, refreshed nightly from our development Aurora database through a three-stage pipeline:

1. Fargate. A scheduled ECS task runs pg_dump against our development Aurora database and uploads the snapshot to S3.
2. CodeBuild. A CodeBuild job pulls the latest snapshot and restores it into seed-dev, keeping our baseline dataset current.
3. Neon branching. Whenever an environment is created, our provisioning workflow forks seed-dev into a dedicated database branch for it.

We pay the cost of copying and restoring our development database once per refresh, not once per environment. Every environment after that branches from the prepared snapshot almost instantly.

Because each environment has its own branch, developers can run migrations, modify records, and test destructive operations without affecting other environments or the shared baseline. When an environment is torn down, its branch is deleted along with the rest of its infrastructure.

The result is exactly what we were after: realistic, isolated Postgres databases provisioned in seconds, with storage costs driven by changes rather than full copies of the dataset.

One frontend deploy, any backend

Our frontend used to bake in its API URL at build time, so testing a backend change through the UI meant asking for a custom frontend build. Most backend engineers skipped testing through the UI.

The fix was simple: move backend selection from build time to runtime.

The dev frontend still defaults to the shared dev backend, so nothing changed for anyone who didn't need it. A new input field lets engineers point their session at any ephemeral backend. No new build, no redeploy, and no coordination with another team.

A backend engineer can now take a pull request from code to a full end-to-end product test without touching the frontend or involving a frontend engineer.

One frontend deployment. Any backend. Zero additional builds.

What we got wrong

We tied environments to pull requests. Our first version created an environment for every PR and destroyed it on merge. That produced dozens of environments nobody actually needed. It also broke longer projects: an engineer who wanted to keep test data across several PRs lost it every time one merged.

An environment's lifecycle isn't a PR's lifecycle.

We switched to an on-demand GitHub Actions workflow: name the environment, pick a branch from each repo, deploy, and tear it down when you're done. It's one more manual step, but it removed far more wasted work than it added.

We turned off every cron. We have dozens of scheduled jobs. Running all of them in every environment would have multiplied our load on third-party APIs, so we launched with nearly all of them disabled. That was safe and also made a lot of features impossible to test. Over the next few weeks we turned jobs and integrations back on per environment, only where someone needed them.

It took about two weeks of this after launch before the environments were stable. Fixing problems as the team ran into them beat waiting for a design that was perfect on paper.

Results

• Engineers work on several features in parallel instead of queueing behind their own environment.
• Product can test features before merge again.
• Migrations and destructive tests are cheap to retry.
• Non-production AWS spend is down about 15%, net of Neon. Each retired personal environment took two load balancers, an ECS cluster, a CloudFront distribution, and a WAF with it.

---

The broader lesson: we spent months treating this as a platform choice when it was really a systems design problem. Once we broke it into its real requirements, we could pick the right tool for each one, and the hard engineering was making them work together as one system.