Chicken or the Egg? Terraform’s Remote Backend

Recently, we have decided to expand our DevOps stack with the addition of Terraform for creating Infrastructure as Code manifests. It became obvious from the start that local backend is not an option, so we had to set up a remote one.

Here’s the tricky part—how do we manage the infrastructure for a remote backend or, in other words, how do we solve the chicken or the egg problem?

Unfortunately, the Terraform docs didn’t give us a clear answer. Likewise, the Internet was not much help in that respect—most people simply recommended that we create a base infrastructure by hand. However, together with Robert Wysocki, our DevOps magician, we decided to avoid that particular approach, and set out to find a more portable solution—one that we would now like to share with you here. 

Our first thought was to use the -backend=false flag in order to temporarily disable the remote backend and reuse single manifest twice:

• first for setting up the infrastructure using the local state
• later for migrating it to the remote one without -backend=false

Unfortunately, this didn't work for us as it looked like the flag simply... did nothing.

So we decided to go with a directory layout like the one outlined below:

• a common directory with backend configuration; reused between separate workspaces acting as environments
• a setup directory containing manifests for setting up the backend infrastructure
• a base directory that symlinks all manifests from setup and additionally has remote backend defined
• and, finally, backend.tfvars with configuration variables for the backend

On their own, manifests do not do anything special. The main file configures the AWS provider and our backend module is responsible for backend infrastructure creation.

Another issue we had to deal with was connected to the backend configuration itself—it is impossible to interpolate variables inside the configuration, which means we couldn’t do something like this:

Ultimately, we had to hardcode the configuration into the backend block itself. That meant duplicating the same variables for our backend module (responsible for the creation of the S3 bucket, DynamoDB, and other components).

Thankfully, Terraform allows defining a partial backend configuration while providing all the missing arguments using the -backend-config option by means of a separate file utilizing the same format as tfvars. This enabled us to reuse the variables file in both the backend configuration as well as creating the backend infrastructure! The only thing we have to remember now is to pass the variables on each terraform init and terraform apply when running manifests that touch backend infrastructure, making the commands look something like this:

• terraform init -backend-config=backend.tfvars
• terraform apply -var-file=backend.tfvars

Finally, we wrapped the flow in a simple bash script:

For our example, we chose what is probably the simplest solution—an S3 remote backend that doesn't require any additional provisioning. Such an approach should also work with different backends; however, it might require some additional effort on your part, eg. for consul, you have to somehow provision consul servers.

For more details please review the repository itself and, as always, we are eager to hear your comments!