Templating Terraform using Python and Jinja2

Have you ever wondered how one can dynamically generate Terraform templates to automate repeating tasks so that the only thing that really needs to be given manually are variables / data? In this article we will discuss programming approach to this problem.

THE STANDARD APPROACH

To discuss the problem, let’s imagine a simple case when one wants to configure a virtual server on BIG-IP device using official F5 BIG-IP Terraform provider. The basic VS configuration template may consist of two files

• provider.tf – static template that configures provider and credentials,
• vs.tf – template that configures a Virtual Server configuration.

provider.tf:

variable hostname {}
variable username {}
variable password {}

terraform {
  required_providers {
    bigip = {
      source = "terraform-providers/bigip"
    }
  }
  required_version = ">= 0.13"
}

provider "bigip" {
  address  = var.hostname
  username = var.username
  password = var.password
}

vs.tf

resource "bigip_ltm_virtual_server" "http" {
  name        = "/Common/terraform_vs_http"
  destination = "10.12.12.12"
  port        = 80
  pool        = "/Common/the-default-pool"
}

Templates shown above simply configure HTTP Virtual Server on BIG-IP with most of the values left as defaults.

Each time one wants to configure a new Virtual Server, one should create a new “vs-xx.tf” file. Each file then contains configuration for a given VS. Alternatively, one can append new VS configuration to the “vs.tf” file each time there is a need for new configuration.

No matter which of the above approaches is taken, the same problem arises – each addition of configuration requires manual copy-paste of configuration with little to no changes (since one only needs to change parameters that actually differ or those that must be unique). This reasoning leads us to the second paragraph which discusses how such configurations can be automated. By the end of this article, the reader should be able to fully automate creation of repeating templates using Python and Jinja2.

THE AUTOMATION APPROACH

To automate the process of creating new configuration we will use Python and Jinja2. This duo is powerful when it comes to templating which is exactly what is needed in this scenario.

Initial installation

The only requirement to follow this guide is to have Python3 installed.

Then, one needs to install Jinja2 package:

pip3 install Jinja2

Code

After installing all of the requirements, one can start to develop Python code that will automate templating of Terraform.

First, let’s create a “main.py” file in the application’s directory:

- application_folder:
  - main.py

Then, let’s prepare main.py file and create a function inside main.py that will generate templates:

import os
from jinja2 import Environment, PackageLoader, select_autoescape

jinja_env = Environment(
    loader = PackageLoader("main"),
    autoescape = select_autoescape()
)

def generate_template(_template, _data):
    return _template.render(_data)

The following libraries are imported:

• os – aids in saving file to correct directory,
• jinja2 (Environment, PackageLoader, select_autoescape) – used for Template generation.

The “generate_template” function simply takes template loaded by Jinja and some data in a form of Dictionary and renders them into a template that can be written to file.

Now, let’s prepare templates which will be loaded.

First, let’s create a folder named “templates” in the root of the application and create an empty jinja2 file inside this subfolder:

- application_folder:
  - main.py
  - templates
    - vs_template.jinja2

This “vs_template.jinja2” file contains a template of the Virtual Server configuration that was discussed previously. Jinja2 file should contain exactly the same template with one difference – each parameter value that one wants to automate creation of, should be changed to “{{ name_of_parameter_data_given }}”. Jinja2 will then change this into an actual value that the user passed to the function.

Let’s look at an example:

Suppose that this is a part of some Terraform template (and disregard the fact that it is not a correct template on its own):

resource "bigip_ltm_virtual_server" "https" {
  name                       = "/Common/terraform_vs_https"
  destination                = var.vip_ip

Now, as discussed previously in the manual approach section, each time a use would like to add new resource of such type he would copy this, paste it somewhere else and manually change the values.

With Jinja2 approach we only need a single template:

resource "bigip_ltm_virtual_server" "https" {
  name                       = "{{ vs_name }}"
  destination                = "{{ vs_ip }}"

with values changed to parameter values given by the user which Jinja2 will change during template generation. If the user gives the following parameters (vs_name = “test”, vs_ip = “10.20.30.40”) then the rendered Template will look like this:

resource "bigip_ltm_virtual_server" "https" {
  name                       = "test"
  destination                = "10.20.30.40"

Now, after the example cleared the idea behind Template generation, let’s move the Terraform template from the previous section into Jinja2 file:

“vs_template.jinja2”:

resource "bigip_ltm_virtual_server" "{{ resource_name }}" {
  name        = "{{ vs_name }}"
  destination = "{{ vs_ip }}"
  port        = 80
  pool        = "/Common/the-default-pool"
}

The template file is ready, so let’s go back to main file and add the rest of the code:

main.py now looks like this:

import os
from jinja2 import Environment, PackageLoader, select_autoescape

jinja_env = Environment(
    loader = PackageLoader("main"),
    autoescape = select_autoescape()
)

def generate_template(_template, _data):
    return _template.render(_data)

if __name__ == "__main__":

    # Generate template

    vs_template = generate_template(
        _template = jinja_env.get_template("vs_template.jinja2"),
        _data = {
            "resource_name": "vs-01",
            "vs_name": "automate",
            "vs_ip": "10.20.30.40"
            }
    )

    # Write template to file

    with open(os.getcwd() + "/example.tf", "a") as fd:
        fd.write(vs_template)

One can run this code and should get the following “example.tf” Terraform template:

example.tf:

resource "bigip_ltm_virtual_server" "vs-01" {
  name        = "automate"
  destination = "10.20.30.40"
  port        = 80
  pool        = "/Common/the-default-pool"
}

From there it can easily be taken further – one can add user input to dynamically assign parameter values or one can create loops etc to quickly create new templates.

SUMMARY

Hopefully this article will shine a light at how one can actually automate Terraform template creation and simnifically decrease amount of copy-paste operations that administrator would normally need to perform each time new configuration was to be created.