Terraform support for job templates and output presets of AWS Elemental MediaConvert

AWS Elemental MediaConvert is a media transcoding service of AWS. For supporting vast VOD service, MediaConvert is very useful and reliable. Terraform is an Infrastructure as Code (IaC) tool which supports multiple Cloud platforms like AWS, Azure, and GCP. Unfortunately due to a lot of configurations of MediaConvert, terraform does not support MediaConvert resource and settings management yet. In this tutorial, we are going to take a look at how can we achieve this.

MediaConvert Job Templates

The job template is a part of MediaConvert. If we create some templates with input(s) and output(s) of transcoding, we can use that for every time to transcoding some files. Let’s say your platform needs 3 types of video outputs with different bitrate and resolution. So we can save that template and reuse for the next transcoding jobs. There is another section called Output presets in MediaConvert console. These preset store output configurations of transcoding jobs. And in the template, we can use one or more output presets.

Setting the Goal

So our plan is to manage MediaConvert templates with terraform as terraform does not support MediaConvert. So we will create templates and output presets at the time of Terraform apply command and destroy with Terraform destroy command.

MediaConvert presets and template

For the MediaConvert preset, we will use one preset for video and one preset for audio for simplicity. And we will use MS Smooth streaming protocol for transcoding.

A1 — Audio preset P1 — Video preset{
"Name": "A1",
"Settings": {
"AudioDescriptions": [
{
"AudioTypeControl": "FOLLOW_INPUT",
"AudioSourceName": "Audio Selector 1",
"CodecSettings": {
"Codec": "AAC",
"AacSettings": {
"AudioDescriptionBroadcasterMix": "NORMAL",
"Bitrate": 128000,
"RateControlMode": "CBR",
"CodecProfile": "LC",
"CodingMode": "CODING_MODE_2_0",
"RawFormat": "NONE",
"SampleRate": 48000,
"Specification": "MPEG4"
}
},
"LanguageCodeControl": "FOLLOW_INPUT"
}
],
"ContainerSettings": {
"Container": "ISMV"
}
}
}
{
  "Name": "P1",
  "Settings": {
    "VideoDescription": {
      "Width": 192,
      "ScalingBehavior": "DEFAULT",
      "Height": 108,
      "VideoPreprocessors": {
        "Deinterlacer": {
          "Algorithm": "INTERPOLATE",
          "Mode": "DEINTERLACE",
          "Control": "NORMAL"
        }
      },
      "TimecodeInsertion": "DISABLED",
      "AntiAlias": "ENABLED",
      "Sharpness": 50,
      "CodecSettings": {
        "Codec": "H_264",
        "H264Settings": {
          "InterlaceMode": "PROGRESSIVE",
          "NumberReferenceFrames": 1,
          "Syntax": "DEFAULT",
          "Softness": 0,
          "GopClosedCadence": 1,
          "HrdBufferInitialFillPercentage": 100,
          "GopSize": 48,
          "Slices": 1,
          "GopBReference": "DISABLED",
          "HrdBufferSize": 112000,
          "SlowPal": "DISABLED",
          "SpatialAdaptiveQuantization": "DISABLED",
          "TemporalAdaptiveQuantization": "DISABLED",
          "FlickerAdaptiveQuantization": "DISABLED",
          "EntropyEncoding": "CAVLC",
          "Bitrate": 112000,
          "FramerateControl": "INITIALIZE_FROM_SOURCE",
          "RateControlMode": "CBR",
          "CodecProfile": "BASELINE",
          "Telecine": "NONE",
          "MinIInterval": 0,
          "AdaptiveQuantization": "OFF",
          "CodecLevel": "LEVEL_3",
          "FieldEncoding": "PAFF",
          "SceneChangeDetect": "ENABLED",
          "QualityTuningLevel": "SINGLE_PASS",
          "FramerateConversionAlgorithm": "DUPLICATE_DROP",
          "UnregisteredSeiTimecode": "DISABLED",
          "GopSizeUnits": "FRAMES",
          "ParControl": "INITIALIZE_FROM_SOURCE",
          "NumberBFramesBetweenReferenceFrames": 0,
          "RepeatPps": "DISABLED"
        }
      },
      "AfdSignaling": "NONE",
      "DropFrameTimecode": "ENABLED",
      "RespondToAfd": "NONE",
      "ColorMetadata": "IGNORE"
    },
    "ContainerSettings": {
      "Container": "ISMV"
    }
  }
}

For the MediaConvert job template, we are going to use the following configuration. The presets A1 and P1 are mentioned in MS Smooth output settings.

{
  "Description": "Test VOD Material",
  "Name": "test_vod",
  "Settings": {
    "OutputGroups": [
      {
        "Name": "MS Smooth",
        "Outputs": [
          {
            "Preset": "A1"
          },
          {
            "Preset": "P1"
          }
        ],
        "OutputGroupSettings": {
          "Type": "MS_SMOOTH_GROUP_SETTINGS",
          "MsSmoothGroupSettings": {
            "FragmentLength": 2,
            "ManifestEncoding": "UTF8",
            "AudioDeduplication": "COMBINE_DUPLICATE_STREAMS"
          }
        }
      }
    ],
    "AdAvailOffset": 0,
    "Inputs": [
      {
        "AudioSelectors": {
          "Audio Selector 1": {
            "Offset": 0,
            "DefaultSelection": "DEFAULT",
            "ProgramSelection": 1
          }
        },
        "VideoSelector": {
          "ColorSpace": "FOLLOW",
          "Rotate": "DEGREE_0"
        },
        "FilterEnable": "AUTO",
        "PsiControl": "USE_PSI",
        "FilterStrength": 0,
        "DeblockFilter": "DISABLED",
        "DenoiseFilter": "DISABLED",
        "TimecodeSource": "EMBEDDED"
      }
    ]
  },
  "AccelerationSettings": {
    "Mode": "DISABLED"
  },
  "StatusUpdateInterval": "SECONDS_60",
  "Priority": 0,
  "HopDestinations": []
}

 

Terraform Configs

For each AWS region, there is a separate MediaConvert endpoint URL. Our first goal is to grab that endpoint URL because to get create preset and template we need MediaConvert endpoint URL. We are going to use AWS CLI for that. Let’s install that from here. Then we will write a simple bash script to grab the MediaConvert endpoint URL. We need to format the output with jq — A JSON processor (will explain why after some moments). Let’s install jq from here. Then our script will be like this in mediaconvert_url.sh file

# mediaconvert_url.sh

#!/bin/bash
set -e

# Extract 'profile' and 'region' from input JSON and set as environment variables
eval "$(jq -r '@sh "export profile=\(.profile) region=\(.region)"')"

# Describe AWS MediaConvert endpoints for the specified profile and region
aws mediaconvert describe-endpoints --profile "$profile" --region "$region" | jq -r .Endpoints[0]

Don’t forget to run

chmod a+x mediaconvert_url.sh

Now let’s set AWS provider in provider.tf

# provider.tf

provider "aws" {
  region  = "eu-west-1"
  profile = local.aws_profile
}

locals {
  # This profile name is used in multiple places, so keeping it in a local variable
  aws_profile = "your-aws-profile"
}

# For getting AWS region dynamically
data "aws_region" "current" {}

 

Now we have to pass the AWS profile set by us and region to mediaconvert_url.sh to get the MediaConvert endpoint URL. So we are going to use an external data source of Terraform and run the mediaconvert_url.sh. But terraform external data source only accepts JSON formatted output. That’s why we are using jq to format the output of the AWS MediaConvert describe-endpoints command. Let’s add an external terraform source in mediaconvert_url.tf

# mediaconvert_url.tf

locals {
  // Keeping the mediaconvert endpoint URL to a local var
  mediaconvert_endpoint = data.external.mediaconvert_endpoint.result.Url
}

# Run external program using terraform's data external
data "external" "mediaconvert_endpoint" {
  program = ["/bin/bash", "-c", "${path.module}/mediaconvert_url.sh"]
  query = {
    profile = local.aws_profile
    region  = data.aws_region.current.name
  }
}

# Terraform output - mediaconvert URL
output "mediaconvert_endpoint" {
  value = local.mediaconvert_endpoint
}

Now let’s keep our presets in presets directory and MediaConvert templates in templates directory like this

So we will use AWS CLI to create presets and job templates. But we will add this using a bash script. Also, our same bash script will be used to destroy. So a flag ACTION will be used to differentiate between creating and destroying. The script will look like this in mediaconvert.sh file.

# mediaconvert.sh
#!/bin/bash

# Process presets
find presets/* -prune -type f | while IFS= read -r preset; do
    echo "$preset"
    PRESET_NAME=$(jq -r '.Name' "$preset")
    PRESET_SETTINGS=$(jq '.Settings' "$preset")

    if [ "$ACTION" = "destroy" ]; then
        aws mediaconvert delete-preset \
            --name "$PRESET_NAME" \
            --endpoint-url "$MEDIACONVERT_URL" \
            --profile "$AWS_PROFILE" \
            --region "$REGION"
    else
        aws mediaconvert create-preset \
            --name "$PRESET_NAME" \
            --settings "$PRESET_SETTINGS" \
            --endpoint-url "$MEDIACONVERT_URL" \
            --profile "$AWS_PROFILE" \
            --region "$REGION"
    fi
done

# Process templates
find templates/* -prune -type f | while IFS= read -r template; do
    echo "$template"
    TEMPLATE_NAME=$(jq -r '.Name' "$template")
    TEMPLATE_SETTINGS=$(jq '.Settings' "$template")

    if [ "$ACTION" = "destroy" ]; then
        aws mediaconvert delete-job-template \
            --name "$TEMPLATE_NAME" \
            --endpoint-url "$MEDIACONVERT_URL" \
            --profile "$AWS_PROFILE" \
            --region "$REGION"
    else
        aws mediaconvert create-job-template \
            --name "$TEMPLATE_NAME" \
            --settings "$TEMPLATE_SETTINGS" \
            --endpoint-url "$MEDIACONVERT_URL" \
            --profile "$AWS_PROFILE" \
            --region "$REGION"
    fi
done

Also, don’t forget to run

chmod a+x mediaconvert.sh

As we can see that we have to pass environment variables ACTION, MEDIACONVERT_URL, AWS_PROFILE, and REGION. To do this we are going to use a local-exec provisioner of Terraform in a null resource. We are going to add 2 provisioner — one to create presets and template and another for destroy. And we will pass environment vars of mediaconvert.sh from this provisioners. Let’s do it in mediaconvert.tf file.

# mediaconvert.tf
resource "null_resource" "mediaconvert_creation" {
  // Provisioner for creation
  provisioner "local-exec" {
    when = create
    command = "./${path.module}/mediaconvert.sh"
    environment = {
      ACTION            = "create"
      AWS_PROFILE       = local.aws_profile
      REGION            = data.aws_region.current.name
      MEDIACONVERT_URL  = local.mediaconvert_endpoint
    }
  }

  // Provisioner for destroying
  provisioner "local-exec" {
    when = destroy
    command = "./${path.module}/mediaconvert.sh"
    environment = {
      ACTION            = "destroy"
      AWS_PROFILE       = local.aws_profile
      REGION            = data.aws_region.current.name
      MEDIACONVERT_URL  = local.mediaconvert_endpoint
    }
  }
}

Finally, our project will look like this

tree .
├── mediaconvert.sh
├── mediaconvert.tf
├── mediaconvert_url.sh
├── mediaconvert_url.tf
├── presets
│   ├── aws-mediaconvert-preset-a-1.json
│   └── aws-mediaconvert-preset-p-1.json
├── provider.tf
├── templates
│   └── aws-mediaconvert-template.json

Time to test: 

# Initialize Terraform in the current directory
terraform init

# Plan the infrastructure changes before applying
terraform plan

# Apply changes to create or update infrastructure
terraform apply -auto-approve

# Destroy all resources managed by Terraform
terraform destroy -auto-approve

 

After terraform apply we can see our output presets and job template in the AWS console

Finally, you can run a MediaConvert job with the template to test it. Before that don’t forget to create an IAM role for MediaConvert with necessary S3 bucket permission.

Picture of Mahabubur Rahaman Melon

Mahabubur Rahaman Melon

Software Development Engineer

Hire Exceptional Developers Quickly

Share this blog on

Hire Your Software Development Team

Let us help you pull out your hassle recruiting potential software engineers and get the ultimate result exceeding your needs.

Contact Us Directly

Address:

Plot # 272, Lane # 3 (Eastern Road) DOHS Baridhara, Dhaka 1206

Talk to Us
Scroll to Top