AWS Elemental MediaConvert is a media transcoding service of AWS. For supporting vast VOD service, MediaConvert is very useful and reliable. Terraform is an Infrastructure as Code (IaC) tool which supports multiple Cloud platforms like AWS, Azure, and GCP. Unfortunately due to a lot of configurations of MediaConvert, terraform does not support MediaConvert resource and settings management yet. In this tutorial, we are going to take a look at how can we achieve this.
MediaConvert Job Templates
The job template is a part of MediaConvert. If we create some templates with input(s) and output(s) of transcoding, we can use that for every time to transcoding some files. Let’s say your platform needs 3 types of video outputs with different bitrate and resolution. So we can save that template and reuse for the next transcoding jobs. There is another section called Output presets in MediaConvert console. These preset store output configurations of transcoding jobs. And in the template, we can use one or more output presets.
Setting the Goal
So our plan is to manage MediaConvert templates with terraform as terraform does not support MediaConvert. So we will create templates and output presets at the time of Terraform apply command and destroy with Terraform destroy command.
MediaConvert presets and template
For the MediaConvert preset, we will use one preset for video and one preset for audio for simplicity. And we will use MS Smooth streaming protocol for transcoding.
A1 — Audio preset P1 — Video preset
{ "Name": "A1", "Settings": { "AudioDescriptions": [ { "AudioTypeControl": "FOLLOW_INPUT", "AudioSourceName": "Audio Selector 1", "CodecSettings": { "Codec": "AAC", "AacSettings": { "AudioDescriptionBroadcasterMix": "NORMAL", "Bitrate": 128000, "RateControlMode": "CBR", "CodecProfile": "LC", "CodingMode": "CODING_MODE_2_0", "RawFormat": "NONE", "SampleRate": 48000, "Specification": "MPEG4" } }, "LanguageCodeControl": "FOLLOW_INPUT" } ], "ContainerSettings": { "Container": "ISMV" } } }
{ "Name": "P1", "Settings": { "VideoDescription": { "Width": 192, "ScalingBehavior": "DEFAULT", "Height": 108, "VideoPreprocessors": { "Deinterlacer": { "Algorithm": "INTERPOLATE", "Mode": "DEINTERLACE", "Control": "NORMAL" } }, "TimecodeInsertion": "DISABLED", "AntiAlias": "ENABLED", "Sharpness": 50, "CodecSettings": { "Codec": "H_264", "H264Settings": { "InterlaceMode": "PROGRESSIVE", "NumberReferenceFrames": 1, "Syntax": "DEFAULT", "Softness": 0, "GopClosedCadence": 1, "HrdBufferInitialFillPercentage": 100, "GopSize": 48, "Slices": 1, "GopBReference": "DISABLED", "HrdBufferSize": 112000, "SlowPal": "DISABLED", "SpatialAdaptiveQuantization": "DISABLED", "TemporalAdaptiveQuantization": "DISABLED", "FlickerAdaptiveQuantization": "DISABLED", "EntropyEncoding": "CAVLC", "Bitrate": 112000, "FramerateControl": "INITIALIZE_FROM_SOURCE", "RateControlMode": "CBR", "CodecProfile": "BASELINE", "Telecine": "NONE", "MinIInterval": 0, "AdaptiveQuantization": "OFF", "CodecLevel": "LEVEL_3", "FieldEncoding": "PAFF", "SceneChangeDetect": "ENABLED", "QualityTuningLevel": "SINGLE_PASS", "FramerateConversionAlgorithm": "DUPLICATE_DROP", "UnregisteredSeiTimecode": "DISABLED", "GopSizeUnits": "FRAMES", "ParControl": "INITIALIZE_FROM_SOURCE", "NumberBFramesBetweenReferenceFrames": 0, "RepeatPps": "DISABLED" } }, "AfdSignaling": "NONE", "DropFrameTimecode": "ENABLED", "RespondToAfd": "NONE", "ColorMetadata": "IGNORE" }, "ContainerSettings": { "Container": "ISMV" } } }
For the MediaConvert job template, we are going to use the following configuration. The presets A1 and P1 are mentioned in MS Smooth output settings.
{ "Description": "Test VOD Material", "Name": "test_vod", "Settings": { "OutputGroups": [ { "Name": "MS Smooth", "Outputs": [ { "Preset": "A1" }, { "Preset": "P1" } ], "OutputGroupSettings": { "Type": "MS_SMOOTH_GROUP_SETTINGS", "MsSmoothGroupSettings": { "FragmentLength": 2, "ManifestEncoding": "UTF8", "AudioDeduplication": "COMBINE_DUPLICATE_STREAMS" } } } ], "AdAvailOffset": 0, "Inputs": [ { "AudioSelectors": { "Audio Selector 1": { "Offset": 0, "DefaultSelection": "DEFAULT", "ProgramSelection": 1 } }, "VideoSelector": { "ColorSpace": "FOLLOW", "Rotate": "DEGREE_0" }, "FilterEnable": "AUTO", "PsiControl": "USE_PSI", "FilterStrength": 0, "DeblockFilter": "DISABLED", "DenoiseFilter": "DISABLED", "TimecodeSource": "EMBEDDED" } ] }, "AccelerationSettings": { "Mode": "DISABLED" }, "StatusUpdateInterval": "SECONDS_60", "Priority": 0, "HopDestinations": [] }
Terraform Configs
For each AWS region, there is a separate MediaConvert endpoint URL. Our first goal is to grab that endpoint URL because to get create preset and template we need MediaConvert endpoint URL. We are going to use AWS CLI for that. Let’s install that from here. Then we will write a simple bash script to grab the MediaConvert endpoint URL. We need to format the output with jq — A JSON processor (will explain why after some moments). Let’s install jq from here. Then our script will be like this in mediaconvert_url.sh file
# mediaconvert_url.sh #!/bin/bash set -e eval "$(jq -r '@sh "export profile=\(.profile) region=\(.region)"')" aws mediaconvert describe-endpoints --profile "$profile" --region "$region" | jq -r .Endpoints[0]
Don’t forget to run
chmod a+x mediaconvert_url.sh
Now lets set AWS provider in provider.tf
# provider.tf provider "aws" { region = "eu-west-1" profile = local.aws_profile } locals { // This profile name used in multiple places // So keeping it in a local var aws_profile = "your-aws-profile" } // For getting AWS region dynamically data "aws_region" "current" {}
Now we have to pass the AWS profile set by us and region to mediaconvert_url.sh to get the MediaConvert endpoint URL. So we are going to use an external data source of Terraform and run the mediaconvert_url.sh. But terraform external data source only accepts JSON formatted output. That’s why we are using jq to format the output of the AWS MediaConvert describe-endpoints command. Let’s add an external terraform source in mediaconvert_url.tf
# mediaconvert_url.tf locals { // Keeping the mediaconvert endpoint URL to a local var mediaconvert_endpoint = data.external.mediaconvert_endpoint.result.Url } // Run external program using terraform's data external data "external" "mediaconvert_endpoint" { program = ["/bin/bash", "-c", "${path.module}/mediaconvert_url.sh"] query = { profile = local.aws_profile region = data.aws_region.current.name } } // Terraform output - mediaconvert URL output "mediaconvert_endpoint" { value = local.mediaconvert_endpoint }
Now let’s keep our presets in presets directory and MediaConvert templates in templates directory like this
So we will use AWS CLI to create presets and job templates. But we will add this using a bash script. Also, our same bash script will be used to destroy. So a flag ACTION will be used to differentiate between creating and destroying. The script will look like this in mediaconvert.sh file.
# mediaconvert.sh #!/bin/bash find presets/* -prune -type f | while IFS= read -r preset; do echo "$preset" PRESET_NAME=$(jq -r '.Name' "$preset") PRESET_SETTINGS=$(jq '.Settings' "$preset") if [ "$ACTION" = "destroy" ]; then aws mediaconvert delete-preset \ --name "$PRESET_NAME" \ --endpoint-url "$MEDIACONVERT_URL" \ --profile "$AWS_PROFILE" \ --region "$REGION" else aws mediaconvert create-preset \ --name "$PRESET_NAME" \ --settings "$PRESET_SETTINGS" \ --endpoint-url "$MEDIACONVERT_URL" \ --profile "$AWS_PROFILE" \ --region "$REGION" fi done find templates/* -prune -type f | while IFS= read -r template; do echo "$template" TEMPLATE_NAME=$(jq -r '.Name' "$template") TEMPLATE_SETTINGS=$(jq '.Settings' "$template") if [ "$ACTION" = "destroy" ]; then aws mediaconvert delete-job-template \ --name "$TEMPLATE_NAME" \ --endpoint-url "$MEDIACONVERT_URL" \ --profile "$AWS_PROFILE" \ --region "$REGION" else aws mediaconvert create-job-template \ --name "$TEMPLATE_NAME" \ --settings "$TEMPLATE_SETTINGS" \ --endpoint-url "$MEDIACONVERT_URL" \ --profile "$AWS_PROFILE" \ --region "$REGION" fi done
Also, don’t forget to run
chmod a+x mediaconvert.sh
As we can see that we have to pass environment variables ACTION, MEDIACONVERT_URL, AWS_PROFILE, and REGION. To do this we are going to use a local-exec provisioner of Terraform in a null resource. We are going to add 2 provisioner — one to create presets and template and another for destroy. And we will pass environment vars of mediaconvert.sh from this provisioners. Let’s do it in mediaconvert.tf file.
# mediaconvert.tf resource "null_resource" "mediaconvert_creation" { //Provisioner for creation provisioner "local-exec" { when = create command = "./${path.module}/mediaconvert.sh" environment = { ACTION = "create" AWS_PROFILE = local.aws_profile REGION = data.aws_region.current.name MEDIACONVERT_URL = local.mediaconvert_endpoint } } // Provisioner for destroying provisioner "local-exec" { when = destroy command = "./${path.module}/mediaconvert.sh" environment = { ACTION = "destroy" AWS_PROFILE = local.aws_profile REGION = data.aws_region.current.name MEDIACONVERT_URL = local.mediaconvert_endpoint } } }
Finally, our project will look like this
tree . ├── mediaconvert.sh ├── mediaconvert.tf ├── mediaconvert_url.sh ├── mediaconvert_url.tf ├── presets │ ├── aws-mediaconvert-preset-a-1.json │ └── aws-mediaconvert-preset-p-1.json ├── provider.tf ├── templates └── aws-mediaconvert-template.json
Time to test 😹 Now lets test with
terraform init # for initialization terraform terraform plan # planning what to create and/or destroy terraform apply -auto-approve # create all terraform destroy -auto-approve # To destroy all
After terraform apply we can see our output presets and job template in the AWS console
Finally, you can run a MediaConvert job with the template to test it. Before that don’t forget to create an IAM role for MediaConvert with necessary S3 bucket permission.