Data engineers often inherit cloud infrastructure by accident: one manually created bucket, one role with unclear permissions, and a spreadsheet explaining who is supposed to use what. That model works until the first team needs reproducibility, reviewable changes, or least-privilege access.

Terraform is useful because it turns infrastructure into versioned configuration. For ML and data workloads, that usually starts with object storage and IAM.

Terraform logo from the official repository

Image source: Terraform repository.

Why Start with S3 and IAM

For many ML systems, S3 becomes the default home for datasets, model artifacts, and pipeline outputs. IAM determines who can read, write, or publish those artifacts. If those two layers are unmanaged, everything above them becomes fragile.

Terraform helps because every change can be planned before it is applied:

$$ Change\ Confidence \propto Reviewability + Repeatability $$

That is the difference between “someone created a bucket” and “the platform can be recreated safely”.

Minimal Terraform Layout

infra/
|-- main.tf
|-- variables.tf
`-- outputs.tf

Keep the first version small. The objective is not abstraction. The objective is predictable provisioning.

Example: S3 Bucket and IAM Role

provider "aws" {
  region = "eu-west-1"
}

resource "aws_s3_bucket" "ml_artifacts" {
  bucket = "my-ml-artifacts-bucket"
}

resource "aws_iam_role" "ml_runner" {
  name = "ml-runner-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy" "ml_artifacts_policy" {
  name = "ml-artifacts-policy"
  role = aws_iam_role.ml_runner.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = ["s3:GetObject", "s3:PutObject", "s3:ListBucket"]
        Resource = [
          aws_s3_bucket.ml_artifacts.arn,
          "${aws_s3_bucket.ml_artifacts.arn}/*"
        ]
      }
    ]
  })
}

This is intentionally basic. It creates one bucket and one role with focused access to that bucket.

The Operational Workflow

The Terraform cycle should stay explicit:

terraform init
terraform fmt
terraform validate
terraform plan
terraform apply

For data teams, plan is the most important step because it exposes what will change before production resources are touched.

What Beginners Usually Get Wrong

Common mistakes include:

  • hardcoding everything into one file forever
  • granting broad IAM permissions too early
  • ignoring state handling
  • creating resources manually after adopting Terraform

That last mistake is especially expensive because it reintroduces drift immediately.

Final Advice

Terraform is most valuable when it removes ambiguity, not when it introduces framework-heavy indirection. For data engineers, a small reviewed module that provisions storage and access cleanly is more useful than a grand platform template nobody understands.

Start with buckets, roles, and a disciplined plan workflow. The rest of the stack can grow from there.

References