Deploying machine learning models on standalone AWS EC2 instances often hits a wall when traffic scales. Idle instances waste budget, while sudden inference bursts trigger latency spikes. Manually managing multiple EC2 instances for variant testing or rolling updates quickly turns into an infrastructure bottleneck.

Moving to Amazon Elastic Kubernetes Service (EKS) resolves these scaling and operational issues by decoupling the model application from the underlying virtual hardware.

The Core Migration Architecture

The transition moves the deployment model from a fixed virtual machine to an orchestrated, containerized environment managed via Infrastructure as Code (IaC).

  • Infrastructure Management: Terraform handles EKS cluster provisioning, VPC peering, and IAM roles for service accounts (IRSA).
  • Containerization: Docker packages the model artifact, web framework such as FastAPI, and runtime dependencies.
  • Orchestration: Kubernetes handles deployments, rolling updates, and horizontal pod autoscaling (HPA) based on CPU or custom inference metrics.

Step 1: Containerizing the ML Application

Before writing Kubernetes manifests, package your inference app into a minimal, reproducible Docker image. Avoid loading heavy model weights directly into the image layer during build time. Instead, configure the container to pull weights from an Amazon S3 bucket at startup.

FROM python:3.11-slim

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ .

EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Step 2: Defining the Kubernetes Deployment

Once you push the image to Amazon ECR, construct the Kubernetes deployment manifest. This file defines compute resource requests, limits, and environmental variables required to fetch the model parameters.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-inference-service
  namespace: ml-production
  labels:
    app: ml-inference
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ml-inference
  template:
    metadata:
      labels:
        app: ml-inference
    spec:
      containers:
        - name: predictor
          image: <your-aws-account-id>.dkr.ecr.eu-west-1.amazonaws.com/ml-inference:v1
          resources:
            requests:
              memory: "2Gi"
              cpu: "1000m"
            limits:
              memory: "4Gi"
              cpu: "2000m"
          ports:
            - containerPort: 8000
          env:
            - name: MODEL_S3_BUCKET
              value: "production-model-artifacts-bucket"

Step 3: Configuring Horizontal Pod Autoscaling (HPA)

To match the elasticity requirements of production traffic without over-provisioning compute nodes, implement an HPA resource targeting your deployment.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ml-inference-hpa
  namespace: ml-production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ml-inference-service
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

By shifting from EC2 to EKS, you gain immediate access to declarative deployments, zero-downtime rollouts, and efficient resource utilization via native cluster autoscaling.