Background Processing

Architecture

Deploy background processing workloads on AWS ECS:

Queue Integration: Process messages from SQS, SNS, or other sources
Worker Scaling: Scale workers based on queue depth
Spot Instances: Use Fargate Spot for cost savings on interruptible work
Long-Running Tasks: Handle tasks that take minutes or hours to complete

When to Use

Background processing on ECS is ideal when:

You need to process jobs from a message queue
Workloads are asynchronous and do not require immediate response
Processing time varies significantly between jobs
You want to decouple processing from your main application
Cost optimization through Spot capacity is acceptable

Considerations

Queue-Based Scaling

Scale workers based on ApproximateNumberOfMessages metric
Set appropriate visibility timeouts for your workload
Implement dead-letter queues for failed messages
Consider batch size for efficient processing

Cost Optimization

Use Fargate Spot for non-critical batch jobs (up to 70% savings)
Right-size task CPU and memory based on actual usage
Scale to zero during off-peak hours when possible
Use scheduled scaling for predictable workloads

Example Configuration

# SQS queue for job processing
resource "aws_sqs_queue" "jobs" {
  name                       = "processing-jobs"
  visibility_timeout_seconds = 300
  message_retention_seconds  = 1209600
}

resource "aws_sqs_queue" "jobs_dlq" {
  name = "processing-jobs-dlq"
}

# Background worker service
module "worker_service" {
  source  = "registry.patterneddesigns.ca/patterneddesigns/ecs-service/aws"
  version = "2.0.0"

  service_name        = "job-worker"
  cluster_arn         = aws_ecs_cluster.main.arn
  task_definition_arn = aws_ecs_task_definition.worker.arn
  desired_count       = 2

  network_configuration = {
    subnets          = var.private_subnet_ids
    security_groups  = [aws_security_group.worker.id]
    assign_public_ip = false
  }

  capacity_provider_strategy = [
    {
      capacity_provider = "FARGATE_SPOT"
      weight            = 4
      base              = 1
    },
    {
      capacity_provider = "FARGATE"
      weight            = 1
      base              = 0
    }
  ]

  autoscaling = {
    min_capacity = 1
    max_capacity = 50
  }
}

# CloudWatch alarm for queue-based scaling
resource "aws_cloudwatch_metric_alarm" "queue_depth_high" {
  alarm_name          = "job-queue-depth-high"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "ApproximateNumberOfMessagesVisible"
  namespace           = "AWS/SQS"
  period              = 60
  statistic           = "Average"
  threshold           = 100
  alarm_description   = "Scale up workers when queue depth exceeds threshold"

  dimensions = {
    QueueName = aws_sqs_queue.jobs.name
  }
}

# Email processing worker
module "email_worker" {
  source  = "registry.patterneddesigns.ca/patterneddesigns/ecs-service/aws"
  version = "2.0.0"

  service_name        = "email-worker"
  cluster_arn         = aws_ecs_cluster.main.arn
  task_definition_arn = aws_ecs_task_definition.email.arn
  desired_count       = 1

  network_configuration = {
    subnets          = var.private_subnet_ids
    security_groups  = [aws_security_group.worker.id]
    assign_public_ip = false
  }

  autoscaling = {
    min_capacity = 1
    max_capacity = 10
  }
}