processing-s3-uploads-with-step-functions
Deploy an event-driven workflow that routes S3 uploads to either Lambda or Fargate via Step Functions based on file size. Uses EventBridge to trigger a Step Functions state machine when objects are uploaded to S3. Small files are processed by Lambda, large files by a Fargate task. Includes VPC, ECR
By aws · 3,576 installs
npx skills add aws/agent-toolkit-for-aws --skill processing-s3-uploads-with-step-functions
Source repository · Upstream listing
Step Functions Workflow: Route S3 Uploads to Lambda or Fargate
Overview
This skill deploys an event driven workflow using AWS CLI. When a file is uploaded to
an S3 bucket, EventBridge triggers a Step Functions state machine. The state machine
checks the file size and routes processing to either a Lambda function (files ≤ 6 MB)
or a Fargate task (files 6 MB).
The architecture includes:
An S3 bucket with EventBridge notifications enabled
An EventBridge rule that triggers Step Functions on S3 object creation
A Step Functions state machine with a Choice state for routing
A Lambda function for processing small files
An ECS Fargate task for processing large files
A VPC with two subnets, internet gateway, and security group
An ECR repository for the Fargate container image
Scoped IAM roles for Lambda, Step Functions, and ECS tasks
Use this skill when:
You need to process S3 uploads with different compute based on file size
You want a serverless workflow that can handle both small and large files
You need Step Functions orchestration with Lambda and Fargate
Do not use this skill when:
All files are small enough for Lambda (use S3 → Lambda directly)
You need real time streaming (use Kinesis)
You don't need file size based routing
Prerequisites
1. AWS CLI v2 — Installed and configured. Verify with aws sts get caller identity .
2. Python 3.12 — For the Lambda function runtime.
3. Docker — For building and pushing the Fargate container image.
Parameters
bucket name (required): Name for the S3 bucket (globally unique, lowercase, 3 63 characters)
region (required): AWS region for all resources
ecr repo name (required): Name for the ECR repository
state machine name (required): Name for the Step Functions state machine
kms key arn (optional): ARN of a KMS key for CloudWatch Logs encryption. If not provided, create one with aws kms create key description "Key for CloudWatch Logs encryption" region {region}
Constraints for parameter acquisition:
You MUST ask for all required parameters upfront in a single prompt
You MUST support multiple input methods (direct input, file path, URL)
You MUST confirm successful acquisition of all parameters before proceeding
You MUST validate that bucket name follows S3 naming rules
Procedures
Step 0: Verify Dependencies
Constraints:
You MUST verify the following tools are available: aws cli, python3 (3.12+), docker
You MUST inform the user about any missing tools with a clear message
You MUST ask if the user wants to proceed despite missing tools
You MUST respect the customer's decision to abort at any point
You MUST explain to the customer what step is being executed, why, and which tool is being called
Step 1: Retrieve AWS Account ID
Constraints:
You MUST retrieve the account ID with: aws sts get caller identity query 'Account' output text
You MUST store the result as {account id} for use in all subsequent steps
You MUST abort if credentials are not configured
Step 2: Get the Default VPC and Networking
Constraints:
You MUST retrieve the default VPC ID with:
aws ec2 describe vpcs filters Name=isDefault,Values=true query 'Vpcs[0].VpcId' output text region {region}
If no default VPC exists, inform the user they must create one with aws ec2 create default vpc region {region} or provide a VPC ID manually
You MUST retrieve two subnet IDs from the default VPC:
aws ec2 describe subnets filters Name=vpc id,Values={vpc id} query 'Subnets[0:2].SubnetId' output text region {region}
You MUST create a security group in the default VPC:
aws ec2 create security group group name fargate sg description "Security group for Fargate tasks" vpc id {vpc id} region {region}
You MUST configure security group egress rules to allow only HTTPS and DNS outbound. First revoke the default allow all egress rule:
aws ec2 revoke security group egress group id {sg id} ip permissions IpProtocol= 1,IpRanges='[{CidrIp=0.0.0.0/0}]' region {region}
Then add scoped rules:
aws ec2 authorize security group egress group id {sg id} protocol tcp port 443 cidr 0.0.0.0/0 region {region} and
aws ec2 authorize security group egress group id {sg id} protocol udp port 53 cidr 0.0.0.0/0 region {region}
You MUST recommend VPC endpoints for S3 and CloudWatch Logs for production workloads to avoid internet routed traffic and eliminate the need for broad egress rules
You MUST capture {vpc id}, {subnet1 id}, {subnet2 id}, and {sg id} for use in later steps
Step 3: Create the ECR Repository
Constraints:
You MUST create the repository with:
aws ecr create repository repository name {ecr repo name} region {region}
You MUST capture the repositoryUri from the response
Step 4: Build and Push the Container Image
Constraints:
You MUST verify Docker is installed by running docker version . If Docker is not installed, instruct the user to install it from https://docs.docker.com/get docker/ and abort until it is available
You MUST authenticate Docker with ECR:
aws ecr get login password region {region} docker login username AWS password stdin {account id}.dkr.ecr.{region}.amazonaws.com
The Dockerfile and processor code are in scripts/Dockerfile and scripts/fargate processor.py
You MUST build and push the image from the scripts directory:
Step 5: Create IAM Roles
Follow the detailed instructions in references/iam roles.md to create all IAM roles (Lambda, ECS task execution, ECS task, Step Functions, and EventBridge roles).
You MUST wait at least 10 seconds for IAM role propagation
Step 6: Create the Lambda Function
Constraints:
The function code is in scripts/lambda function.py
You MUST be in the skill root directory before packaging and creating the function
You MUST package it with: python3 c "import zipfile,io; z=io.BytesIO(); f=zipfile.ZipFile(z,'w'); f.writestr('lambda function.py', open('scripts/lambda function.py').read()); f.close(); open('/tmp/lambda function.zip','wb').write(z.getvalue())"
You MUST create the function with:
You MUST verify the function was created with:
aws lambda get function function name sfn file processor region {region}
Step 7: Create the CloudWatch Log Group
Constraints:
You MUST create the log group for Fargate:
aws logs create log group log group name /StepFunctionFargateTask region {region}
You MUST encrypt the log group with a KMS key:
aws logs associate kms key log group name /StepFunctionFargateTask kms key arn {kms key arn} region {region}
Step 8: Create the ECS Cluster and Task Definition
Follow the detailed instructions in references/ecs task definition.md to create the ECS cluster and register the Fargate task definition.
You MUST capture the task definition ARN from the response
Step 9: Create the S3 Bucket with EventBridge Notifications
Constraints:
You MUST create the bucket with:
aws s3api create bucket bucket {bucket name} region {region} create bucket configuration LocationConstraint={region}
You MUST NOT include create bucket configuration if region is us east 1
You MUST enable EventBridge notifications on the bucket:
aws s3api put bucket notification configuration bucket {bucket name} notification configuration '{"EventBridgeConfiguration": {}}' region {region}
You MUST enable default encryption on the bucket:
aws s3api put bucket encryption bucket {bucket name} server side encryption configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms"}}]}' region {region}
Step 10: Create the Step Functions State Machine
Constraints:
The state machine definition is in scripts/statemachine.asl.json
You MUST create a working copy and replace all placeholders:
You MUST create the state machine with:
You MUST capture the stateMachineArn from the response
Step 11: Create the EventBridge Rule
Constraints:
You MUST create the EventBridge rule to trigger on S3 object creation:
You MUST add the state machine as a target:
Step 12: Configure Monitoring
Constraints:
You MUST create a Dead Letter Queue for failed EventBridge invocations:
aws sqs create queue queue name s3 to stepfunctions dlq region {region}
You MUST update the EventBridge target to attach the DLQ:
You MUST create a CloudWatch alarm for Step Functions execution failures:
aws cloudwatch put metric alarm alarm name sfn execution failures metric name ExecutionsFailed namespace AWS/States statistic Sum period 300 threshold 1 comparison operator GreaterThanOrEqualToThreshold evaluation periods 1 dimensions Name=StateMachineArn,Value={state machine arn} region {region}
Step 13: Validate
Constraints:
You MUST test with a small file (< 6 MB) to verify Lambda processing:
You MUST wait 15 seconds then check the Step Functions execution:
aws stepfunctions list executions state machine arn {state machine arn} region {region}
You MUST verify the execution succeeded and routed to Lambda
You MUST provide a summary of all created resources including: VPC ID, subnet IDs, security group ID, ECR repo URI, ECS cluster ARN, task definition ARN, Lambda function ARN, state machine ARN, bucket name, and EventBridge rule name
Troubleshooting
EventBridge rule not triggering
Verify EventBridge notifications are enabled on the bucket: aws s3api get bucket notification configuration bucket {bucket name}
Verify the rule exists: aws events describe rule name s3 to stepfunctions region {region}
Check that the target has the correct state machine ARN and role
Step Functions execution fails at Fargate task
Verify the container image exists in ECR: aws ecr describe images repository name {ecr repo name} region {region}
Check that the subnets have internet access (route table with IGW)
Verify the security group allows outbound traffic
Check CloudWatch Logs at /StepFunctionFargateTask
Lambda invocation fails
Check CloudWatch Logs: aws logs tail /aws/lambda/sfn file processor region {region}
Verify the Step Functions role has lambda:InvokeFunction permission
IAM PassRole errors
The Step Functions role must have iam:PassRole for both the ECS execution role and task role ARNs
Fargate task stuck in PROVISIONING
Verify the subnets have auto assign public IP enabled
Verify the internet gateway is attached and route table has 0.0.0.0/0 route
Security Considerations
Fargate tasks with public IPs are exposed to the internet. Revoke the default allow all egress rule and configure scoped egress: aws ec2 revoke security group egress group id {sg id} ip permissions IpProtocol= 1,IpRanges='[{CidrIp=0.0.0.0/0}]' then add aws ec2 authorize security group egress group id {sg id} protocol tcp port 443 cidr 0.0.0.0/0 and aws ec2 authorize security group egress group id {sg id} protocol udp port 53 cidr 0.0.0.0/0 . For production, consider using VPC endpoints for S3 and CloudWatch Logs instead of internet routed traffic.
Scan container images for vulnerabilities before pushing to ECR. Enable ECR image scanning with: aws ecr put image scanning configuration repository name {ecr repo name} image scanning configuration scanOnPush=true region {region}
Use IAM roles for credentials — never hardcode access keys in container code.
Enable encryption at rest for the S3 bucket: aws s3api put bucket encryption bucket {bucket name} server side encryption configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms"}}]}'
Enable CloudWatch Logs encryption for Fargate container logs: aws logs associate kms key log group name /StepFunctionFargateTask kms key arn <KMS KEY ARN
Configure a Dead Letter Queue on the EventBridge rule for failed invocations
Set up CloudWatch alarms on Step Func