AWS: running it

Everyday deploys

Merge to main. CI runs; when it passes, Release builds and pushes the images, and Deploy rolls them out. Follow it in the repository's Actions tab, or with gh run watch.

A backend release takes about 15 minutes: around 2 minutes for the images when the build cache is warm, then the migrations and the rollouts. Commits that only change docs or Markdown files don't start CI or a release.

Rolling back

Run Deploy by hand with the full SHA of any earlier commit whose images are still in ECR (the last 50 are kept):

gh workflow run deploy.yml -R your-org/hypersaas-backend -f environment=staging -f sha=<earlier full SHA>

Or, in GitHub: Actions → Deploy → Run workflow.

Migrations don't roll back. The earlier code runs against the newer schema, which is fine as long as the migrations followed the rule in 2.5.

ECS also rolls back by itself when the new tasks keep failing to start or to pass their health checks. The Deploy job then fails and says so, and the previous version keeps serving.

Logs and service state

aws logs tail /ecs/hypersaas-staging/web --follow          # also worker, beat, frontend, migrate
aws ecs describe-services --cluster hypersaas-staging --services web worker beat frontend \
  --query 'services[].[serviceName, runningCount, desiredCount, deployments[0].rolloutState]' --output table
aws ecs describe-services --cluster hypersaas-staging --services worker \
  --query 'services[0].events[0:5].[createdAt, message]' --output text   # what ECS did recently

The web log has a line for every request (health checks aside): the client's address, the request, its status, size and time in milliseconds, and the user agent. For example, 203.0.113.7 "GET /api/chat/sessions/ HTTP/1.1" 200 2 1ms "Mozilla/5.0 …".

Running commands

Commands that don't ask questions run as one-off tasks, with the run function from step 10.

Interactive commands, such as createsuperuser or a Django shell, run inside a web task through ECS Exec. That needs AWS's Session Manager plugin (brew install --cask session-manager-plugin):

TASK=$(aws ecs list-tasks --cluster hypersaas-staging --service-name web --query 'taskArns[0]' --output text)
aws ecs execute-command --cluster hypersaas-staging --task "$TASK" --container web --interactive \
  --command "python manage.py shell"

Changing a setting

• Non-secret settings (the DOCUMENT_* tuning, LANGFUSE_BASE_URL and so on):
  1. Add them to app_environment in terraform.tfvars, for example app_environment = { DJANGO_SECURE_HSTS_SECONDS = "31536000" }.
  2. Run terraform apply.
  3. Run Deploy for the commit that's running, so the services pick up the new task definition.
• Sizes and options (CPU and memory, the database class, failover): change them in terraform.tfvars, apply, then deploy the running commit. Turning Spot on or off redeploys the services as part of the apply.
• The number of tasks: Terraform sets it only when it creates a service, so change it directly:

  aws ecs update-service --cluster hypersaas-staging --service web --desired-count 2

  Never run more than one beat.

Changing a key

Edit the external secret (step 7). Then restart the services so they read it:

for s in web worker beat frontend; do
  aws ecs update-service --cluster hypersaas-staging --service "$s" --force-new-deployment > /dev/null
done
run python /app/manage.py check_keys --generate

Backups and restoring

RDS takes a backup every day and keeps 7 days, and it can restore to any point in that window. A restore creates a new database; the old one keeps running until you switch:

SG=$(aws rds describe-db-instances --db-instance-identifier hypersaas-staging \
  --query 'DBInstances[0].VpcSecurityGroups[0].VpcSecurityGroupId' --output text)
aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier hypersaas-staging \
  --target-db-instance-identifier hypersaas-staging-restored \
  --use-latest-restorable-time \
  --db-subnet-group-name hypersaas-staging --vpc-security-group-ids "$SG" --no-publicly-accessible
aws rds wait db-instance-available --db-instance-identifier hypersaas-staging-restored

To check the restored data, run a one-off task with POSTGRES_HOST set to the new instance's address. In the run function's override, add environment: [{name: "POSTGRES_HOST", value: "<address>"}] next to command.

To switch over for good, rename the instances so the restored one takes the original name and address, then delete the old one. Practise this on staging before you need it.

Alarms and costs

Alarms arrive by email from SNS; confirm the subscription first. The budget alert covers the whole AWS account.

Staging in eu-west-2 costs about $120 a month before tax, from AWS's list prices in October 2026:

Tax comes on top; UK VAT adds 20%. Regular Fargate instead of Spot adds about $105. To spend less on staging, scale every service to 0 outside working hours, or shrink the worker if you don't index large documents.

Going to production

Production is the same Terraform with production's settings. You make it by copying the staging folder, without its local files, and giving the copy its own settings and its own state:

cd infra/terraform
rsync -a --exclude .terraform --exclude terraform.tfvars --exclude '*.tfstate*' envs/staging/ envs/production/
cd envs/production
cp production.tfvars.example terraform.tfvars    # then fill in your domains, sender address and image tags

1. In envs/production/versions.tf, set the state key to production/terraform.tfstate. It's the one line in the copy that changes, and it keeps production's state apart from staging's.
2. The example terraform.tfvars already has production's settings:
   • environment = "production"
   • regular Fargate instead of Spot
   • 2 web and 2 frontend tasks, so a deploy or a lost zone never leaves none running
   • a db.t4g.medium database with a standby in a second zone, and deletion protection
   • a Valkey replica that takes over if the primary fails
   • DJANGO_SECURE_HSTS_SECONDS at a year
3. Create the certificate, add its DNS records, apply, and point production's domains at the load balancer, as in steps 5 and 6.
4. In both repositories, create a GitHub environment called production:
   • limit it to main
   • add required reviewers, so a person approves each deploy
   • set its AWS_DEPLOY_ROLE_ARN from the production output
5. Production deploys run by hand from Actions → Deploy: choose production, with a SHA that has already passed staging. Merges keep deploying to staging only.
6. Use live Stripe keys and a live webhook, and production's own keys for every other service (step 7, in hypersaas-production/external).

Production costs roughly $300–370 a month before tax with these settings, from AWS's list prices; the database and cache failover are about half of the difference from staging.

The services run a fixed number of tasks. When traffic calls for more, ECS Service Auto Scaling can add tasks based on CPU or request count; the module doesn't set it up yet.

Removing staging

cd infra/terraform/envs/staging
terraform destroy

This deletes the database without a final snapshot (staging has no deletion protection), along with the bucket's files. Also delete the DNS records and the GitHub environments. The bootstrap stack can stay: it only costs the storage of the images.

When something goes wrong

Names on AWS

On AWS, names start with hypersaas-<environment>:

• the cluster hypersaas-staging
• the task definition families hypersaas-staging-web, -worker, -beat, -migrate and -frontend
• the log groups /ecs/hypersaas-staging/<process>
• the database hypersaas-staging
• the secrets hypersaas-staging/generated and hypersaas-staging/external

Your own name and region

Everything defaults to the name hypersaas and the region eu-west-2. To use your own, set them before the first apply; changing them later replaces the resources.