AWS: first deployment

This page takes you from an empty AWS account to a staging site that deploys itself. Plan on two to three hours. Most of it is waiting: for the images to build, for the certificate, and for the database.

flowchart TD
    s1["1. Tools and accounts"] --> s2["2. The two repositories"]
    s2 --> s3["3. Bootstrap the AWS account<br/>(Terraform, once)"]
    s3 --> s4["4. Build the first images<br/>(GitHub Actions)"]
    s4 --> s5["5. Create staging<br/>(Terraform)"]
    s5 --> s6["6. DNS records"]
    s6 --> s7["7. Third-party keys"]
    s7 --> s8["8. Connect GitHub to staging"]
    s8 --> s9["9. First deploy"]
    s9 --> s10["10. Stripe, an admin user, checks"]

Step 1: Tools and accounts

On your computer:

You don't need Docker: GitHub Actions builds the images for the x86 processors the tasks run on.

Accounts:

AWS access for Terraform. Use IAM Identity Center (single sign-on) rather than access keys:

1. In the AWS console, open IAM Identity Center and enable it in your region.
2. Create a user for yourself.
3. Create a permission set with the AdministratorAccess policy.
4. Under AWS accounts, assign your user to the account with that permission set.
5. Accept the invitation email, and set up the user's password and MFA.

Then, on your computer:

aws configure sso --profile hypersaas   # the start URL from IAM Identity Center; region eu-west-2
aws sso login --profile hypersaas       # again whenever the session expires (after a few hours)
export AWS_PROFILE=hypersaas
aws sts get-caller-identity             # shows your account number

Step 2: The two repositories

Create two private repositories, for example your-org/hypersaas-backend and your-org/hypersaas-frontend, and push the code to main in each.

The AWS roles trust repositories by their numeric IDs. Print each repository's name in that form:

gh api repos/your-org/hypersaas-backend  --jq '"\(.owner.login)@\(.owner.id)/\(.name)@\(.id)"'
gh api repos/your-org/hypersaas-frontend --jq '"\(.owner.login)@\(.owner.id)/\(.name)@\(.id)"'
# your-org@1234567/hypersaas-backend@1234567890

Keep both lines: steps 3 and 5 use them.

Step 3: Bootstrap the AWS account (once)

The bootstrap stack creates what every environment shares:

• the S3 bucket for Terraform's state
• the ECR image repositories, and the build cache
• GitHub's identity provider in IAM
• the image publisher role
• a monthly budget alert

cd infra/terraform/bootstrap
cp terraform.tfvars.example terraform.tfvars

Fill in terraform.tfvars:

alert_email         = "you@example.com"
monthly_budget_usd  = 200   # the whole account, tax included
github_repositories = ["your-org@1234567/hypersaas-backend@1234567890", "your-org@1234567/hypersaas-frontend@1234567891"]
# If the account already has GitHub's OIDC provider (another project created it):
# create_github_oidc_provider = false
# Optional (see "Your own name and region" at the end):
# project = "acme"       # names start acme- instead of hypersaas-
# region  = "us-east-1"  # default eu-west-2

The state bucket doesn't exist yet, so the first apply keeps its state locally. Then the state moves into the bucket it just created:

terraform init -backend=false
terraform apply                       # review the plan, then type yes
terraform init -migrate-state -backend-config="bucket=$(terraform output -raw state_bucket)"   # another region: add -backend-config="region=…"
terraform state list                  # everything is listed: the state is in S3 now
rm terraform.tfstate terraform.tfstate.backup
terraform output image_publisher_role_arn

AWS emails you to confirm the budget alert's subscription. From now on, terraform init in this folder needs -backend-config="bucket=<project>-terraform-state-<account id>" (and the region, if it isn't eu-west-2).

Step 4: Build the first images

The staging environment needs images to start from. GitHub Actions builds them:

ROLE=$(terraform -chdir=infra/terraform/bootstrap output -raw image_publisher_role_arn)
gh variable set AWS_IMAGE_PUBLISHER_ROLE_ARN -R your-org/hypersaas-backend  --body "$ROLE"
gh variable set AWS_IMAGE_PUBLISHER_ROLE_ARN -R your-org/hypersaas-frontend --body "$ROLE"

gh workflow run release.yml -R your-org/hypersaas-backend
gh workflow run release.yml -R your-org/hypersaas-frontend
gh run watch -R your-org/hypersaas-backend      # about 10 minutes the first time: the worker image is large

Both runs should end green. Their Deploy job says "staging isn't set up yet … nothing was deployed", which is expected at this point. Note the commit each run built; those are the image tags:

gh api repos/your-org/hypersaas-backend/commits/main  --jq .sha
gh api repos/your-org/hypersaas-frontend/commits/main --jq .sha

Step 5: Create staging

cd infra/terraform/envs/staging
cp terraform.tfvars.example terraform.tfvars

Fill in terraform.tfvars:

app_domain          = "staging.example.com"
api_domain          = "api.staging.example.com"
email_from          = "Your App Staging <noreply@example.com>"   # its domain must be verified in Resend
alert_email         = "you@example.com"
github_repositories = ["your-org@1234567/hypersaas-backend@1234567890", "your-org@1234567/hypersaas-frontend@1234567891"]
image_tags = {
  backend  = "<backend SHA from step 4>"
  frontend = "<frontend SHA from step 4>"
}

Sizes and options (Fargate Spot, CPU and memory, the database, failover) are settings too. Their defaults in variables.tf are staging's: small, on Spot, one zone. Add any you want to change to terraform.tfvars. Then:

terraform init -backend-config="bucket=hypersaas-terraform-state-$(aws sts get-caller-identity --query Account --output text)"

Create the certificate first. The load balancer can't be created until the certificate is issued, and issuing it needs DNS records:

terraform apply -target=module.staging.aws_acm_certificate.this
terraform output acm_validation_records

Go to step 6 and add those records, then come back.

Then create everything else. Terraform waits for the certificate, then creates the network, load balancer, database, cache, bucket, secrets, roles, cluster and services. The database and the cache take 10 to 15 minutes.

terraform apply
terraform output load_balancer_dns_name

AWS emails you to confirm the alarms' subscription. Until the first deploy runs the migrations (step 9), beat restarts every few minutes, and the other services run without third-party keys. That's expected.

Step 6: DNS records

At your DNS host, add these CNAME records:

• Host names: many DNS hosts, Namecheap among them, want the host without your domain: _abc123.staging rather than _abc123.staging.example.com..
• Keep the validation records after the certificate is issued: ACM uses them to renew it every year.
• The bare domain (example.com) can't have a CNAME. Use subdomains as above, or a DNS host with ALIAS records, such as Route 53.

Check the records from your computer:

dig +short CNAME staging.example.com
curl -s https://api.staging.example.com/healthz/   # {"status": "ok"} once DNS has spread

Step 7: Third-party keys

The two secrets

Terraform creates two secrets in Secrets Manager:

Fill in the external secret in the AWS console, so the keys never sit in a file:

1. Open Secrets Manager in your region, and select hypersaas-staging/external.
2. Click Retrieve secret value, then Edit.
3. Fill in the keys you use. Leave the others empty, but don't delete any key: a task won't start if its secret is missing a key it expects.
4. Save.

Use separate keys for staging and production where the provider allows it, so you can revoke one without the other.

Step 8: Connect GitHub to staging

Each repository gets a GitHub environment called staging. It's limited to the main branch and holds the deploy role's ARN:

cd infra/terraform/envs/staging
DEPLOY_ROLE=$(terraform output -raw deploy_role_arn)

for REPO in your-org/hypersaas-backend your-org/hypersaas-frontend; do
  gh api -X PUT "repos/$REPO/environments/staging" --input - <<< '{"deployment_branch_policy":{"protected_branches":false,"custom_branch_policies":true}}'
  gh api -X POST "repos/$REPO/environments/staging/deployment-branch-policies" -f name=main -f type=branch
  gh variable set AWS_DEPLOY_ROLE_ARN -R "$REPO" --env staging --body "$DEPLOY_ROLE"
done

Step 9: First deploy

Deploy the commits from step 4. The backend's deploy also runs the migrations, and both deploys restart their services so they read the keys from step 7:

gh workflow run deploy.yml -R your-org/hypersaas-backend  -f environment=staging -f sha=<backend SHA>
gh workflow run deploy.yml -R your-org/hypersaas-frontend -f environment=staging -f sha=<frontend SHA>
gh run watch -R your-org/hypersaas-backend

From now on, every merge to main that passes CI deploys to staging by itself.

Step 10: Stripe, an admin user, and checks

These commands run on staging as one-off tasks. Define this shell function once (in bash or zsh). It runs a command in a new task, waits, then prints the output and the exit code:

run() {  # run python /app/manage.py <command> [args] on staging
  local network task
  network=$(aws ecs describe-services --cluster hypersaas-staging --services web \
    --query 'services[0].networkConfiguration' --output json)
  task=$(aws ecs run-task --cluster hypersaas-staging --task-definition hypersaas-staging-migrate --launch-type FARGATE \
    --network-configuration "$network" \
    --overrides "$(printf '%s\n' "$@" | jq -R . | jq -sc '{containerOverrides: [{name: "migrate", command: .}]}')" \
    --query 'tasks[0].taskArn' --output text)
  aws ecs wait tasks-stopped --cluster hypersaas-staging --tasks "$task"
  aws logs get-log-events --log-group-name /ecs/hypersaas-staging/migrate --log-stream-name "migrate/migrate/${task##*/}" \
    --start-from-head --query 'events[].[message]' --output text
  aws ecs describe-tasks --cluster hypersaas-staging --tasks "$task" --query 'tasks[0].containers[0].exitCode' --output text
}

Check the keys. check_keys makes the smallest authenticated call to each service and never prints a key. --generate also asks each AI provider for one token, which catches a valid key on an account that can't be billed:

run python /app/manage.py check_keys --generate

Stripe:

1. In Stripe, create a sandbox for staging. Copy its secret and publishable keys into the external secret (step 7).

2. Under Developers → Webhooks, add the endpoint https://api.staging.example.com/api/subscriptions/webhook, with no trailing slash. Select these events:

   • customer.subscription.created, customer.subscription.updated, customer.subscription.deleted
   • product.created, product.updated, product.deleted
   • price.created, price.updated, price.deleted

   Copy the endpoint's signing secret into STRIPE_WEBHOOK_SECRET.

3. Restart the services so they read the new keys (see Changing a key).

4. Create the plans in Stripe from backend/subscriptions/plans.py:

run python /app/manage.py sync_stripe_plans --dry-run   # shows what it would change
run python /app/manage.py sync_stripe_plans

An admin user. Sign up at https://staging.example.com like any user, and verify your email. Then make that account an admin:

run python /app/manage.py shell -c "from django.contrib.auth import get_user_model; print(get_user_model().objects.filter(email='you@example.com').update(is_staff=True, is_superuser=True))"
# prints 1

The admin lives at a random path, so scanners don't find it: https://api.staging.example.com/<path>. Print the path with:

aws secretsmanager get-secret-value --secret-id hypersaas-staging/generated --query SecretString --output text | jq -r .DJANGO_ADMIN_URL

Admin sign-in goes through allauth, with two-factor if you've set it up.

Check that it works:

• https://api.staging.example.com/healthz/ and https://staging.example.com/api/health return {"status": "ok"}
• Sign up, verify your email, sign in, and set up two-factor
• Create a workspace and invite a second address
• A chat that uses a tool, and one with a PDF attachment
• Upload a document, wait until it's processed, and ask about it: the answer cites it
• Subscribe with Stripe's test card 4242 4242 4242 4242, then open the billing portal
• check_keys --generate shows every key you set as ok

Next: AWS: running it.