One line of curl did it — the command-line tool engineers use to send HTTP requests by hand, the same kind your browser sends when it loads a page:

curl -H "X-Forwarded-For: 203.0.113.7" https://api.client.example/auth/signin

The -H flag attaches a header to the request. X-Forwarded-For is the header proxies use to record the visitor's real IP address as the request travels through them. We will come back to why setting it by hand breaks the limiter.

The rest of this post is the reproduction recipe, the root cause, and the fix that closed the gap without breaking real traffic.

The Client Architecture

The intended request path looked like this:

Client → CloudFront → ALB → Express (Node.js)

What each of those four boxes actually is:

• Client — the visitor's browser, or an attacker's terminal.
• CloudFront — AWS's content delivery network. It sits at the edge of the internet, caches static files, and forwards everything else inward.
• ALB — Application Load Balancer. AWS's traffic distributor. Takes requests from CloudFront and routes them to whatever application server is healthy.
• Express (Node.js) — the application itself. The login endpoint and the rate limiter live here.

Two proxy hops between the visitor and the application. Each proxy prepends — adds to the front of — the X-Forwarded-For header. CloudFront writes the visitor's real IP at the front first, then the ALB writes CloudFront's IP at the front of that. By the time the request reaches Express, the header reads:

X-Forwarded-For: <real-client-ip>, <cloudfront-edge-ip>

Express needs the real client IP for rate limiting. That is what trust proxy exists for:

// express.ts:12
app.set('trust proxy', 2);

trust proxy: 2 tells Express that there are exactly two trusted proxies between the visitor and the application, and to skip the last two IPs in the X-Forwarded-For header (XFF for short) to find the real one:

XFF: <real-client>, <cloudfront>
         ↑              ↑
      req.ip         skip (proxy 1)
                    ALB adds its own (proxy 2, not in XFF)

Express reads the real client IP. The rate limiter counts per IP. Everything works — until you skip the proxy chain entirely.

What We Caught

The first signal was the ALB responding directly:

curl -v "$API/api/config" 2>&1 | grep -iE "server|x-cache|via"
# server: awselb/2.0

server: awselb/2.0 with no x-cache or via header. The request had reached the ALB without going through CloudFront. That changes the math. When traffic skips CloudFront, there is only one proxy hop, not two.

Direct to ALB (1 hop):
  XFF: <attacker-injected-ip>
           ↑
        Express reads this as req.ip
        (skips 2, but there is only 1 real proxy)

With trust proxy: 2, Express trims two entries from the right of X-Forwarded-For. There was only one real proxy. So Express reached into the part of the header the attacker controls and used whatever the attacker put there as the source IP.

We confirmed the rate limiter worked normally first:

# Five sign-in attempts from the same IP — rate limited correctly
for i in $(seq 1 5); do
  curl -s -o /dev/null -w "%{http_code} " -X POST \
    "$API/api/auth/signin" \
    -H "Content-Type: application/json" \
    -d '{"email":"test@example.com","password":"wrong"}'
done
# 429 429 429 429 429

Then with rotating fake IPs:

# Five attempts with rotating X-Forwarded-For — limiter sees five different "users"
for i in $(seq 1 5); do
  curl -s -o /dev/null -w "%{http_code} " -X POST \
    "$API/api/auth/signin" \
    -H "Content-Type: application/json" \
    -H "X-Forwarded-For: 10.0.$((RANDOM % 255)).$((RANDOM % 255))" \
    -d '{"email":"test@example.com","password":"wrong"}'
done
# 400 400 400 400 400

400s, not 429s. Every request reached the sign-in logic. The rate limiter saw each request as a different IP because Express trusted the attacker's X-Forwarded-For. Each fake IP carries its own counter, and no counter ever hits the threshold.

Brute force went through unimpeded. We reproduced it in staging and in production.

Why the ALB Was Reachable At All

The ALB security group was supposed to restrict inbound traffic to CloudFront only. The intended rule was present:

ingress {
  description = "lb_cloudfront_https_ingress_only"
  from_port   = 443
  to_port     = 443
  protocol    = "tcp"
  prefix_list = [data.aws_ec2_managed_prefix_list.cloudfront.id]
}

But three other rules were also present, left over from an earlier migration:

ingress {
  from_port   = 80
  to_port     = 80
  protocol    = "tcp"
  cidr_blocks = ["0.0.0.0/0"]
}

ingress {
  from_port   = 443
  to_port     = 443
  protocol    = "tcp"
  cidr_blocks = ["0.0.0.0/0"]
}

ingress {
  from_port   = 444
  to_port     = 444
  protocol    = "tcp"
  cidr_blocks = ["0.0.0.0/0"]
}

Ports 80, 443, and 444 open to the world. Security groups are additive — if any rule allows traffic, it is allowed. The CloudFront-only rule was correct. It was also useless, because the permissive rules let everyone in alongside it.

The Fix Isn't Dropping trust proxy

The reflex when a trust proxy: 2 system gets bypassed is to drop the number:

app.set('trust proxy', 1);  // We did NOT do this.

That breaks the rest of the stack. With trust proxy: 1 and legitimate traffic going through CloudFront + ALB (two real hops), Express would skip only one proxy and read CloudFront's edge IP as the source. Every user behind the same CloudFront edge node would share a single rate-limit counter. One user fails to log in five times, the entire region gets locked out.

The real fix is upstream: close the ALB to direct traffic and re-enforce the assumption that two-hop is always the path.

# Keep only this ingress rule for HTTPS to the ALB.
ingress {
  description = "CloudFront prefix list only"
  from_port   = 443
  to_port     = 443
  protocol    = "tcp"
  prefix_list = [data.aws_ec2_managed_prefix_list.cloudfront.id]
}

# Remove every 0.0.0.0/0 ingress rule.

After the apply, a direct curl against the ALB DNS name gets a connection refused. All traffic flows through CloudFront. trust proxy: 2 is correct again. The rate limiter counts the right IP.

Defense in Depth — What We Added Alongside

We did not stop at the security group fix. Two further layers went in alongside it, so that a future regression does not, on its own, re-open the bypass.

1. CloudFront custom-header validation at the app layer

CloudFront can sign every origin request with a custom header. The application then refuses to serve any request that does not carry it.

# CloudFront distribution origin config
custom_header {
  name  = "X-CloudFront-Secret"
  value = "<long-random-string>"
}

// Express middleware, runs before route handlers
app.use((req, res, next) => {
  if (req.headers['x-cloudfront-secret'] !== process.env.CF_SECRET) {
    return res.status(403).send('Direct access not allowed');
  }
  next();
});

Even if the security group regresses, non-CloudFront traffic is rejected at the application layer. The secret is rotated quarterly via Terraform and a CloudFront cache invalidation.

2. Account-based rate limiting on auth endpoints

Per-IP rate limiting assumes one person per IP. That assumption is fragile — corporate NATs, mobile carriers, and shared offices all break it. We added a second limiter keyed by email address on the sign-in endpoint:

const loginLimiter = rateLimit({
  keyGenerator: (req) => req.body.email || req.ip,
  max: 5,
  windowMs: 15 * 60 * 1000,
});

Now even if the attacker manages to rotate source IPs through some other path, the target account is protected after five attempts. The legitimate user gets a clear "too many attempts" response and an account-recovery email instead of a silent brute force in the background.

What We Shipped Back

We did not hand the client a thirty-page report. We handed them three things, all merged in the same sprint as the finding:

1. The reproduction recipe above, copy-pasteable, run against staging.
2. The Terraform diff that removes the 0.0.0.0/0 ingress rules.
3. The middleware diff that adds the CloudFront-secret check.

We re-ran the reproduction after the apply landed and got 429s back across the board. End-to-end from "this is broken" to "this is closed" was four working days — and on the client's side, the only ask was "can we point you at the staging environment for a day?" The rest is what they pay us for.

The Checklist We Carry Forward

Five checks we now run on every Express + ALB + CloudFront engagement.

1. trust proxy: N is only correct if there are always exactly N proxies between the client and the application. Any path that reduces the hop count makes the source IP attacker-controlled.
2. The CloudFront-prefix-list ingress rule is not a filter on its own. It is one allow-rule alongside whatever else is in the security group. Add it, and remove every 0.0.0.0/0 rule that competes with it.
3. Sign CloudFront origin requests with a header secret and check it in Express middleware. A defense-in-depth measure that survives security-group regressions.
4. Rate-limit auth endpoints by account, not just by IP. IPs lie. Account identifiers do not.
5. Reproduce the bypass from a fresh shell before assigning severity. The reproduction recipe is what a fix gets validated against — it is worth writing carefully even when the conclusion is obvious.

How To Check Your Own Setup

If you are running the same shape — Express behind CloudFront and an ALB — three commands tell you whether you have the same gap:

# 1. Is the ALB reachable directly?
curl -sI "https://api.client.example/health" | grep -iE "server|via|x-cache"
# server: awselb/2.0  →  yes, you are hitting the ALB directly

# 2. Does X-Forwarded-For rotation slip past rate limiting?
for i in $(seq 1 10); do
  curl -s -o /dev/null -w "%{http_code} " \
    -H "X-Forwarded-For: 10.0.$((RANDOM%255)).$((RANDOM%255))" \
    "https://api.client.example/api/auth/signin"
done
# 429 every time = fine.  Mix of 400/401 = problem.

# 3. Inspect the ALB security group
aws ec2 describe-security-groups --group-ids sg-xxx \
  --query 'SecurityGroups[].IpPermissions[].IpRanges[].CidrIp'
# 0.0.0.0/0 on port 443 = the CloudFront-only rule is doing nothing.

Common AWS WAF rate-limit questions

Four questions we hit while working through this — each answered against the same composite-key rule pattern from the fix section.

How do I rate-limit per hostname on AWS WAF?

Add HTTP_HEADER: Host as one of the aggregation keys in a RateBasedStatement CustomKey. The rule then counts requests per distinct Host: value instead of per source IP, so api.example.com and admin.example.com behind the same WAF get independent budgets. Combine with a ForwardedIP key for a per-host + per-client counter.

Can AWS WAF rate-limit on a 1-minute window instead of 5?

Not directly — RateBasedStatement supports a fixed set of evaluation windows (60, 120, 300, 600 seconds as of 2026). EvaluationWindowSec: 60 gets you the 1-minute window; the older 5-minute-only limit was removed. If you need a sub-minute window, that is Lambda-at-edge territory, not WAF.

How do I rate-limit by URL path segment?

Use Scope-down statement on the rate-based rule with a ByteMatchStatement on URI_PATH (starts-with, contains, or a regex). The rate rule only counts requests matching the scope-down predicate, so /api/auth/* and /api/search/* get separate budgets. Composite keys on HTTP_HEADER: Host + path prefix let a single WAF instance run distinct rules per route family.

How do I exclude a known good IP from an AWS WAF rate rule?

Two shapes. (a) An IPSet rule that allow-listed IPs (your office egress, a partner's server pool) evaluated before the rate rule with Action: Allow and Priority set lower — WAF short- circuits on first match. (b) A Scope-down statement on the rate rule with a NotStatement wrapping the IPSetReferenceStatement so the rate counter never even increments for those IPs.

CORS, in one sentence, is the browser's rule about which other websites are allowed to send authenticated requests to your API. Set it too narrow and your own subdomains can't talk to the backend. Set it too wide and any website on the internet can.

The team had already moved past the obvious mistake of allowing every site on the internet (origin: "*"). Instead, they had written a small text pattern (called a regex) that was meant to allow only the client's own subdomains through.

The regex was the problem.

The rest is the configuration we found, why "matches the regex" is not the same as "is safe to trust," and the explicit allowlist we shipped instead — including the answer every frontend team wants the moment they hear "allowlist": "what happens to our feature-branch preview URLs?"

The Configuration We Found

The CORS middleware was wired up like this:

// cors.config.ts
const corsOptions = {
  origin: /^https:\/\/([a-z0-9-]+\.)?client\.com$/,
  credentials: true,
  methods: ['GET', 'POST', 'PUT', 'DELETE', 'PATCH', 'OPTIONS', 'HEAD'],
};

That regex matches https://X.client.com for any subdomain X made of lowercase letters, digits, and hyphens. It also matches the bare apex domain https://client.com. The team's intent was clear: trust the client's own properties, reject everything else.

What it actually does is accept any subdomain that could be registered or pointed at the client's domain in the future — including ones that do not exist yet and ones that are owned by someone else.

How We Demonstrated the Problem

A quick test from the curl side:

# Legitimate origin — accepted, as expected.
curl -sI "$API/api/config" \
  -H "Origin: https://app.client.com" \
  | grep -i access-control
# access-control-allow-origin: https://app.client.com
# access-control-allow-credentials: true

# A subdomain that does not exist — also accepted.
curl -sI "$API/api/config" \
  -H "Origin: https://anything.client.com" \
  | grep -i access-control
# access-control-allow-origin: https://anything.client.com
# access-control-allow-credentials: true

The middleware reflects the supplied Origin header back as Access-Control-Allow-Origin, and sets Access-Control-Allow-Credentials: true, and accepts the request. The browser will happily send cookies and authorisation headers along with cross-origin requests from any matching subdomain.

CORS with credentials means the browser is authorising the cross-origin destination to act on the user's behalf with their session.

// Imagine this running on evil.client.com — a subdomain registered by an attacker
// after the client lets a wildcard DNS record lapse, or after a takeover.
fetch('https://api.client.com/user/profile', {
  credentials: 'include',
})
  .then((r) => r.json())
  .then((data) => sendToMyServer(data));

The legitimate user is on app.client.com. The malicious code is on evil.client.com. Both match the regex. The browser sends the user's session cookies along. The API returns the profile data. The attacker exfiltrates it.

The Preflight Was Also Verbose

While running the test matrix, we noticed the preflight response was generous:

curl -sI -X OPTIONS "$API/user" \
  -H "Origin: https://evil.client.com" \
  -H "Access-Control-Request-Method: DELETE"
# access-control-allow-methods: GET,POST,PUT,DELETE,PATCH,OPTIONS,HEAD
# access-control-allow-credentials: true

Every HTTP method advertised, credentials allowed. A preflight response is an explicit invitation list — listing methods the application does not even use on most endpoints tells an attacker which verbs to try.

Why This Happens — The Regex Trap

The regex is technically correct for the stated intent: "match any subdomain of client.com." The trap is that "any subdomain" includes subdomains that

• do not exist yet,
• could be registered by an attacker,
• could be taken over via a dangling DNS record,
• could be pointed at attacker-controlled infrastructure if a wildcard DNS entry is misconfigured.

Regex CORS configurations are routinely flagged because they conflate syntactic match with semantic trust. The regex says "the name looks right." It cannot say "the name is owned by us."

The Fix We Shipped — An Explicit Allowlist

We replaced the regex with a list of strings:

// cors.config.ts
const allowedOrigins = [
  'https://app.client.com',
  'https://www.client.com',
  'https://admin.client.com',
  'https://api.client.com',
];

// Feature-branch deployments add origins via env var at boot.
const featureBranchOrigins =
  process.env.FEATURE_BRANCH_ORIGINS?.split(',').filter(Boolean) ?? [];

const corsOptions = {
  origin: (origin, callback) => {
    if (!origin) {
      // Same-origin and curl/server-to-server have no Origin header.
      // Accept them — they are not browser cross-origin requests.
      return callback(null, true);
    }
    if (
      allowedOrigins.includes(origin) ||
      featureBranchOrigins.includes(origin)
    ) {
      return callback(null, true);
    }
    return callback(null, false);  // 403, not 500.
  },
  credentials: true,
};

Four properties on the allowlist by default. Feature branches plug in through an environment variable — the deployment pipeline appends the preview URL to FEATURE_BRANCH_ORIGINS when it provisions a branch deployment, and removes it when the branch is torn down.

Three things to note about the callback form:

1. callback(null, false) is the rejection path — the middleware returns a 403 instead of letting an exception bubble up to a 500. The application no longer leaks the existence of the endpoint to non-allowlisted origins.
2. The empty-origin case is explicit. Curl and server-to-server callers do not send an Origin header. Treating them as allowed prevents internal jobs from breaking when the same API serves machine traffic.
3. The credentials: true flag is still on — that is correct because the allowlist is now strict.

What We Had To Fix Alongside CORS

The middleware previously threw an Error on unknown origins:

// Previously
if (!isAllowed) throw new Error('Not allowed by CORS');

Throwing inside the CORS middleware turns a "denied" outcome into a 500 Internal Server Error from Express's default error handler. That tells the attacker the endpoint exists and is doing something interesting. The new callback path returns a clean 403, and the error handler stays out of it.

The Feature-Branch Conversation

The frontend team's first response was the one we expect on every engagement that touches CORS:

"But we need dynamic subdomains for feature branches. We cannot maintain a list."

The list is not maintained by hand — it is maintained by the deployment pipeline.

• When a branch deployment is created, the pipeline appends its origin (e.g. https://branch-1234.preview.client.com) to FEATURE_BRANCH_ORIGINS in the API's environment.
• When the branch is torn down or merged, the pipeline removes it.
• The API restarts on env change. Cold start is sub-second; the redeploy is a non-event.

The result is the same flexibility the regex used to give — any branch can talk to the API — without giving the same flexibility to anyone who can register any subdomain.

Verifying After the Change

The reproduction script we ran before the fix returns clean results afterwards:

# 1. Legitimate origin — still works
curl -sI "$API/api/config" \
  -H "Origin: https://app.client.com" \
  | grep -i access-control
# access-control-allow-origin: https://app.client.com
# access-control-allow-credentials: true

# 2. Unknown subdomain — no allow-origin header at all
curl -sI "$API/api/config" \
  -H "Origin: https://evil.client.com" \
  | grep -i access-control
# (no output — header not set)

# 3. Status code for blocked preflight — 403, not 500
curl -s -o /dev/null -w "%{http_code}\n" -X OPTIONS "$API/api/config" \
  -H "Origin: https://evil.client.com" \
  -H "Access-Control-Request-Method: GET"
# 403

The browser, on the legitimate side, sees the same response as before. The browser, on the attacker side, sees a CORS error and never gets to the response body.

What We Added Alongside CORS — The Layered Story

CORS is a browser policy. A determined attacker can still hit the API directly without a browser. Three companion changes shipped alongside the allowlist:

1. Allowed methods narrowed per endpoint. The middleware no longer advertises DELETE on endpoints that do not support it. Each route declares its own method set.
2. JWT validation on every authenticated request. A separate audience check — covered in its own case study — runs alongside CORS. Even on the days CORS gets misconfigured again, the audience check holds.
3. Origin telemetry. We log every rejected origin to CloudWatch with a 30-day retention window. If a hostname starts showing up, the team sees it before it shows up in a support ticket.

Five things we now check on every CORS engagement

1. Regex CORS allowlists are almost always wrong. The string form is more verbose but it is also unambiguous. Add a linter rule that fails the build on a non-string cors.origin value.
2. Wildcard subdomain regex matches subdomains you do not control. If you do not own every subdomain of your apex, the regex hands trust to whoever does.
3. callback(null, false) is the rejection path. Throwing inside CORS middleware turns a 403 into a 500 and tells the attacker something interesting happened.
4. Feature branches do not need wildcard CORS. A pipeline that updates an FEATURE_BRANCH_ORIGINS env var on create and tear-down delivers the same DX without the security gap.
5. Tighten Access-Control-Allow-Methods per route. A blanket method allowlist is an open menu of verbs for an attacker to probe. Routes declare the verbs they accept; the middleware reflects only those.

The obvious question was whether we could replace it with all native AWS services and keep the same browser-direct upload pattern. Yes, with a Cognito + S3 setup. We built a three-layer permission model that most Cognito-on-S3 tutorials skip, which is why they leak.

Two AWS services do the work here, both worth one sentence each before we go further:

• S3 — AWS's object-storage service. The bucket the files actually land in. Cheap, durable, billed per gigabyte stored and per request.
• Cognito Identity Pool — AWS's way of handing a browser a short-lived AWS credential so it can talk to S3 directly, without an API server in the middle. The "boundary" the rest of this post tightens.

What the Frontend Had to Do

The browser uploads files directly to storage — never through the backend API. This pattern matters for two reasons:

1. Throughput. A 200 MB upload doesn't tie up a backend worker for two minutes.
2. Cost. Backend egress through the API doubles the bandwidth bill.

The managed service handled this with a hosted SDK: the browser calls the service, gets a one-shot signed URL, uploads to the vendor's S3, and the vendor pushes the object into the client's bucket via a webhook. Clean DX. Big invoice.

What we needed to replicate:

• Browser uploads directly to the client's S3 bucket
• No long-lived AWS credentials in the browser
• Per-user isolation (user A cannot read or overwrite user B's files)
• Content-type and size guards
• Anti-abuse layer for anonymous (pre-signup) uploads

The Architecture

The pattern is well-known: AWS Cognito Identity Pool → temporary AWS credentials → direct PUT to S3. AWS Amplify ships this same architecture as defineStorage, which gives us a useful reference point.

The flow on a happy path:

The whole thing lives in three Terraform files: cognito.tf, s3.tf, and iam.tf. The browser code is roughly thirty lines of @aws-sdk/client-s3 and the standard Cognito credential provider.

The catch — and the reason most Cognito-on-S3 implementations leak — is that three independent layers decide whether a given operation is allowed. Get any one of them wrong and you've either bricked uploads or opened the bucket to the internet.

The Three Layers of Permission Evaluation

This is the mental model we built before writing a single line of Terraform.

The non-obvious behavior:

• The session scope-down for unauthenticated identities does not include S3. So an IAM grant of s3:PutObject to the unauthenticated role is silently blocked by the session policy. This trips up everyone the first time.
• Bucket policies bypass the session scope-down. A grant of s3:PutObject to the Cognito role principal inside the bucket policy is honored.
• Therefore: put your real grants in the bucket policy, not the IAM policy (or grant via both for defense in depth — only the bucket-policy grant will actually fire for unauthenticated identities).

Print it out.

The Terraform — Identity Pool

resource "aws_cognito_identity_pool" "uploads" {
  identity_pool_name               = "client-uploads"
  allow_unauthenticated_identities = true
}

resource "aws_cognito_identity_pool_roles_attachment" "uploads" {
  identity_pool_id = aws_cognito_identity_pool.uploads.id
  roles = {
    "unauthenticated" = aws_iam_role.cognito_unauth.arn
  }
}

We do allow unauthenticated identities, because the client's app needs to accept uploads before signup completes. The risk is real, and the rest of the Terraform is how we tightened around it.

The Terraform — IAM Role

The IAM role for the unauthenticated identity gets a minimal trust policy and a permission policy that grants only s3:PutObject to a per-identity prefix. Note: this is a defense-in-depth layer; the actual grant that fires for unauthenticated identities will be the bucket policy below.

data "aws_iam_policy_document" "cognito_unauth_assume" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = ["cognito-identity.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "cognito-identity.amazonaws.com:aud"
      values   = [aws_cognito_identity_pool.uploads.id]
    }
    condition {
      test     = "ForAnyValue:StringLike"
      variable = "cognito-identity.amazonaws.com:amr"
      values   = ["unauthenticated"]
    }
  }
}

resource "aws_iam_role" "cognito_unauth" {
  name               = "cognito-uploads-unauth"
  assume_role_policy = data.aws_iam_policy_document.cognito_unauth_assume.json
}

data "aws_iam_policy_document" "cognito_unauth_perms" {
  statement {
    actions   = ["s3:PutObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts"]
    resources = [
      "${aws_s3_bucket.uploads.arn}/uploads/$${cognito-identity.amazonaws.com:sub}/*",
      "${aws_s3_bucket.uploads.arn}/documents/$${cognito-identity.amazonaws.com:sub}/*",
    ]
    condition {
      test     = "StringLike"
      variable = "s3:content-type"
      values   = ["image/*", "video/*", "application/pdf"]
    }
  }
}

resource "aws_iam_role_policy" "cognito_unauth" {
  role   = aws_iam_role.cognito_unauth.id
  policy = data.aws_iam_policy_document.cognito_unauth_perms.json
}

Three constraints, all important:

1. s3:PutObject only. No GetObject, no DeleteObject, no ListBucket. The browser writes; it does not read or enumerate.
2. Per-identity prefix via ${cognito-identity.amazonaws.com:sub}. AWS interpolates this at request time. User A's credentials can only write to uploads/<A's-sub-id>/*. They cannot guess their way into User B's folder. Terraform's $$ is the escape for AWS template syntax — easy to miss the first time.
3. s3:content-type allowlist. The browser declares the content type; AWS checks it before accepting the PUT. text/html is denied at the IAM boundary. (Note: s3:content-type works in IAM policies but not in bucket policies — keep this one here.)

The Terraform — Bucket and Bucket Policy

Because the session scope-down blocks the IAM grant for unauthenticated identities, the actual PutObject allow has to live in the bucket policy:

resource "aws_s3_bucket" "uploads" {
  bucket = "client-uploads-prod"
}

resource "aws_s3_bucket_public_access_block" "uploads" {
  bucket                  = aws_s3_bucket.uploads.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_versioning" "uploads" {
  bucket = aws_s3_bucket.uploads.id
  versioning_configuration { status = "Enabled" }
}

data "aws_iam_policy_document" "uploads_bucket_policy" {
  # The grant that actually fires for unauthenticated Cognito identities.
  statement {
    sid       = "AllowScopedUploads"
    effect    = "Allow"
    actions   = ["s3:PutObject", "s3:AbortMultipartUpload"]
    principals {
      type        = "AWS"
      identifiers = [aws_iam_role.cognito_unauth.arn]
    }
    resources = [
      "${aws_s3_bucket.uploads.arn}/uploads/$${cognito-identity.amazonaws.com:sub}/*",
      "${aws_s3_bucket.uploads.arn}/documents/$${cognito-identity.amazonaws.com:sub}/*",
    ]
  }

  # Deny everything else, explicitly, for the Cognito role.
  statement {
    sid       = "DenyEverythingElse"
    effect    = "Deny"
    actions = [
      "s3:GetObject", "s3:DeleteObject", "s3:ListBucket",
      "s3:ListBucketVersions", "s3:ListBucketMultipartUploads",
    ]
    principals {
      type        = "AWS"
      identifiers = [aws_iam_role.cognito_unauth.arn]
    }
    resources = [
      aws_s3_bucket.uploads.arn,
      "${aws_s3_bucket.uploads.arn}/*",
    ]
  }
}

resource "aws_s3_bucket_policy" "uploads" {
  bucket = aws_s3_bucket.uploads.id
  policy = data.aws_iam_policy_document.uploads_bucket_policy.json
}

block_public_acls plus block_public_policy mean the bucket can never accidentally drift to "world-readable" — even if a future engineer adds a Principal: * statement, AWS rejects the policy at apply time.

Verifying With curl

Before handing off, we validated the boundary the same way an attacker would — with curl and no SDK in between.

# 1. Get an anonymous identity.
curl -s -X POST "https://cognito-identity.us-east-1.amazonaws.com/" \
  -H "Content-Type: application/x-amz-json-1.1" \
  -H "X-Amz-Target: AWSCognitoIdentityService.GetId" \
  -d '{"IdentityPoolId":"us-east-1:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"}'

# 2. Exchange it for temporary AWS credentials.
curl -s -X POST "https://cognito-identity.us-east-1.amazonaws.com/" \
  -H "Content-Type: application/x-amz-json-1.1" \
  -H "X-Amz-Target: AWSCognitoIdentityService.GetCredentialsForIdentity" \
  -d '{"IdentityId":"us-east-1:yyyyyyyy-..."}'

We then exported the credentials and ran the matrix of operations our policy is supposed to block:

# Reads — denied
aws s3 cp s3://client-uploads-prod/uploads/foo/bar.jpg /tmp/x
# An error occurred (AccessDenied) ...

# Listing — denied
aws s3 ls s3://client-uploads-prod/
# An error occurred (AccessDenied) ...

# Deletes — denied
aws s3 rm s3://client-uploads-prod/uploads/foo/bar.jpg
# An error occurred (AccessDenied) ...

# Writing outside your own prefix — denied
echo "x" | aws s3 cp - s3://client-uploads-prod/uploads/somebody-else/file.txt
# An error occurred (AccessDenied) ...

# Writing wrong content type to your own prefix — denied
aws s3 cp evil.html s3://client-uploads-prod/uploads/$MY_SUB/evil.html \
  --content-type "text/html"
# An error occurred (AccessDenied) ...

# Writing the right content type to your own prefix — allowed
aws s3 cp ok.jpg s3://client-uploads-prod/uploads/$MY_SUB/ok.jpg
# upload: ./ok.jpg to s3://client-uploads-prod/...

Every line in that matrix is part of the acceptance test. We re-run it in CI against staging on every Terraform change.

What We Added Back From the Managed Service

Replacing a managed service means rebuilding the safety net the vendor wrapped around the happy path. Three pieces in particular needed deliberate replacement.

1. File-size limits and content-type validation, properly

The IAM-level s3:content-type check trusts the header — easy to spoof. We added a Lambda triggered by s3:ObjectCreated:* that:

• Reads the first 16 bytes of the new object (magic bytes)
• Matches against an allowlist of file signatures (JPEG, PNG, MP4, PDF…)
• Deletes the object and writes a denied/<sub>/<timestamp>.log audit entry if the magic bytes do not match the prefix policy

This is the same defense-in-depth pattern the managed service used; we now own it.

2. Anti-abuse for guest uploads

For genuinely anonymous flows (pre-signup), we put a small backend endpoint in front. The browser proves it is a human with reCAPTCHA, the backend validates the token, and only then does the frontend get to call GetCredentialsForIdentity. This converts "anyone can get AWS credentials" into "any verified human can get AWS credentials" — the bar we wanted.

3. Bucket abuse limits

We layered in two things the managed service used to do for us:

• S3 Lifecycle rule that deletes anything in uploads/$sub/ older than 30 days unless the object has been promoted (a tag set when the user completes signup).
• CloudWatch alarm on BucketSizeBytes and on NumberOfObjects per prefix, with thresholds set at 5× the 30-day rolling average. A bored teenager filling the bucket with zeros sets off the alarm before it sets off the invoice.

Migration Plan (No Downtime)

We rolled this out in three phases over two sprints. The client never lost an upload.

1. Dual-write. Frontend hits the managed service and the new Cognito flow in parallel. Compare file hashes in the backend. Surface any mismatch in a Slack channel.
2. Read switch. Once dual-write was clean for two weeks, the application started reading from S3. The managed service's pipeline stayed running, but downstream consumers stopped touching it.
3. Cutover. Frontend disables the managed-service call. Vendor account moved to "evergreen" tier. After 60 days of clean operation we cancelled the contract.

What the Client Walked Away With

In rough numbers (exact figures live in the client's invoice records, not ours):

• The monthly bill dropped sharply. Most of the savings came from the dominant cost shifting from "you pay per upload" to "you pay per gigabyte stored" — and on this workload, per-GB storage on S3 is a fraction of what the managed service was charging per request.
• They own the pipeline now. The whole upload pipeline lives in their own Terraform repository. If they ever want to swap a piece out, they can.
• They can do things the managed service did not let them do. Their backend can now sign upload URLs based on any business rule they care about — has the user finished signup, are they on a paid plan, what country are they in. None of that was possible before.
• The boundary is tighter than the tutorial version. Per-user prefixes, magic-byte validation, abuse alarms, versioning so we can undo a bad upload, CloudTrail data events on every S3 operation. The kind of plumbing that gets noticed only when something tries to abuse it.

The client's reaction, paraphrased: "This is the architecture we wished the vendor's docs had explained in the first place."

Two gotchas the tutorials skip

First, s3:prefix only applies to s3:ListBucket, not to s3:ListBucketMultipartUploads — split them into separate statements or AWS rejects the policy at apply time. Second, SDK helpers like --dryrun check client-side logic only; they do not call AWS IAM. Test the deny path with curl and the AWS CLI directly, or the acceptance test is lying to you.

Two things I would enable from day one on the next build: versioning on the bucket, so a bad upload is one command away from being undone, and CloudTrail data events, so the forensic answer to "what did the temporary credential actually do" is available at all.

The team had two constraints. "Do not block the rest of the work" and "do not break production." On a service that deploys every day, with a long backlog of feature work that the product team had committed to, an upgrade that needed the whole team to stop and migrate together would have been the wrong shape. We had to find the order that let the migration happen alongside the feature work, with the test suite green at every commit and production unaware that anything had changed.

The order we ended up with: a staging branch that absorbed the breaking changes, the test-harness migration ahead of the runtime, the Docker image pinned in parallel, and the eight or so smaller follow-ups that landed after the main upgrade so nobody had to read a five-thousand-line PR.

Two terms before we go further

• TypeORM is a TypeScript ORM for SQL databases. It maps decorated TypeScript classes to tables and gives you a query builder, an entity manager, and a migration runner. The 0.x to 1.0 upgrade is the kind of release that touches every file in the codebase that talks to the database.
• Test harness, in this post, is the infrastructure around the test suite: the factories that build entities, the per-test database setup and teardown, the mocked services, the rate-limit mocks, the fixtures. None of it ships to production. All of it has to be green for CI to pass.

Four places the upgrade broke

The breaking surface on the upgrade landed in four places.

Factory APIs. TypeORM's setSeederFactory shape changed. Every factory we had (about thirty of them, one per major entity) needed a new signature.

Repository methods. A handful of methods that had a "first one wins" behaviour in 0.x became "explicit single or throw" in 1.0. findOne() without an options argument is no longer valid; you have to pass findOne({ where: { id } }) or use findOneBy({ id }). This was a global codemod.

Entity manager transactions. The transaction API tightened. Some callsites that had been relying on implicit transaction propagation needed explicit manager.withRepository(...) calls.

Decorator metadata. @PrimaryGeneratedColumn got stricter about its options. A handful of entities with custom configurations needed a small rewrite.

None of those were hard individually. All four at once, on a codebase with hundreds of entities, was the upgrade.

The order we ran it in

Each step below has its own commit (or branch) and was reviewable on its own.

One. Pin the new Docker image and Node version on a parallel branch.

TypeORM 1.0 requires Node 22. Our production image was on Node 20. Before any code changed, the new image had to be built and tested. We landed Node 22.17.1 + production stage yarn setup for typeorm@1.0.0 as the first commit on the upgrade branch:

# Dockerfile (shortened)
FROM node:22.17.1-bookworm-slim AS builder
WORKDIR /app
COPY package.json yarn.lock .yarnrc.yml ./
COPY .yarn .yarn
RUN yarn install --immutable

FROM node:22.17.1-bookworm-slim AS production
WORKDIR /app
COPY --from=builder /app/node_modules ./node_modules
COPY . .
RUN yarn install --immutable --production
CMD ["node", "dist/server.js"]

The build target uses the production-stage yarn setup because the new TypeORM brings transitive dependencies that the old --immutable flag was tolerant of and the new one is not. We ran the new image against the old code for a week in staging to make sure the runtime change itself was inert.

Two. Migrate the test harness before the runtime.

This is the step that decided whether the upgrade would block the team or not. The test harness is what runs in CI on every PR. If the test harness is half-migrated, nobody can ship anything. So the test harness goes first, in its own PR, with the old TypeORM still in production:

// database/factories/address.factory.ts (before)
import { define } from "typeorm-seeding";
import { Address } from "../../modules/address/address.entity";

define(Address, (faker) => {
  const address = new Address();
  address.street = faker.location.street();
  address.city = faker.location.city();
  return address;
});

// database/factories/address.factory.ts (after)
import { setSeederFactory } from "typeorm-extension";
import { Address } from "../../modules/address/address.entity";

export default setSeederFactory(Address, (faker) => {
  const address = new Address();
  address.street = faker.location.street();
  address.city = faker.location.city();
  return address;
});

About thirty factories, all the same shape, all migrated in one commit. We landed the test harness work as a series of "complete TypeORM 1.0 test-harness migration" commits. Runtime was still on 0.x, test setup was ready for 1.0. The two coexisted because the factories are dev-time only.

Three. Bump the runtime on a staging branch.

The actual package.json change. Roughly fifteen lines changed in the manifest, a hundred or so callsites changed across the source for the findOne → findOneBy codemod and the transaction-API tightening. We landed it on a staging branch that lived for about a week.

{
  "dependencies": {
    "typeorm": "^1.0.0",
    // ... transitive updates for things that wanted Node 22
  }
}

During that week, every other PR rebased onto the staging branch as it was merged. Conflicts happened (entity decorators and query builders both got touched), but they stayed bounded. We had a Slack channel for "my PR is breaking on the staging rebase, who else has touched this file" and it never had more than one or two messages a day.

Four. Fix the small tests that quietly broke.

Once the runtime was on 1.0, a small set of tests broke for reasons the code diff itself did not show. The framework was behaving slightly differently now, and the tests were catching it. A rate-limit mock had assumed the old library called it at one specific moment; the new library called it a moment later. A recaptcha mock had relied on a method name that had been renamed. Some test suites we had not touched in a long time were quietly assuming old data-object shapes and stopped compiling once the underlying types changed.

We landed these as a series of small commits with names like "test: align auth assertions and complete rate-limit mock" and "test: fix constructor drift in the compile-failing test suites." Each one was a few hours of work and a small PR. Nothing dramatic.

Five. Ship the security fixes we had queued up.

This one was a bonus. The TypeORM upgrade gave us a single PR window where every service was already being touched. We pulled in a handful of unrelated security findings (bugs where an attacker could swap in another user's ID, and write endpoints that let one user modify another user's data) and shipped them in the same staging branch. "While we are in here" is a real shipping pattern; we used it once, deliberately, on the boundary the upgrade had opened anyway.

Two conflicts we planned for and didn't need

Two conflicts I want to call out because they are the ones I was expecting and did not get.

Production rollback. We had a plan. The plan was "deploy the new image, watch the error rate, roll back if anything spikes." We did not need it. The week of staging burn-in caught everything that would have spiked.

Migration runner. TypeORM 1.0 has a different migrations API than 0.x. We were prepared to rewrite our migration files. We did not need to. The existing migrations are run-once historical records that the new runner reads cleanly without changes. The new migrations going forward use the new API, but the catalogue of historical migrations is untouched.

What we'd do again on the next one

1. Test harness first, runtime second. If the harness is broken, every other PR is blocked. If the runtime is broken, only the upgrade PR is blocked.
2. Pin the new base image on a parallel branch. Node version upgrades that come bundled with the main one should be tested as their own change before the code change lands on top.
3. Codemod the global API changes in one commit. findOne() → findOneBy() is hundreds of callsites. One mechanical commit beats fifty thoughtful ones.
4. Keep the staging branch alive for a week. Let other PRs rebase onto it. The conflicts are real but they are bounded if the staging branch is fresh.
5. Pull in the "while we are in here" fixes deliberately. Big PRs are scary, but a major version upgrade is the one PR window where every file gets touched anyway. The marginal cost of an extra security fix is much lower in that window than in any other.

Sixteen distributions shared the same copy-pasted Terraform block.

CloudFront is AWS's content delivery network (CDN). A JSON Web Token (JWT) is the signed session string the API gives a user after login. curl is the command-line tool we use to send HTTP requests by hand. Three terms are enough to follow the finding.

CloudFront was handling TLS at the edge, HTTP/2 and HTTP/3 connections, DDoS protection, and branded error pages from a separate S3 bucket. The staging setup had been running for weeks. The distributions were stable, the error pages rendered correctly, and the TLS grades looked good.

The One-Line Proof

curl https://your-api.example.com/custom_error_pages/502.html \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.YOUR_ACTUAL_TOKEN"

If your response looks like this, you're vulnerable:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>InvalidArgument</Code>
  <Message>Unsupported Authorization Type</Message>
  <ArgumentName>Authorization</ArgumentName>
  <ArgumentValue>Bearer eyJhbGciOiJIUzI1NiJ9.YOUR_ACTUAL_TOKEN</ArgumentValue>
</Error>

Your full JWT (user ID, email, role, expiry, everything in the payload) reflected back in an S3 XML error response. The path exists on practically every CloudFront distribution with custom error pages. Any GET request with an Authorization header triggers it, no authentication and no origin failure required.

S3 reflects any Authorization scheme, not just Bearer. Basic auth credentials (Basic dXNlcjpwYXNzd29yZA==), API tokens (Token API_TOKEN_VALUE), anything non-AWS gets dumped into the XML. If your API uses any of these, the same leak applies.

Cookies, X-API-Key, and custom headers are forwarded to S3 too but not reflected in the XML. S3 only complains about the one header it tries to parse.

Why did CloudFront forward the Authorization header to S3?

A CloudFront distribution sits between users and one or more origins (the actual servers that hold your content). Each origin gets a behavior that tells CloudFront which requests to route there and what to forward along.

In this pattern, there are two origins behind one distribution:

The API origin is your application: an Application Load Balancer (ALB) in front of Elastic Container Service (ECS) containers, a Lambda function, or an EC2 instance. It needs the viewer's Authorization header to authenticate requests, Content-Type to parse bodies, and cookies for sessions. Forwarding everything to this origin is correct.

The error page origin is an S3 bucket containing three static HTML files: 502.html, 503.html, and 404.html. It serves the same branded "We'll be right back" page to every user regardless of who they are.

It needs nothing from the viewer: not their identity, not their cookies, not their query strings. It just needs CloudFront to ask for a file by path, and it returns the HTML.

The problem is that both origins were given the same origin request policy: the one designed for the API. So when CloudFront routes a request to the S3 error page bucket, it sends along everything the viewer included, including Authorization: Bearer <JWT>.

S3 is not your API. It doesn't understand Bearer tokens. It tries to interpret the Authorization header as AWS Signature Version 4, fails, and returns an XML error that includes the header value it couldn't parse. Your JWT is now in the response body.

The setup looks like this in Terraform (CloudFormation and the console follow the same pattern):

# Origin 1: Your API (ALB, ECS, Lambda, etc.)
origin {
  domain_name = aws_lb.api.dns_name
  origin_id   = "api"
}

# Origin 2: S3 bucket with static error pages
origin {
  domain_name = aws_s3_bucket.error_pages.website_endpoint
  origin_id   = "error-pages"
}

# When the API returns 502, serve the S3 error page instead
custom_error_response {
  error_code         = 502
  response_page_path = "/custom_error_pages/502.html"
}

# Behavior for the API (default): forwards all viewer headers
default_cache_behavior {
  target_origin_id         = "api"
  origin_request_policy_id = aws_cloudfront_origin_request_policy.AllViewerExceptHostHeader.id
}

# Behavior for the error pages: ALSO forwards all viewer headers
ordered_cache_behavior {
  path_pattern             = "/custom_error_pages/*"
  target_origin_id         = "error-pages"
  origin_request_policy_id = aws_cloudfront_origin_request_policy.AllViewerExceptHostHeader.id
  # ↑ THIS IS THE PROBLEM
}

The AllViewerExceptHostHeader policy tells CloudFront: "forward every header the viewer sent (Authorization, Cookie, X-Custom-Whatever) to the origin, except Host." That's correct for the API behavior, where your backend needs those headers. It's catastrophically wrong for the error page behavior, where the origin is an S3 bucket serving static HTML.

Three ways the token leaks

The error page path isn't linked anywhere. No browser navigates to /custom_error_pages/502.html during normal use. Three practical routes still reach it.

The path is trivially discoverable by scanners. It's in CloudFront's own documentation examples, and tools like ffuf, feroxbuster, and nuclei include /error/, /custom_error_pages/, and /502.html in their default wordlists. A scanner that hits the path and sees <Code>InvalidArgument</Code> with <ArgumentName>Authorization</ArgumentName> in the XML response can identify the misconfiguration from the shape alone. Our pentest tooling found this one that way.

A production outage can leak the token without any scanner. When the origin returns 502/503, CloudFront intercepts the error and fetches the S3 error page with the original request's Authorization header attached. The user didn't navigate to the error page; CloudFront sent them there. Their token leaks through a normal deployment, container restart, or scaling event, visible in the response body to any browser extension, corporate proxy, or compromised CDN edge watching that window.

Most of the distributions we audited had the plain-HTTP setting. The API behavior used redirect-to-https; the error-page behavior was a separate block with separate settings.

The Terraform Template Problem

We found this in a client's infrastructure codebase. We audited every CloudFront distribution across their staging and production accounts. The same ordered_cache_behavior block for /custom_error_pages/* showed up on distribution after distribution, all pointing to the same S3 error pages bucket, all using AllViewerExceptHostHeader.

The block was written once for the API distribution, where it was correct. Then it was copied to every other distribution (SPAs, admin dashboards, static sites, upload CDNs) because it "worked" and nobody questioned whether a static error page bucket needed to receive the viewer's Authorization header.

Terraform's copy-paste turned one mistake into a pattern across the whole account.

A handful of the affected distributions served authenticated traffic (APIs and legacy apps). Those were fully exploitable: every authenticated request that hit a 502 leaked the caller's JWT. The rest served static frontends that don't normally receive Authorization headers, but the direct-access path still worked on all of them.

The Chain That Made It Critical

The JWT leak alone is a HIGH: you can capture a token and replay it for the remaining access-token lifetime. But during this engagement, we found it chained with something else that pushed it to CRITICAL.

The API signs session tokens with a JWT secret. On staging, that secret was hardcoded as a trivially guessable value in the ECS task definition's plaintext environment block. Not in Secrets Manager, not pulled from SSM Parameter Store, just a short, dictionary-word string sitting in Terraform.

We confirmed the secret remotely without any internal access. The API returns different error messages depending on whether a token's signature verifies:

# Token signed with the guessable secret (correct):
curl -X POST https://api.staging.example.com/api/session/token \
  -H "Cookie: session=<forged-token-signed-with-guessable-secret>"
# → "Session has been revoked" (signature PASSED, DB hash lookup failed)

# Token signed with wrong secret:
curl -X POST https://api.staging.example.com/api/session/token \
  -H "Cookie: session=<token-with-wrong-signature>"
# → "Invalid session" (signature FAILED)

That error difference acts as an oracle. "Session has been revoked" means the JWT signature verified successfully; the server moved past cryptographic verification to the database lookup phase. "Invalid session" means the signature check failed. The two code paths are distinguishable remotely.

We proved the full chain end-to-end

We registered a test account and signed in. We captured the session token, leaked it through the error page, and validated it offline. Then we replayed it and had the account. All with curl.

Step 1: Sign in. We registered a test account and signed in. The server set an httpOnly session cookie. It was a long-lived JWT signed with the guessable secret.

Step 2: Leak the token. We sent a GET request to the error page path. The session token was in the Authorization header. The S3 XML error returned it verbatim. Byte for byte, it matched the original.

Step 3: Validate offline. We computed the HMAC-SHA256 signature using the guessable secret. It matched the token's signature exactly. No server interaction needed. The attacker now knows the token is real, who it belongs to, and when it expires.

Step 4: Replay. We sent the leaked session token to the token endpoint. The server verified the JWT signature (passed) and looked up the token hash (found a real active session). It minted a fresh access token.

Step 5: Account takeover. We used the fresh access token to call the user-profile endpoint. The server returned the full profile: ID, email, name, role, all PII fields. Full account access with no password. All from a token that was never meant to leave the httpOnly cookie jar.

We ran the entire chain with curl against a live staging environment. We didn't need source code access to exploit it. We only needed it to discover the hardcoded secret in Terraform. An external attacker reaches the same outcome by guessing the secret or by intercepting a token through the error page leak.

The public bucket

The bucket is also enumerable:

curl https://your-error-pages-bucket.s3.amazonaws.com/

Returns a full ListBucketResult XML with every file, size, ETag, and last-modified timestamp, no authentication required.

It's public because the error page origin uses an S3 website endpoint, which forces public read access. OAC (Origin Access Control) only works with the REST API endpoint. The pages themselves are benign HTML, but the bucket is one IAM policy drift away from attacker-writable: a phishing form in a 502 page would be served by your own domain, from your own CloudFront distribution, with your own TLS certificate.

The Fix

The error page behavior serves static HTML from S3. It doesn't need the viewer's auth token, cookies, query strings, or any other header.

When you omit origin_request_policy_id entirely, CloudFront sends only the minimum required headers to the origin. That is enough for an S3 website endpoint to fetch a file by path, and there is nothing left for S3 to reflect back.

The simplest fix is to delete the origin_request_policy_id line from every error page behavior. It shouldn't have been there in the first place.

But if you want to be explicit about the intent (and make it harder for someone to re-add a permissive policy later thinking it was accidentally removed), define a policy that forwards nothing:

resource "aws_cloudfront_origin_request_policy" "ErrorPagesNoAuth" {
  name    = "ErrorPages-NoAuth"
  comment = "Error pages are static HTML, no viewer headers needed"

  headers_config {
    header_behavior = "none"
  }
  cookies_config {
    cookie_behavior = "none"
  }
  query_strings_config {
    query_string_behavior = "none"
  }
}

Then in every ordered_cache_behavior for /custom_error_pages/*:

ordered_cache_behavior {
  path_pattern             = "/custom_error_pages/*"
  target_origin_id         = "error-pages"
  origin_request_policy_id = aws_cloudfront_origin_request_policy.ErrorPagesNoAuth.id
  viewer_protocol_policy   = "redirect-to-https"
  # ... rest unchanged
}

Verification:

# Before fix:
curl https://your-api.com/custom_error_pages/502.html \
  -H "Authorization: Bearer TEST"
# → <ArgumentValue>Bearer TEST</ArgumentValue>  ← LEAKED

# After fix:
curl https://your-api.com/custom_error_pages/502.html \
  -H "Authorization: Bearer TEST"
# → <!DOCTYPE html><html>...(your 502 page HTML)...  ← SAFE

For defense in depth:

• Switch the S3 origin from a public website endpoint to a private bucket with OAC. This closes the public listing vulnerability and eliminates the unencrypted HTTP origin protocol.
• Set error_caching_min_ttl = 0 so error responses aren't cached at all.
• Set viewer_protocol_policy = "redirect-to-https" on the error page behavior so the path can't be accessed over plain HTTP.

Forwarding headers to error pages is valid when the origin needs the viewer's identity: a Lambda@Edge function rendering personalized error pages, a Cognito-gated bucket, or a debugging service logging which user hit the error. In those cases the origin is an application that can safely consume auth headers. If your error page origin is S3, the answer is always header_behavior = "none".

What we shipped

Sixteen distributions closed. The JWT reflection path is gone across the client's account, and HTTP-to-HTTPS is enforced on every error-page behavior. The one residual risk (the public S3 website endpoint) is on the follow-up list.

How to Check If You're Vulnerable

Step 1: Find your error page path. Look for custom_error_response blocks in your CloudFront distribution. The response_page_path tells you where the error pages live.

Step 2: Send a request with an auth header.

curl -s https://your-domain.com/custom_error_pages/502.html \
  -H "Authorization: Bearer CHECK_THIS_TOKEN"

Step 3: Check the response. If you see <ArgumentValue>Bearer CHECK_THIS_TOKEN</ArgumentValue> in the XML, you're vulnerable. If you see your HTML error page, you're fine. A 403 means the path doesn't exist or the behavior doesn't allow GET, which is also fine.

Step 4: Check all your distributions. If you use the same Terraform module or copy-paste pattern across distributions, check every one. In one engagement we found the majority of distributions sharing the same misconfiguration. The one you check might be fine while a dozen others are leaking.

Check every distribution

Nobody asked whether a static HTML bucket needed the same headers as an authenticated API. If you're running CloudFront with custom S3 error pages, spend sixty seconds running the curl command above against each distribution.

A note on responsible testing: every test account created during this engagement was registered with a non-existent @example.com address, used only for curl-based proof-of-concept, and cleaned up after the findings were documented. No real user sessions were intercepted or replayed. The full chain was proven end-to-end using our own credentials against a staging environment with explicit authorization from the infrastructure owner.

Probably not often because unsubscription is now something that you have to legally think of, especially if you send marketing emails.

So how do you implement an unsubscribe link that is highly accessible, doesn't require any login, secure and unspoofable, and respect the RFC 8058 so that most email clients can automatically unsubscribe the user using your link? An email that respect your user so that they won't tap on that "Mark as spam" instead; and an email that won't trigger accidental unsubscribe because the email client crawls through every link to check for viruses.

JWT

JSON Web Token. A signed string with three parts (header, payload, signature) separated by dots.

HMAC · HS256

A keyed hash. A shared secret plus the payload produces a signature that only holders of the secret can create.

RFC 8058

The "one-click unsubscribe" standard that Gmail and Apple Mail support via the `List-Unsubscribe` and `List-Unsubscribe-Post` headers.

kid

A JWT header field naming which signing key was used, so a verifier can support key rotation — try current, then previous.

Possible Approaches

Random Token In The Database

The simplest approach would be to create a table that contains a token associated with an email that the user can use to unsubscribe.

CREATE TABLE unsubscribe_tokens (
  token uuid PRIMARY KEY,
  user_id uuid NOT NULL,
  email varchar NOT NULL,
  created_at timestamp NOT NULL
)

Generate a UUID per email, write the row, put the token in the URL. Endpoint gets hit, looks up the token, finds the user.

The problem with this approach is that you have to create a lot of them, and if you want to keep track of where the user unsubscribed from, you will create one token per email. For email that are sent 4 times a month for 100.000 users, that's 400.000 inserts on this table. By the end of the year, you will have 4.8M rows.

Signed Token In The Email

The alternative is to use the email itself as the storage of the token. A signed token that can't be spoofed and by default "distributed".

JWT is a good candidates for this, because we can sign it and then encode it as a link on the email.

The minimum fields that we need is just the email. That's it — of course you can add additional fields for tracking and other verifications. But you just need an email associated with the token and sign it and you're good to go.

We sign it with a shared HMAC secret — or if you're distributed and do key-pair verification, you can always do it too:

export function signEmailSubscriptionToken(
  payload: EmailSubscriptionTokenPayload,
): string {
  return jwt.sign(payload, EMAIL_SUBSCRIPTION_SECRET, {
    algorithm: 'HS256',
  });
}

And the URL that goes into the email footer looks like:

https://example.com/manage-email-subscriptions?token=<jwt-here>

Ideally, this will open a page that will allow the user to manage their subscriptions, and only execute the unsubcription via user interactions and not on page load. Otherwise, crawler might accidentally trigger unsubscriptions when this URL is accessed.

So you will need another endpoint, that accepts the token as well, but called with POST method.

Is This Safe?

There are three real threats to an unsubscribe URL:

Someone forges a URL to unsubscribe a stranger. The signature stops this. You cannot produce a valid token without our secret. Try to change the sub claim on a token you already have, and the JWT verification fails immediately. There is no way to forge a token that verifies without the secret. That is the whole point of signing.

A crawler enumerates URLs and unsubscribes everyone. The token is a random-looking base64 string that is different for every user. You cannot walk from one URL to the next. There is no ?userId=1234 you can increment to find the next one.

Someone steals a token from a leaked email and tries to escalate — use it to change the user's password, or log in as them. The thing that stops it is not the token — it is the endpoint.

The unsubscribe endpoint only knows how to do one thing, and that thing is "change this user's email subscription preferences." It does not accept requests to change the password. It does not accept requests to issue a session cookie. It does not accept requests to update the email address. So, even if the token leaks — say the email got forwarded, or someone got access to an old inbox — the worst outcome is that a stranger unsubscribes them from an email.

And the only line of code you need to write to verify the token is just this:

payload = jwt.verify(token, EMAIL_SUBSCRIPTION_SECRET);

RFC 8058

Gmail and Apple Mail have supported RFC 8058 for a while — the "one-click unsubscribe" standard. You set two headers on your outgoing email:

List-Unsubscribe: <https://example.com/one-click/unsubscribe?token=abc123>
List-Unsubscribe-Post: List-Unsubscribe=One-Click

The URL above can be the POST endpoint that you also use for the actual unsubscription instead of the manage subscription page.

And Gmail renders a small "Unsubscribe" button next to the sender name. When the user taps it, Gmail POSTs to your URL on the user's behalf. The user never opens the email. They just tap once and the mail client confirms "unsubscribed."

A few things worth knowing

Be careful with the expiration of the token, some jurisdiction might require you to have specific expiration date. The easiest would just to not set any expiration date. I know it feels natural to set expiresIn: '30d' on the sign call.

Keep the payload small. URLs in emails are already long. Do not stuff extra fields into the token unless you have to.

Do log the unsubscribe. Not for legal reasons (although also for those) — for debugging. If a user complains that they were unsubscribed and did not remember doing it, having a log of when the token was verified and from what IP is very useful.

Rotate your secret. The signing key is a secret. If it leaks, every unsubscribe URL you have ever sent is compromisable. A kid field in the JWT header lets you support multiple keys, and the verifier can try current, then previous.

Feel free to share if you have a different approach to unsubscribing users from emails without asking them to log in. This pattern has worked for us but there is definitely more than one way to do it.

We split the file into four sub-services behind a stable facade, landed the whole thing in one PR without changing a test, and wrote the procedure up as a reusable Claude skill for the next time.

Two terms before we go further

• Facade, in this post, is a class that exposes the same public methods as the old service did, but delegates each method to the right one of the new sub-services. The old import path keeps working; the internal shape moves.
• Sub-service is one of the new files the mega-service is split into. Each sub-service owns one slice of responsibility (querying, mutating, membership-state, stats) and is named after that slice.

The split we shipped

src/modules/user/
├── user.service.ts                   #  facade — was 1,307 lines, now ~120
├── user-query.service.ts             #  349 lines — read paths
├── user-mutation.service.ts          #  460 lines — write paths
├── user-membership.service.ts        #  288 lines — membership-state changes
├── user-stats.service.ts             #  307 lines — derived counts and aggregates
└── user.service.password-reset.spec.ts

The split ran along responsibility inside the service layer, not up through the controller and model layers. Reading a user and updating a user are different jobs on the same domain object, and they grow at different rates. On this service, the read methods stayed small and stable while the write methods kept picking up business rules.

Query — anything that returned a user (or a list of users) without mutating state. findById, findByEmail, findByCompany, searchByKeyword. The query service does not write.

Mutation — anything that wrote a user row (create, update, updatePreferences, softDelete, restore). The mutation service loads a row, mutates it, saves it.

Membership — methods that flip the user's membership state: upgrade tier, downgrade tier, transfer membership, attach to company, detach from company. The state transitions are tangled enough to justify their own file.

Stats — derived counts and aggregates. "How many users in this company are active," "how many memberships expire this month," "how many users by role." Reading across many rows is a different job from finding a single user.

A few methods did not fit any of the four. Some were doing two things and got split before they moved. Some were orchestration — they touched two or more sub-services in a single flow — and stayed on the facade.

The facade — keep the public API stable

The facade is the file every other module imports. Its surface area is identical to the old service: every method other modules called still exists, with the same signature.

// user.service.ts — the facade, after the split
@Service()
export class UserService {
  constructor(
    private readonly query: UserQueryService,
    private readonly mutation: UserMutationService,
    private readonly membership: UserMembershipService,
    private readonly stats: UserStatsService,
  ) {}

  // ─── Read methods delegate to query ─────────────────────────────────
  findById = this.query.findById.bind(this.query);
  findByEmail = this.query.findByEmail.bind(this.query);
  findByCompany = this.query.findByCompany.bind(this.query);
  searchByKeyword = this.query.searchByKeyword.bind(this.query);

  // ─── Write methods delegate to mutation ─────────────────────────────
  create = this.mutation.create.bind(this.mutation);
  update = this.mutation.update.bind(this.mutation);
  updatePreferences = this.mutation.updatePreferences.bind(this.mutation);
  softDelete = this.mutation.softDelete.bind(this.mutation);
  restore = this.mutation.restore.bind(this.mutation);

  // ─── Membership ─────────────────────────────────────────────────────
  upgradeMembership = this.membership.upgrade.bind(this.membership);
  downgradeMembership = this.membership.downgrade.bind(this.membership);
  attachMemberToCompany = this.membership.attachToCompany.bind(this.membership);

  // ─── Stats ──────────────────────────────────────────────────────────
  countActiveByCompany = this.stats.countActiveByCompany.bind(this.stats);
  countExpiringMemberships = this.stats.countExpiringMemberships.bind(this.stats);

  // ─── Orchestration methods stay here ────────────────────────────────
  // Methods that genuinely coordinate across two or more sub-services.
  async passwordResetFlow(email: string, newPassword: string) {
    const user = await this.query.findByEmail(email);
    if (!user) return { success: false };

    await this.mutation.updatePassword(user.id, newPassword);
    await this.stats.recordSecurityEvent(user.id, "password-reset");
    return { success: true };
  }
}

The bind-and-assign shape looks unusual on first read, but each method on the facade is the same function object as the one on the sub-service — no wrapper indirection, no extra logging. userService.findById(42) runs with the same overhead it always had.

Orchestration methods stay on the facade because they touch several sub-services in one flow. passwordResetFlow above hits the query, mutation, and stats services in sequence, so it belongs to none of them alone.

How the split landed in one PR

The PR added around sixteen hundred lines and deleted around twelve hundred. Most of it was file moves. The tests did not need to change: because the facade's public API is byte-identical to the old service, every test that imported UserService kept passing untouched. That is the safety net the whole refactor rides on.

Three disciplines made the PR reviewable.

One sub-service per commit. Four code commits, one per sub-service, plus a fifth that thins the facade. The reviewer reads one sub-service at a time.

Method-by-method migration. Each method moved with its tests. If a test in the suite was targeting a single method, that test stayed near the method, in the same commit as the move.

No behavioural changes. Every method moved is byte-identical to what was in the old service. Behavioural refactors — N+1 fixes, added logging, tighter authorization — go in follow-up PRs after the split lands.

The reusable skill — service-facade-refactor

The skill file lives at .claude/skills/service-facade-refactor/SKILL.md in this codebase and in our internal skill library. The next mega-service does not need to re-derive any of this.

A Claude skill is a small Markdown file documenting a reusable procedure in enough detail that an LLM-assisted contributor can apply it without having lived through the original work.

The skill in full:

# Service Facade Refactor

A pattern for splitting a single mega-service file into multiple
responsibility-tagged sub-services behind a stable public facade,
without breaking any caller.

## When to use this skill

A single service file has crossed roughly 800 lines, the team is
avoiding it on PRs, and the file mixes more than one of:
read paths, write paths, state-machine transitions, derived stats.

If the file is large but coherent (one responsibility, just verbose),
this is not the right skill — use a method-extraction refactor instead.

## Inputs

- The mega-service file (path).
- The full test suite (must be green before starting).
- One hour of uninterrupted reading time, before any code moves.

## Steps

1. **Tag every method on the existing service.** Read top-to-bottom.
   Each method gets exactly one tag:
   - **Read** — returns the entity, no writes.
   - **Write** — mutates the entity, may read first.
   - **State machine** — tier upgrades, status transitions, lifecycle.
   - **Stats** — counts, aggregates, derived values across many entities.
   - **Orchestration** — coordinates across two or more of the above.

   If a method does not fit one tag, it is doing two things; split
   the method *before* you split the service.

2. **Each non-orchestration tag becomes a sub-service.**
   File naming: `<domain>-<tag>.service.ts`. Example: `user-query.service.ts`.

3. **Move methods one tag at a time, in their own commit, with
   their tests.** The test file moves with the method, or, if the
   tests are integration-style, stay in place and reference the
   sub-service through the facade.

4. **Rewrite the original service file as a facade.**
   - For each method on a sub-service, `bind`-and-assign on the facade:
     `findById = this.query.findById.bind(this.query);`
   - For orchestration methods (touch two or more sub-services), keep
     the implementation on the facade.

5. **Run the full test suite. It should pass with no test changes.**
   The facade's public API is byte-identical to the original service.

6. **Land as a single PR with one commit per sub-service plus one
   for the facade.** Behavioural changes (N+1 fixes, new logging,
   tightened authz) are *follow-up PRs*, not part of the split.

## Output shape

- One facade file, exposing the same methods the original did.
- Four (or fewer) sub-service files, each named by its tag.
- Zero test changes.
- A single PR, multi-commit, readable one sub-service at a time.

## What this is not

- Not a layered split (controller / service / model). The split is
  by responsibility within the service layer, not by layer.
- Not a behavioural change. Save the perf and authz fixes for after.
- Not appropriate for files under ~500 lines. The overhead of the
  split is worse than the readability win at that size.

Step 1 is where the split gets decided. The tagging walks top-to-bottom on five yes/no questions, and the first "yes" wins.

The skill works across codebases because it does not name a specific domain. Any TypeScript codebase with service classes and a mega-service problem can apply it the same way.

The judgement calls in the skill took longer to write than the steps. The "if the file is large but coherent, this is not the right skill" line at the top and the "what this is not" section at the bottom are what keep the skill from getting misapplied. A skill that only describes the happy path gets used wrong the first time somebody reaches for it.

The next mega-service on this codebase — finance.service.ts, around eleven hundred lines — is queued up for the same treatment.

The week before Ramadan, as it is the company's custom, Amina booked us a table for iftar at Apetit, the food spot everyone in Sarajevo recommends. It sits on Vilsonovo šetalište, the riverside promenade locals call Vils, five minutes from most of our apartments.

svejedno

whatever, fine with me

može

works for me

ekipa

the crew, our people

fakat

honestly, for real

meni

for me / put me down for

pa kad je iftar

well, whenever iftar is (the punchline)

Here is the thread that got us there, translated and lightly compressed:

A few things in that thread stayed with me.

"Svejedno."

Half the thread is people saying svejedno. In this team, that word means trust. Amina booked the restaurant, picked the menu options, chased the head-count. Nobody in the thread felt the need to second-guess any of it. When one of us takes on the coordination work, the rest of us get out of the way. And we take any excuse to sit at the same table — Clearview picks up the tab, and a distributed team never says no to a paid-for meal.

The same pattern shows up on client work. When someone picks up a task, the rest of the team stays out of the way. We ask when we have something to add, and we say svejedno — or the English equivalent, "sounds good, go for it" — when we don't.

Sometimes the person out front is the senior engineer on a client sprint. This time it was Amina, the one who knows every food spot in Sarajevo.

"Pa kad je iftar."

Amina's reply to my what-time question assumes that everyone at the table already knows when iftar is. A non-Bosnian reader would probably miss it.

Iftar is the meal Muslims eat to break the daily fast during the month of Ramadan. It happens at sunset, which means the exact time slides ten to fifteen minutes later every week as the days get longer. In Sarajevo in mid-March, sunset was somewhere around 6:15 pm. If you knew the calendar, you knew the time. If you did not, you asked, and Amina answered. Because she is the boss.

The cultural detail did not have to be spelled out in the thread. Everyone on the invite already got it.

Time for meza

The corporate menu did land in the end. The five of us who made it — Amina, Aid, Adnan, Ahmed, and me — ate at Apetit as the sun set behind the hills over the Miljacka. Redzy joined for part of it. There were the standard meza, the Bosnian word for the little starter plates other cuisines call mezze or mezes. In Bosnia, meza is the part of the meal where the real talking happens, where a table settles in and stops being in a hurry. Then came the mains, and then someone ordered Bosanska kafa and the conversation wandered, as it always does, into a mix of shop talk and family news.

Nobody talked about "team building."

We had not all sat at the same table since the last one, and we would probably not sit at the same table for another couple of months. That is the rhythm of a distributed team — long stretches of async work broken by a shared meal every few months.

Why the dinner is worth it

For a distributed team, in-person meals matter more than they look. The grand summit in Bali counts — Clearview gathered in Ubud in January — but the small local dinner counts too, and it happens more often. The one where Amina's message goes out, five people RSVP within the day, and the reservation goes under her name because she is the one who called the restaurant. Typical Amina. She knows the best places to eat in Sarajevo, and the rest of us are smart enough to show up where she tells us to.

Every time we do one of these, the async work that follows for the next few weeks is smoother. Threaded replies come with more benefit of the doubt, less second-guessing, nobody a stranger. If you have never worked distributed, that is the reason to spend real money on getting people in a room. It makes the rest of the year work better.

We are lucky, in Sarajevo, that most of the Clearviewers here can be in the same city inside of an hour. Having enough of us in one city to fill a table on short notice is a real gift. When Ahmed messages that he doesn't like dolma and sarma, or Amina jokes that we should all know when iftar is, that only feels normal because we have eaten together before.

Nedim  ·  Sarajevo  ·  March 2026

Two terms before we go further:

• CORS — the browser's rule about which other websites are allowed to send authenticated requests to your API. Mis-set, it lets any site act on a logged-in user's behalf.
• JWT — JSON Web Token. The short signed string the API hands a user after they log in. Every subsequent request carries it as proof of who the user is. The token has an audience field that names which application the token was minted for, and the API can refuse a token presented to the wrong audience.

Two independent controls — the stricter one held while the weaker one was broken.

The CORS Misconfiguration

The Express CORS middleware allowed any subdomain through a regex:

// cors.loader.ts
const pattern = /^https:\/\/([a-z0-9-]+\.)?client\.example\.org$/;

This matches any subdomain, including ones that do not exist yet, ones that could be registered by attackers, and ones that could be taken over via a dangling DNS record. We confirmed the breadth with a small matrix:

# Evil subdomain — accepted
curl -sI "$API/api/config" \
  -H "Origin: https://evil.client.example.org"
# access-control-allow-origin: https://evil.client.example.org
# access-control-allow-credentials: true

# Punycode subdomain — accepted
curl -sI "$API/api/config" \
  -H "Origin: https://xn--evil.client.example.org"
# access-control-allow-origin: https://xn--evil.client.example.org

# Numeric prefix — accepted
curl -sI "$API/api/config" \
  -H "Origin: https://123evil.client.example.org"
# access-control-allow-origin: https://123evil.client.example.org

CORS with credentials: true, wildcard subdomains, and DELETE listed in the preflight Access-Control-Allow-Methods. Every box on the checklist of "trivially exploitable CORS misconfiguration."

We were about to write HIGH — any subdomain can steal authenticated session data. We tried to actually exploit it first.

The Exploit That Failed

For the exploit to work in a real browser, the attacker needs to make authenticated requests from their evil subdomain. With JWT auth (Authorization header, not cookies), the attacker first has to obtain a token from the evil origin — either by signing in there, or by stealing a token from the legitimate frontend's localStorage and replaying it.

We tried signing in from the evil origin:

curl -s -X POST "$API/api/auth/signin" \
  -H "Content-Type: application/json" \
  -H "Origin: https://evil.client.example.org" \
  -d '{"email":"test@example.org","password":"Secret123"}'

{ "success": false, "message": "Unknown origin" }

CORS accepted the origin. The API rejected the sign-in. Something independent of CORS was checking.

The JWT Audience Map

Digging into jwt.config.ts surfaced this:

const audienceMap: Record<string, string> = {
  // Production
  'https://app.client.example': 'main-app',
  'https://admin.client.example':         'admin-app',
  'https://partners.client.example':  'partner-portal',
  // Staging
  'https://app.staging.client.example': 'main-app',
  'https://partners.staging.client.example':  'partner-portal',
  'https://admin.staging.client.example':         'admin-app',
  // API-internal calls
  'https://api.staging.client.example': 'api-internal',
};

export function resolveAudience(origin: string | undefined): string | null {
  if (!origin) return 'api-internal';
  return audienceMap[origin] ?? null;  // null = rejected
}

Seven explicit origins. Anything else returns null, which the calling code translates into a 401 Unknown origin. The CORS policy says "any subdomain is fine." The JWT system says "only these seven specific origins get tokens." Two access-control systems with conflicting policies — and the stricter one wins.

Testing Every Authenticated Path

We needed to know whether the audience check covers every authenticated operation or just sign-in:

# 1. Get a token from a legitimate origin.
TOKEN=$(curl -s -X POST "$API/api/auth/signin" \
  -H "Origin: https://partners.staging.client.example" \
  -d '{"email":"test@example.org","password":"Secret123"}' \
  | jq -r '.data.access_token')

# 2. Try the token from the evil origin.
curl -s "$API/api/auth/me" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Origin: https://evil.client.example.org"
# → "Unknown origin"

# 3. Try with no Origin header at all.
curl -s "$API/api/auth/me" \
  -H "Authorization: Bearer $TOKEN"
# → "Audience mismatch"
#   (token has aud=partner-portal, no origin resolves to api-internal, mismatch)

# 4. Try with a different legitimate origin.
curl -s "$API/api/auth/me" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Origin: https://app.staging.client.example"
# → "Audience mismatch"
#   (partner-portal ≠ main-app)

Every path blocked. The audience check fires on every authenticated request, not just sign-in. A token minted for partner-portal only works when sent with the partners.staging.client.example Origin header — any other origin (evil, no-origin, different legitimate origin) returns 401.

The Refresh Token Path

Token refresh has the same check:

curl -s -X POST "$API/api/auth/refresh-token" \
  -H "Origin: https://evil.client.example.org" \
  -d "{\"refresh_token\": \"$REFRESH\"}"
# → "Unknown origin"

The attacker cannot refresh a stolen token from an evil origin either.

What Is Still Vulnerable

The audience check only runs on authenticated endpoints. Public endpoints do not need it, and do not get it:

# Public config — readable from any CORS-accepted origin.
curl -s "$API/api/config" \
  -H "Origin: https://evil.client.example.org"
# 200 OK — feature flags returned

# Public catalog — same shape.
curl -s "$API/api/catalog/" \
  -H "Origin: https://evil.client.example.org"
# 200 OK

Public data is readable from any origin that the CORS policy lets through. The data itself is not sensitive — feature flags and public catalog entries — so the impact is low. The client's product owner confirmed this explicitly.

The real remaining risk is XSS on a legitimate subdomain. If an attacker achieves XSS on, say, partners.staging.client.example, they can read the JWT out of localStorage and use it from script running on that legitimate origin. Neither control fires. The audience check sees a legitimate origin and CORS sees an approved one. The two controls only stop arbitrary origins, not legitimate origins compromised via a different bug.

The Severity Downgrade

We originally drafted the CORS wildcard as HIGH. After the exploitation attempt, we downgraded it to MEDIUM, with the residual risk path documented explicitly:

We still recommended the CORS fix, and it landed in the same sprint. Defense in depth means not relying on one independent control to do the work of two. Without the audience map, the CORS bug would have been trivially exploitable. With it, an attacker first needs script execution on a legitimate origin (XSS on partners.staging.client.example or similar), which is a much higher bar.

Why Two Independent Controls Worked

The CORS policy and the JWT audience map were written at different times by different people for different reasons. CORS was set up early in the project, when the team was thinking about subdomain flexibility. The audience map was added later, when multi-portal authentication came in. Neither was deliberately a "backup" for the other.

The two controls do not need to know about each other. CORS asks whether the browser is allowed to send the request; the audience check asks whether the token is valid for this origin. Different questions, enforced in different places. When one is weaker, the other holds — which is what happened here.

Two things this engagement changed for us

First: try to exploit a finding end to end before assigning severity. The path from "this check is missing" to "I can steal user data" often has unexpected obstacles, and the severity should reflect the real impact, not the impact of the broken control in isolation.

Second: unify the CORS allowlist and the JWT audience list into one source-of-truth file, two readers. When the two lists drift, defense in depth becomes defense in name only. On this engagement, we consolidated both into a single origins.config.ts at boot; the CORS middleware and the audience resolver now read from the same list.

The residual risk (XSS on a legitimate subdomain) went into the report explicitly, because a layered defense that holds against arbitrary origins still needs the reader to know what would open the attack.

Our first onboarding didn't treat the permission with that much respect. We did it the way most apps do, at some point in the flow we called the API, the iOS dialog popped up with no context, and the user had to decide on the spot why an app they installed five minutes ago wants to send them things. Predictably, a lot of them tapped "Don't Allow".

We assumed we could ask again later, once the user had seen enough of the app to change their mind. That assumption was wrong, and it's the reason this post exists. On iOS the native prompt is a one-shot, once the user says no, calling requestPermissionsAsync() again just returns "denied" without showing anything. The only recovery is the user going to Settings, Notifications, Your App and flipping the switch by hand, and very few people who decline ever do that.

So every "Don't Allow" was a permanent opt-out from a core part of the product, which made the decline rate the most expensive metric in the whole onboarding.

Let me show you what we did instead, the screen we put in front of the prompt, the hook that guards the request, and what we're still measuring.

What the iOS docs don't tell you

The native prompt is a one-shot, that part is documented. What isn't documented as prominently is when it becomes a one-shot. It's not the first time you call requestPermissionsAsync(), it's the first time the user sees it. Once they've seen the prompt and declined, you can't show it again from your app code.

You can detect the state with getPermissionsAsync():

import * as Notifications from "expo-notifications";

const { status } = await Notifications.getPermissionsAsync();
// "granted" | "denied" | "undetermined"

"undetermined" is the only state where calling requestPermissionsAsync() will actually show the system prompt. "denied" means the user already saw it and said no, and your only recovery is the Settings deep link. "granted" means you're clear to send notifications.

The whole pattern in this post is built on one rule: never call request while the status is undetermined unless you're confident the user will say yes.

Timing: a lesson from game design

Mobile games solved this years ago. A well-designed game never asks for notification permission on first launch, it waits until you've finished the first level, earned your first reward, or started a building that takes an hour to complete. At that moment the notification has an obvious job, "we'll tell you when it's done", and the player has already felt the value, so the ask answers a question they actually have.

The worst time to ask is before the user has experienced anything, at that point the permission prompt is a cost with no visible benefit and "Don't Allow" is the rational answer. The best time is right after a moment of value, when the notification is clearly in service of something the user just chose.

That's why our screen sits where it does in the onboarding. The user has already entered their name, picked their goals and invested a few minutes in the flow, so the notification ask is framed as the thing that makes those goals happen. Same principle as the game asking after the first level, earn the moment first, then ask.

The screen we show before the prompt

The screen we shipped is one step in the onboarding flow, right after the user has entered their name, picked their goals and invested a few minutes in the flow. It's deliberately simple, a title, a short description and two buttons:

<NotificationExplainerScreen
  title="Stay on track"
  description="We'll send you gentle reminders for your daily habits, lab result updates the moment they're ready, and progress milestones worth celebrating. One nudge at the right time, never spam. You're in control, turn off or customize every notification type anytime in Settings."
  primaryAction={{
    label: "Turn on notifications",
    onPress: requestNotificationPermission,
  }}
  secondaryAction={{ label: "Skip for now", onPress: advanceOnboarding }}
/>

Every line of that copy is doing a job, and most of the jobs come straight from game design.

"Gentle reminders for your daily habits" names the concrete thing the user just set up, the goals from two screens ago, so the notification is in service of something they chose. "Progress milestones worth celebrating" is the reward loop, games learned a long time ago that people say yes to notifications about wins, a level completed, a streak kept, much more readily than to notifications about obligations. "Lab result updates the moment they're ready" gives the notification an obvious job, the same way a game asks for permission when your building has an hour left, we'll tell you when it's done.

And "you're in control, turn off or customize anytime in Settings" lowers the perceived cost of saying yes. Good games always let you toggle notification types individually, and telling the user up front that they keep that control makes "Turn on notifications" feel reversible instead of a commitment.

Then there's "Skip for now", and this one needs explaining because it looks like a mistake. It isn't. Skipping our screen costs nothing, the permission status stays undetermined, so we can ask again later at a better moment, after the user's first completed habit or their first lab result. Declining the OS prompt costs everything, it burns the one-shot permanently. So the skip button is there on purpose, we'd much rather collect a cheap "not now" on our own screen than push an unsure user into an expensive, irreversible "Don't Allow" on Apple's.

The hook

The screen calls a single hook on continue:

// hooks/useNotificationSetup.ts — abridged
import * as Notifications from "expo-notifications";

export function useNotificationSetup() {
  const requestNotificationPermission = useCallback(async () => {
    const current = await Notifications.getPermissionsAsync();

    if (current.status === "granted") {
      // Already on. Skip the prompt; advance the onboarding.
      return "granted";
    }

    if (current.status === "denied") {
      // Already declined. Show the "open Settings" sheet instead of the prompt.
      showSettingsSheet();
      return "denied";
    }

    // status === "undetermined" — the one case where the prompt will actually fire.
    const result = await Notifications.requestPermissionsAsync({
      ios: {
        allowAlert: true,
        allowBadge: true,
        allowSound: true,
        allowAnnouncements: false,
      },
    });

    if (result.status === "granted") {
      await registerForPushToken();
    }

    return result.status;
  }, []);

  return { requestNotificationPermission };
}

Three guards in three lines. "Granted" skips the prompt entirely, "denied" opens the Settings deep-link sheet so the user can recover, and "undetermined" is the only path that calls the OS request.

Asking for the four iOS flags explicitly is worth doing. We didn't enable allowAnnouncements, that's the Siri "read this aloud through AirPods" permission and the product doesn't need it. Asking for permissions you don't need just nudges the user toward decline.

The Settings sheet, for the users who already said no

We show a separate small sheet for users whose status is already "denied":

<BottomSheet
  title="Notifications are off"
  body="You said no the first time, which is fine. The only way to turn them on now is in your phone's Settings."
  primaryAction={{
    label: "Open Settings",
    onPress: () => Linking.openSettings(),
  }}
  secondaryAction={{ label: "Maybe later", onPress: dismiss }}
/>

The copy matters here. "You said no the first time, which is fine" respects the user's earlier choice, and "the only way to turn them on now is in your phone's Settings" is the honest mechanic, not a sales pitch. The share of declined users who open Settings from this sheet is meaningfully higher than the share who responded to the previous "Notifications are off, enable them" banner inside the app shell.

The measurement

We're tracking three numbers, weekly.

Our screen's tap rate. What share of users who land on our screen tap "Turn on notifications".

Permission grant rate after the CTA. Of the users who tap our CTA, how many then tap "Allow" on the system prompt that follows.

Net grant rate. Our screen's tap rate times the system-prompt grant rate. The lift over the bare system-prompt baseline is the metric the product team watches, and on this product it's meaningful, meaningful enough that showing our own screen first is now the model for every new permission ask, not just notifications.

What I would tell another engineer setting this up

1. Never call requestPermissionsAsync without checking the status first. The one-shot rule on iOS is unforgiving, the three-branch guard is one of those patterns where the lines you don't call matter more than the ones you do.
2. Write the screen copy around concrete, positive jobs. Habit reminders the user just set up, lab results the moment they're ready, milestones worth celebrating. Specific and rewarding beats "stay engaged with the app", the same reason game notifications lead with wins, not obligations.
3. Let users skip your screen, never let them skip into the OS prompt. A "Skip for now" on your own screen is cheap, the status stays undetermined and you can ask again at a better moment. A "Don't Allow" on the system prompt is permanent. Design the screen so the unsure user takes the cheap no.
4. Settings deep link for the declined cohort. Most users who declined the first time won't come back, some will, so make the recovery a one-tap path.
5. Ask for the iOS flags you actually need. allowAnnouncements isn't free, every permission you ask for raises the bar to "yes".