Nedim Hadzimahmutovic · Remote Since Forever

Closing the Rate-Limit Bypass: A CloudFront + ALB Case Study

2026-09-14T11:00:00+02:00

Three weeks before a client's public launch, we were reviewing the security of their Express API — the usual sweep of every endpoint the public would be able to reach. The login rate limiter was already in place. The intent was right. The code worked. And yet, with one HTTP header on each request, an attacker could make unlimited login attempts and the limiter would count every one as a different visitor.

One line of curl did it — the command-line tool engineers use to send HTTP requests by hand, the same kind your browser sends when it loads a page:

curl -H "X-Forwarded-For: 203.0.113.7" https://api.client.example/auth/signin

The -H flag attaches a header to the request. X-Forwarded-For is the header proxies use to record the visitor's real IP address as the request travels through them. We will come back to why setting it by hand breaks the limiter.

The rest of this post is the reproduction recipe, the root cause, and the fix that closed the gap without breaking real traffic.

The Client Architecture

The intended request path looked like this:

Client → CloudFront → ALB → Express (Node.js)

What each of those four boxes actually is:

Client — the visitor's browser, or an attacker's terminal.
CloudFront — AWS's content delivery network. It sits at the edge of the internet, caches static files, and forwards everything else inward.
ALB — Application Load Balancer. AWS's traffic distributor. Takes requests from CloudFront and routes them to whatever application server is healthy.
Express (Node.js) — the application itself. The login endpoint and the rate limiter live here.

Two proxy hops between the visitor and the application. Each proxy prepends — adds to the front of — the X-Forwarded-For header. CloudFront writes the visitor's real IP at the front first, then the ALB writes CloudFront's IP at the front of that. By the time the request reaches Express, the header reads:

X-Forwarded-For: <real-client-ip>, <cloudfront-edge-ip>

Express needs the real client IP for rate limiting. That is what trust proxy exists for:

// express.ts:12
app.set('trust proxy', 2);

trust proxy: 2 tells Express that there are exactly two trusted proxies between the visitor and the application, and to skip the last two IPs in the X-Forwarded-For header (XFF for short) to find the real one:

XFF: <real-client>, <cloudfront>
         ↑              ↑
      req.ip         skip (proxy 1)
                    ALB adds its own (proxy 2, not in XFF)

Express reads the real client IP. The rate limiter counts per IP. Everything works — until you skip the proxy chain entirely.

What We Caught

The first signal was the ALB responding directly:

curl -v "$API/api/config" 2>&1 | grep -iE "server|x-cache|via"
# server: awselb/2.0

server: awselb/2.0 with no x-cache or via header. The request had reached the ALB without going through CloudFront. That changes the math. When traffic skips CloudFront, there is only one proxy hop, not two.

Direct to ALB (1 hop):
  XFF: <attacker-injected-ip>
           ↑
        Express reads this as req.ip
        (skips 2, but there is only 1 real proxy)

With trust proxy: 2, Express trims two entries from the right of X-Forwarded-For. There was only one real proxy. So Express reached into the part of the header the attacker controls and used whatever the attacker put there as the source IP.

We confirmed the rate limiter worked normally first:

# Five sign-in attempts from the same IP — rate limited correctly
for i in $(seq 1 5); do
  curl -s -o /dev/null -w "%{http_code} " -X POST \
    "$API/api/auth/signin" \
    -H "Content-Type: application/json" \
    -d '{"email":"test@example.com","password":"wrong"}'
done
# 429 429 429 429 429

Then with rotating fake IPs:

# Five attempts with rotating X-Forwarded-For — limiter sees five different "users"
for i in $(seq 1 5); do
  curl -s -o /dev/null -w "%{http_code} " -X POST \
    "$API/api/auth/signin" \
    -H "Content-Type: application/json" \
    -H "X-Forwarded-For: 10.0.$((RANDOM % 255)).$((RANDOM % 255))" \
    -d '{"email":"test@example.com","password":"wrong"}'
done
# 400 400 400 400 400

400s, not 429s. Every request reached the sign-in logic. The rate limiter saw each request as a different IP because Express trusted the attacker's X-Forwarded-For. Each fake IP carries its own counter, and no counter ever hits the threshold.

Brute force went through unimpeded. We reproduced it in staging and in production.

Why the ALB Was Reachable At All

The ALB security group was supposed to restrict inbound traffic to CloudFront only. The intended rule was present:

ingress {
  description = "lb_cloudfront_https_ingress_only"
  from_port   = 443
  to_port     = 443
  protocol    = "tcp"
  prefix_list = [data.aws_ec2_managed_prefix_list.cloudfront.id]
}

But three other rules were also present, left over from an earlier migration:

ingress {
  from_port   = 80
  to_port     = 80
  protocol    = "tcp"
  cidr_blocks = ["0.0.0.0/0"]
}

ingress {
  from_port   = 443
  to_port     = 443
  protocol    = "tcp"
  cidr_blocks = ["0.0.0.0/0"]
}

ingress {
  from_port   = 444
  to_port     = 444
  protocol    = "tcp"
  cidr_blocks = ["0.0.0.0/0"]
}

Ports 80, 443, and 444 open to the world. Security groups are additive — if any rule allows traffic, it is allowed. The CloudFront-only rule was correct. It was also useless, because the permissive rules let everyone in alongside it.

The Fix Isn't Dropping `trust proxy`

The reflex when a trust proxy: 2 system gets bypassed is to drop the number:

app.set('trust proxy', 1);  // We did NOT do this.

That breaks the rest of the stack. With trust proxy: 1 and legitimate traffic going through CloudFront + ALB (two real hops), Express would skip only one proxy and read CloudFront's edge IP as the source. Every user behind the same CloudFront edge node would share a single rate-limit counter. One user fails to log in five times, the entire region gets locked out.

The real fix is upstream: close the ALB to direct traffic and re-enforce the assumption that two-hop is always the path.

# Keep only this ingress rule for HTTPS to the ALB.
ingress {
  description = "CloudFront prefix list only"
  from_port   = 443
  to_port     = 443
  protocol    = "tcp"
  prefix_list = [data.aws_ec2_managed_prefix_list.cloudfront.id]
}

# Remove every 0.0.0.0/0 ingress rule.

After the apply, a direct curl against the ALB DNS name gets a connection refused. All traffic flows through CloudFront. trust proxy: 2 is correct again. The rate limiter counts the right IP.

Defense in Depth — What We Added Alongside

We did not stop at the security group fix. Two further layers went in alongside it, so that a future regression does not, on its own, re-open the bypass.

1. CloudFront custom-header validation at the app layer

CloudFront can sign every origin request with a custom header. The application then refuses to serve any request that does not carry it.

# CloudFront distribution origin config
custom_header {
  name  = "X-CloudFront-Secret"
  value = "<long-random-string>"
}

// Express middleware, runs before route handlers
app.use((req, res, next) => {
  if (req.headers['x-cloudfront-secret'] !== process.env.CF_SECRET) {
    return res.status(403).send('Direct access not allowed');
  }
  next();
});

Even if the security group regresses, non-CloudFront traffic is rejected at the application layer. The secret is rotated quarterly via Terraform and a CloudFront cache invalidation.

2. Account-based rate limiting on auth endpoints

Per-IP rate limiting assumes one person per IP. That assumption is fragile — corporate NATs, mobile carriers, and shared offices all break it. We added a second limiter keyed by email address on the sign-in endpoint:

const loginLimiter = rateLimit({
  keyGenerator: (req) => req.body.email || req.ip,
  max: 5,
  windowMs: 15 * 60 * 1000,
});

Now even if the attacker manages to rotate source IPs through some other path, the target account is protected after five attempts. The legitimate user gets a clear "too many attempts" response and an account-recovery email instead of a silent brute force in the background.

What We Shipped Back

We did not hand the client a thirty-page report. We handed them three things, all merged in the same sprint as the finding:

The reproduction recipe above, copy-pasteable, run against staging.
The Terraform diff that removes the 0.0.0.0/0 ingress rules.
The middleware diff that adds the CloudFront-secret check.

We re-ran the reproduction after the apply landed and got 429s back across the board. End-to-end from "this is broken" to "this is closed" was four working days — and on the client's side, the only ask was "can we point you at the staging environment for a day?" The rest is what they pay us for.

The Checklist We Carry Forward

Five checks we now run on every Express + ALB + CloudFront engagement.

trust proxy: N is only correct if there are always exactly N proxies between the client and the application. Any path that reduces the hop count makes the source IP attacker-controlled.
The CloudFront-prefix-list ingress rule is not a filter on its own. It is one allow-rule alongside whatever else is in the security group. Add it, and remove every 0.0.0.0/0 rule that competes with it.
Sign CloudFront origin requests with a header secret and check it in Express middleware. A defense-in-depth measure that survives security-group regressions.
Rate-limit auth endpoints by account, not just by IP. IPs lie. Account identifiers do not.
Reproduce the bypass from a fresh shell before assigning severity. The reproduction recipe is what a fix gets validated against — it is worth writing carefully even when the conclusion is obvious.

How To Check Your Own Setup

If you are running the same shape — Express behind CloudFront and an ALB — three commands tell you whether you have the same gap:

# 1. Is the ALB reachable directly?
curl -sI "https://api.client.example/health" | grep -iE "server|via|x-cache"
# server: awselb/2.0  →  yes, you are hitting the ALB directly

# 2. Does X-Forwarded-For rotation slip past rate limiting?
for i in $(seq 1 10); do
  curl -s -o /dev/null -w "%{http_code} " \
    -H "X-Forwarded-For: 10.0.$((RANDOM%255)).$((RANDOM%255))" \
    "https://api.client.example/api/auth/signin"
done
# 429 every time = fine.  Mix of 400/401 = problem.

# 3. Inspect the ALB security group
aws ec2 describe-security-groups --group-ids sg-xxx \
  --query 'SecurityGroups[].IpPermissions[].IpRanges[].CidrIp'
# 0.0.0.0/0 on port 443 = the CloudFront-only rule is doing nothing.

Common AWS WAF rate-limit questions

Four questions we hit while working through this — each answered against the same composite-key rule pattern from the fix section.

How do I rate-limit per hostname on AWS WAF?

Add HTTP_HEADER: Host as one of the aggregation keys in a RateBasedStatement CustomKey. The rule then counts requests per distinct Host: value instead of per source IP, so api.example.com and admin.example.com behind the same WAF get independent budgets. Combine with a ForwardedIP key for a per-host + per-client counter.

Can AWS WAF rate-limit on a 1-minute window instead of 5?

Not directly — RateBasedStatement supports a fixed set of evaluation windows (60, 120, 300, 600 seconds as of 2026). EvaluationWindowSec: 60 gets you the 1-minute window; the older 5-minute-only limit was removed. If you need a sub-minute window, that is Lambda-at-edge territory, not WAF.

How do I rate-limit by URL path segment?

Use Scope-down statement on the rate-based rule with a ByteMatchStatement on URI_PATH (starts-with, contains, or a regex). The rate rule only counts requests matching the scope-down predicate, so /api/auth/* and /api/search/* get separate budgets. Composite keys on HTTP_HEADER: Host + path prefix let a single WAF instance run distinct rules per route family.

How do I exclude a known good IP from an AWS WAF rate rule?

Two shapes. (a) An IPSet rule that allow-listed IPs (your office egress, a partner's server pool) evaluated before the rate rule with Action: Allow and Priority set lower — WAF short- circuits on first match. (b) A Scope-down statement on the rate rule with a NotStatement wrapping the IPSetReferenceStatement so the rate counter never even increments for those IPs.

Replacing a Wildcard CORS Policy: An Express Allowlist Case Study

2026-08-31T11:00:00+02:00

A client was weeks from going live with their Express API. We had staging access and a brief to check every endpoint the public would be able to reach. The CORS configuration looked clean.

CORS, in one sentence, is the browser's rule about which other websites are allowed to send authenticated requests to your API. Set it too narrow and your own subdomains can't talk to the backend. Set it too wide and any website on the internet can.

The team had already moved past the obvious mistake of allowing every site on the internet (origin: "*"). Instead, they had written a small text pattern (called a regex) that was meant to allow only the client's own subdomains through.

The regex was the problem.

The rest is the configuration we found, why "matches the regex" is not the same as "is safe to trust," and the explicit allowlist we shipped instead — including the answer every frontend team wants the moment they hear "allowlist": "what happens to our feature-branch preview URLs?"

The Configuration We Found

The CORS middleware was wired up like this:

// cors.config.ts
const corsOptions = {
  origin: /^https:\/\/([a-z0-9-]+\.)?client\.com$/,
  credentials: true,
  methods: ['GET', 'POST', 'PUT', 'DELETE', 'PATCH', 'OPTIONS', 'HEAD'],
};

That regex matches https://X.client.com for any subdomain X made of lowercase letters, digits, and hyphens. It also matches the bare apex domain https://client.com. The team's intent was clear: trust the client's own properties, reject everything else.

What it actually does is accept any subdomain that could be registered or pointed at the client's domain in the future — including ones that do not exist yet and ones that are owned by someone else.

How We Demonstrated the Problem

A quick test from the curl side:

# Legitimate origin — accepted, as expected.
curl -sI "$API/api/config" \
  -H "Origin: https://app.client.com" \
  | grep -i access-control
# access-control-allow-origin: https://app.client.com
# access-control-allow-credentials: true

# A subdomain that does not exist — also accepted.
curl -sI "$API/api/config" \
  -H "Origin: https://anything.client.com" \
  | grep -i access-control
# access-control-allow-origin: https://anything.client.com
# access-control-allow-credentials: true

The middleware reflects the supplied Origin header back as Access-Control-Allow-Origin, and sets Access-Control-Allow-Credentials: true, and accepts the request. The browser will happily send cookies and authorisation headers along with cross-origin requests from any matching subdomain.

CORS with credentials means the browser is authorising the cross-origin destination to act on the user's behalf with their session.

// Imagine this running on evil.client.com — a subdomain registered by an attacker
// after the client lets a wildcard DNS record lapse, or after a takeover.
fetch('https://api.client.com/user/profile', {
  credentials: 'include',
})
  .then((r) => r.json())
  .then((data) => sendToMyServer(data));

The legitimate user is on app.client.com. The malicious code is on evil.client.com. Both match the regex. The browser sends the user's session cookies along. The API returns the profile data. The attacker exfiltrates it.

The Preflight Was Also Verbose

While running the test matrix, we noticed the preflight response was generous:

curl -sI -X OPTIONS "$API/user" \
  -H "Origin: https://evil.client.com" \
  -H "Access-Control-Request-Method: DELETE"
# access-control-allow-methods: GET,POST,PUT,DELETE,PATCH,OPTIONS,HEAD
# access-control-allow-credentials: true

Every HTTP method advertised, credentials allowed. A preflight response is an explicit invitation list — listing methods the application does not even use on most endpoints tells an attacker which verbs to try.

Why This Happens — The Regex Trap

The regex is technically correct for the stated intent: "match any subdomain of client.com." The trap is that "any subdomain" includes subdomains that

do not exist yet,
could be registered by an attacker,
could be taken over via a dangling DNS record,
could be pointed at attacker-controlled infrastructure if a wildcard DNS entry is misconfigured.

Regex CORS configurations are routinely flagged because they conflate syntactic match with semantic trust. The regex says "the name looks right." It cannot say "the name is owned by us."

The Fix We Shipped — An Explicit Allowlist

We replaced the regex with a list of strings:

// cors.config.ts
const allowedOrigins = [
  'https://app.client.com',
  'https://www.client.com',
  'https://admin.client.com',
  'https://api.client.com',
];

// Feature-branch deployments add origins via env var at boot.
const featureBranchOrigins =
  process.env.FEATURE_BRANCH_ORIGINS?.split(',').filter(Boolean) ?? [];

const corsOptions = {
  origin: (origin, callback) => {
    if (!origin) {
      // Same-origin and curl/server-to-server have no Origin header.
      // Accept them — they are not browser cross-origin requests.
      return callback(null, true);
    }
    if (
      allowedOrigins.includes(origin) ||
      featureBranchOrigins.includes(origin)
    ) {
      return callback(null, true);
    }
    return callback(null, false);  // 403, not 500.
  },
  credentials: true,
};

Four properties on the allowlist by default. Feature branches plug in through an environment variable — the deployment pipeline appends the preview URL to FEATURE_BRANCH_ORIGINS when it provisions a branch deployment, and removes it when the branch is torn down.

Three things to note about the callback form:

callback(null, false) is the rejection path — the middleware returns a 403 instead of letting an exception bubble up to a 500. The application no longer leaks the existence of the endpoint to non-allowlisted origins.
The empty-origin case is explicit. Curl and server-to-server callers do not send an Origin header. Treating them as allowed prevents internal jobs from breaking when the same API serves machine traffic.
The credentials: true flag is still on — that is correct because the allowlist is now strict.

What We Had To Fix Alongside CORS

The middleware previously threw an Error on unknown origins:

// Previously
if (!isAllowed) throw new Error('Not allowed by CORS');

Throwing inside the CORS middleware turns a "denied" outcome into a 500 Internal Server Error from Express's default error handler. That tells the attacker the endpoint exists and is doing something interesting. The new callback path returns a clean 403, and the error handler stays out of it.

The Feature-Branch Conversation

The frontend team's first response was the one we expect on every engagement that touches CORS:

"But we need dynamic subdomains for feature branches. We cannot maintain a list."

The list is not maintained by hand — it is maintained by the deployment pipeline.

When a branch deployment is created, the pipeline appends its origin (e.g. https://branch-1234.preview.client.com) to FEATURE_BRANCH_ORIGINS in the API's environment.
When the branch is torn down or merged, the pipeline removes it.
The API restarts on env change. Cold start is sub-second; the redeploy is a non-event.

The result is the same flexibility the regex used to give — any branch can talk to the API — without giving the same flexibility to anyone who can register any subdomain.

Verifying After the Change

The reproduction script we ran before the fix returns clean results afterwards:

# 1. Legitimate origin — still works
curl -sI "$API/api/config" \
  -H "Origin: https://app.client.com" \
  | grep -i access-control
# access-control-allow-origin: https://app.client.com
# access-control-allow-credentials: true

# 2. Unknown subdomain — no allow-origin header at all
curl -sI "$API/api/config" \
  -H "Origin: https://evil.client.com" \
  | grep -i access-control
# (no output — header not set)

# 3. Status code for blocked preflight — 403, not 500
curl -s -o /dev/null -w "%{http_code}\n" -X OPTIONS "$API/api/config" \
  -H "Origin: https://evil.client.com" \
  -H "Access-Control-Request-Method: GET"
# 403

The browser, on the legitimate side, sees the same response as before. The browser, on the attacker side, sees a CORS error and never gets to the response body.

What We Added Alongside CORS — The Layered Story

CORS is a browser policy. A determined attacker can still hit the API directly without a browser. Three companion changes shipped alongside the allowlist:

Allowed methods narrowed per endpoint. The middleware no longer advertises DELETE on endpoints that do not support it. Each route declares its own method set.
JWT validation on every authenticated request. A separate audience check — covered in its own case study — runs alongside CORS. Even on the days CORS gets misconfigured again, the audience check holds.
Origin telemetry. We log every rejected origin to CloudWatch with a 30-day retention window. If a hostname starts showing up, the team sees it before it shows up in a support ticket.

Five things we now check on every CORS engagement

Regex CORS allowlists are almost always wrong. The string form is more verbose but it is also unambiguous. Add a linter rule that fails the build on a non-string cors.origin value.
Wildcard subdomain regex matches subdomains you do not control. If you do not own every subdomain of your apex, the regex hands trust to whoever does.
callback(null, false) is the rejection path. Throwing inside CORS middleware turns a 403 into a 500 and tells the attacker something interesting happened.
Feature branches do not need wildcard CORS. A pipeline that updates an FEATURE_BRANCH_ORIGINS env var on create and tear-down delivers the same DX without the security gap.
Tighten Access-Control-Allow-Methods per route. A blanket method allowlist is an open menu of verbs for an attacker to probe. Routes declare the verbs they accept; the middleware reflects only those.

Cutting the File-Upload Bill: A Cognito + S3 Case Study

2026-08-24T11:00:00+02:00

The client had a managed file-upload service in their browser, the kind that gives you a hosted widget, signs every upload, runs a virus scan, and bills you per request. It worked, and the team liked the developer experience. The CFO did not like the invoice.

The obvious question was whether we could replace it with all native AWS services and keep the same browser-direct upload pattern. Yes, with a Cognito + S3 setup. We built a three-layer permission model that most Cognito-on-S3 tutorials skip, which is why they leak.

Two AWS services do the work here, both worth one sentence each before we go further:

S3 — AWS's object-storage service. The bucket the files actually land in. Cheap, durable, billed per gigabyte stored and per request.
Cognito Identity Pool — AWS's way of handing a browser a short-lived AWS credential so it can talk to S3 directly, without an API server in the middle. The "boundary" the rest of this post tightens.

What the Frontend Had to Do

The browser uploads files directly to storage — never through the backend API. This pattern matters for two reasons:

Throughput. A 200 MB upload doesn't tie up a backend worker for two minutes.
Cost. Backend egress through the API doubles the bandwidth bill.

The managed service handled this with a hosted SDK: the browser calls the service, gets a one-shot signed URL, uploads to the vendor's S3, and the vendor pushes the object into the client's bucket via a webhook. Clean DX. Big invoice.

What we needed to replicate:

Browser uploads directly to the client's S3 bucket
No long-lived AWS credentials in the browser
Per-user isolation (user A cannot read or overwrite user B's files)
Content-type and size guards
Anti-abuse layer for anonymous (pre-signup) uploads

The Architecture

The pattern is well-known: AWS Cognito Identity Pool → temporary AWS credentials → direct PUT to S3. AWS Amplify ships this same architecture as defineStorage, which gives us a useful reference point.

The flow on a happy path:

The whole thing lives in three Terraform files: cognito.tf, s3.tf, and iam.tf. The browser code is roughly thirty lines of @aws-sdk/client-s3 and the standard Cognito credential provider.

The catch — and the reason most Cognito-on-S3 implementations leak — is that three independent layers decide whether a given operation is allowed. Get any one of them wrong and you've either bricked uploads or opened the bucket to the internet.

The Three Layers of Permission Evaluation

This is the mental model we built before writing a single line of Terraform.

Layer	What it controls	Where it lives
Session scope-down policy	Which AWS services unauthenticated credentials can call at all	`AmazonCognitoUnAuthedIdentitiesSessionPolicy` (AWS-managed)
IAM role policy	What actions the role can take on which resources	Attached to the role you create
Bucket policy	What principals can do directly to the bucket	Attached to the S3 bucket itself

The non-obvious behavior:

The session scope-down for unauthenticated identities does not include S3. So an IAM grant of s3:PutObject to the unauthenticated role is silently blocked by the session policy. This trips up everyone the first time.
Bucket policies bypass the session scope-down. A grant of s3:PutObject to the Cognito role principal inside the bucket policy is honored.
Therefore: put your real grants in the bucket policy, not the IAM policy (or grant via both for defense in depth — only the bucket-policy grant will actually fire for unauthenticated identities).

Print it out.

The Terraform — Identity Pool

resource "aws_cognito_identity_pool" "uploads" {
  identity_pool_name               = "client-uploads"
  allow_unauthenticated_identities = true
}

resource "aws_cognito_identity_pool_roles_attachment" "uploads" {
  identity_pool_id = aws_cognito_identity_pool.uploads.id
  roles = {
    "unauthenticated" = aws_iam_role.cognito_unauth.arn
  }
}

We do allow unauthenticated identities, because the client's app needs to accept uploads before signup completes. The risk is real, and the rest of the Terraform is how we tightened around it.

The Terraform — IAM Role

The IAM role for the unauthenticated identity gets a minimal trust policy and a permission policy that grants only s3:PutObject to a per-identity prefix. Note: this is a defense-in-depth layer; the actual grant that fires for unauthenticated identities will be the bucket policy below.

data "aws_iam_policy_document" "cognito_unauth_assume" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = ["cognito-identity.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "cognito-identity.amazonaws.com:aud"
      values   = [aws_cognito_identity_pool.uploads.id]
    }
    condition {
      test     = "ForAnyValue:StringLike"
      variable = "cognito-identity.amazonaws.com:amr"
      values   = ["unauthenticated"]
    }
  }
}

resource "aws_iam_role" "cognito_unauth" {
  name               = "cognito-uploads-unauth"
  assume_role_policy = data.aws_iam_policy_document.cognito_unauth_assume.json
}

data "aws_iam_policy_document" "cognito_unauth_perms" {
  statement {
    actions   = ["s3:PutObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts"]
    resources = [
      "${aws_s3_bucket.uploads.arn}/uploads/$${cognito-identity.amazonaws.com:sub}/*",
      "${aws_s3_bucket.uploads.arn}/documents/$${cognito-identity.amazonaws.com:sub}/*",
    ]
    condition {
      test     = "StringLike"
      variable = "s3:content-type"
      values   = ["image/*", "video/*", "application/pdf"]
    }
  }
}

resource "aws_iam_role_policy" "cognito_unauth" {
  role   = aws_iam_role.cognito_unauth.id
  policy = data.aws_iam_policy_document.cognito_unauth_perms.json
}

Three constraints, all important:

s3:PutObject only. No GetObject, no DeleteObject, no ListBucket. The browser writes; it does not read or enumerate.
Per-identity prefix via ${cognito-identity.amazonaws.com:sub}. AWS interpolates this at request time. User A's credentials can only write to uploads/<A's-sub-id>/*. They cannot guess their way into User B's folder. Terraform's $$ is the escape for AWS template syntax — easy to miss the first time.
s3:content-type allowlist. The browser declares the content type; AWS checks it before accepting the PUT. text/html is denied at the IAM boundary. (Note: s3:content-type works in IAM policies but not in bucket policies — keep this one here.)

The Terraform — Bucket and Bucket Policy

Because the session scope-down blocks the IAM grant for unauthenticated identities, the actual PutObject allow has to live in the bucket policy:

resource "aws_s3_bucket" "uploads" {
  bucket = "client-uploads-prod"
}

resource "aws_s3_bucket_public_access_block" "uploads" {
  bucket                  = aws_s3_bucket.uploads.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_versioning" "uploads" {
  bucket = aws_s3_bucket.uploads.id
  versioning_configuration { status = "Enabled" }
}

data "aws_iam_policy_document" "uploads_bucket_policy" {
  # The grant that actually fires for unauthenticated Cognito identities.
  statement {
    sid       = "AllowScopedUploads"
    effect    = "Allow"
    actions   = ["s3:PutObject", "s3:AbortMultipartUpload"]
    principals {
      type        = "AWS"
      identifiers = [aws_iam_role.cognito_unauth.arn]
    }
    resources = [
      "${aws_s3_bucket.uploads.arn}/uploads/$${cognito-identity.amazonaws.com:sub}/*",
      "${aws_s3_bucket.uploads.arn}/documents/$${cognito-identity.amazonaws.com:sub}/*",
    ]
  }

  # Deny everything else, explicitly, for the Cognito role.
  statement {
    sid       = "DenyEverythingElse"
    effect    = "Deny"
    actions = [
      "s3:GetObject", "s3:DeleteObject", "s3:ListBucket",
      "s3:ListBucketVersions", "s3:ListBucketMultipartUploads",
    ]
    principals {
      type        = "AWS"
      identifiers = [aws_iam_role.cognito_unauth.arn]
    }
    resources = [
      aws_s3_bucket.uploads.arn,
      "${aws_s3_bucket.uploads.arn}/*",
    ]
  }
}

resource "aws_s3_bucket_policy" "uploads" {
  bucket = aws_s3_bucket.uploads.id
  policy = data.aws_iam_policy_document.uploads_bucket_policy.json
}

block_public_acls plus block_public_policy mean the bucket can never accidentally drift to "world-readable" — even if a future engineer adds a Principal: * statement, AWS rejects the policy at apply time.

Verifying With `curl`

Before handing off, we validated the boundary the same way an attacker would — with curl and no SDK in between.

# 1. Get an anonymous identity.
curl -s -X POST "https://cognito-identity.us-east-1.amazonaws.com/" \
  -H "Content-Type: application/x-amz-json-1.1" \
  -H "X-Amz-Target: AWSCognitoIdentityService.GetId" \
  -d '{"IdentityPoolId":"us-east-1:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"}'

# 2. Exchange it for temporary AWS credentials.
curl -s -X POST "https://cognito-identity.us-east-1.amazonaws.com/" \
  -H "Content-Type: application/x-amz-json-1.1" \
  -H "X-Amz-Target: AWSCognitoIdentityService.GetCredentialsForIdentity" \
  -d '{"IdentityId":"us-east-1:yyyyyyyy-..."}'

We then exported the credentials and ran the matrix of operations our policy is supposed to block:

# Reads — denied
aws s3 cp s3://client-uploads-prod/uploads/foo/bar.jpg /tmp/x
# An error occurred (AccessDenied) ...

# Listing — denied
aws s3 ls s3://client-uploads-prod/
# An error occurred (AccessDenied) ...

# Deletes — denied
aws s3 rm s3://client-uploads-prod/uploads/foo/bar.jpg
# An error occurred (AccessDenied) ...

# Writing outside your own prefix — denied
echo "x" | aws s3 cp - s3://client-uploads-prod/uploads/somebody-else/file.txt
# An error occurred (AccessDenied) ...

# Writing wrong content type to your own prefix — denied
aws s3 cp evil.html s3://client-uploads-prod/uploads/$MY_SUB/evil.html \
  --content-type "text/html"
# An error occurred (AccessDenied) ...

# Writing the right content type to your own prefix — allowed
aws s3 cp ok.jpg s3://client-uploads-prod/uploads/$MY_SUB/ok.jpg
# upload: ./ok.jpg to s3://client-uploads-prod/...

Every line in that matrix is part of the acceptance test. We re-run it in CI against staging on every Terraform change.

What We Added Back From the Managed Service

Replacing a managed service means rebuilding the safety net the vendor wrapped around the happy path. Three pieces in particular needed deliberate replacement.

1. File-size limits and content-type validation, properly

The IAM-level s3:content-type check trusts the header — easy to spoof. We added a Lambda triggered by s3:ObjectCreated:* that:

Reads the first 16 bytes of the new object (magic bytes)
Matches against an allowlist of file signatures (JPEG, PNG, MP4, PDF…)
Deletes the object and writes a denied/<sub>/<timestamp>.log audit entry if the magic bytes do not match the prefix policy

This is the same defense-in-depth pattern the managed service used; we now own it.

2. Anti-abuse for guest uploads

For genuinely anonymous flows (pre-signup), we put a small backend endpoint in front. The browser proves it is a human with reCAPTCHA, the backend validates the token, and only then does the frontend get to call GetCredentialsForIdentity. This converts "anyone can get AWS credentials" into "any verified human can get AWS credentials" — the bar we wanted.

3. Bucket abuse limits

We layered in two things the managed service used to do for us:

S3 Lifecycle rule that deletes anything in uploads/$sub/ older than 30 days unless the object has been promoted (a tag set when the user completes signup).
CloudWatch alarm on BucketSizeBytes and on NumberOfObjects per prefix, with thresholds set at 5× the 30-day rolling average. A bored teenager filling the bucket with zeros sets off the alarm before it sets off the invoice.

Migration Plan (No Downtime)

We rolled this out in three phases over two sprints. The client never lost an upload.

Dual-write. Frontend hits the managed service and the new Cognito flow in parallel. Compare file hashes in the backend. Surface any mismatch in a Slack channel.
Read switch. Once dual-write was clean for two weeks, the application started reading from S3. The managed service's pipeline stayed running, but downstream consumers stopped touching it.
Cutover. Frontend disables the managed-service call. Vendor account moved to "evergreen" tier. After 60 days of clean operation we cancelled the contract.

What the Client Walked Away With

In rough numbers (exact figures live in the client's invoice records, not ours):

The monthly bill dropped sharply. Most of the savings came from the dominant cost shifting from "you pay per upload" to "you pay per gigabyte stored" — and on this workload, per-GB storage on S3 is a fraction of what the managed service was charging per request.
They own the pipeline now. The whole upload pipeline lives in their own Terraform repository. If they ever want to swap a piece out, they can.
They can do things the managed service did not let them do. Their backend can now sign upload URLs based on any business rule they care about — has the user finished signup, are they on a paid plan, what country are they in. None of that was possible before.
The boundary is tighter than the tutorial version. Per-user prefixes, magic-byte validation, abuse alarms, versioning so we can undo a bad upload, CloudTrail data events on every S3 operation. The kind of plumbing that gets noticed only when something tries to abuse it.

The client's reaction, paraphrased: "This is the architecture we wished the vendor's docs had explained in the first place."

Two gotchas the tutorials skip

First, s3:prefix only applies to s3:ListBucket, not to s3:ListBucketMultipartUploads — split them into separate statements or AWS rejects the policy at apply time. Second, SDK helpers like --dryrun check client-side logic only; they do not call AWS IAM. Test the deny path with curl and the AWS CLI directly, or the acceptance test is lying to you.

Two things I would enable from day one on the next build: versioning on the bucket, so a bad upload is one command away from being undone, and CloudTrail data events, so the forensic answer to "what did the temporary credential actually do" is available at all.

How a CloudFront Custom Error Page Leaked JWTs to S3: A Case Study

2026-08-13T11:00:00+02:00

A client was preparing to put an API behind CloudFront in production. During the final penetration test, one curl request showed that every distribution with custom error pages was reflecting users' JWT tokens in S3 XML error responses.

Sixteen distributions shared the same copy-pasted Terraform block.

CloudFront is AWS's content delivery network (CDN). A JSON Web Token (JWT) is the signed session string the API gives a user after login. curl is the command-line tool we use to send HTTP requests by hand. Three terms are enough to follow the finding.

CloudFront was handling TLS at the edge, HTTP/2 and HTTP/3 connections, DDoS protection, and branded error pages from a separate S3 bucket. The staging setup had been running for weeks. The distributions were stable, the error pages rendered correctly, and the TLS grades looked good.

The One-Line Proof

curl https://your-api.example.com/custom_error_pages/502.html \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.YOUR_ACTUAL_TOKEN"

If your response looks like this, you're vulnerable:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>InvalidArgument</Code>
  <Message>Unsupported Authorization Type</Message>
  <ArgumentName>Authorization</ArgumentName>
  <ArgumentValue>Bearer eyJhbGciOiJIUzI1NiJ9.YOUR_ACTUAL_TOKEN</ArgumentValue>
</Error>

Your full JWT (user ID, email, role, expiry, everything in the payload) reflected back in an S3 XML error response. The path exists on practically every CloudFront distribution with custom error pages. Any GET request with an Authorization header triggers it, no authentication and no origin failure required.

S3 reflects any Authorization scheme, not just Bearer. Basic auth credentials (Basic dXNlcjpwYXNzd29yZA==), API tokens (Token API_TOKEN_VALUE), anything non-AWS gets dumped into the XML. If your API uses any of these, the same leak applies.

Cookies, X-API-Key, and custom headers are forwarded to S3 too but not reflected in the XML. S3 only complains about the one header it tries to parse.

Why did CloudFront forward the `Authorization` header to S3?

A CloudFront distribution sits between users and one or more origins (the actual servers that hold your content). Each origin gets a behavior that tells CloudFront which requests to route there and what to forward along.

In this pattern, there are two origins behind one distribution:

The API behavior needs viewer headers. The S3 error-page behavior does not.

The API origin is your application: an Application Load Balancer (ALB) in front of Elastic Container Service (ECS) containers, a Lambda function, or an EC2 instance. It needs the viewer's Authorization header to authenticate requests, Content-Type to parse bodies, and cookies for sessions. Forwarding everything to this origin is correct.

The error page origin is an S3 bucket containing three static HTML files: 502.html, 503.html, and 404.html. It serves the same branded "We'll be right back" page to every user regardless of who they are.

It needs nothing from the viewer: not their identity, not their cookies, not their query strings. It just needs CloudFront to ask for a file by path, and it returns the HTML.

The problem is that both origins were given the same origin request policy: the one designed for the API. So when CloudFront routes a request to the S3 error page bucket, it sends along everything the viewer included, including Authorization: Bearer <JWT>.

S3 is not your API. It doesn't understand Bearer tokens. It tries to interpret the Authorization header as AWS Signature Version 4, fails, and returns an XML error that includes the header value it couldn't parse. Your JWT is now in the response body.

The setup looks like this in Terraform (CloudFormation and the console follow the same pattern):

# Origin 1: Your API (ALB, ECS, Lambda, etc.)
origin {
  domain_name = aws_lb.api.dns_name
  origin_id   = "api"
}

# Origin 2: S3 bucket with static error pages
origin {
  domain_name = aws_s3_bucket.error_pages.website_endpoint
  origin_id   = "error-pages"
}

# When the API returns 502, serve the S3 error page instead
custom_error_response {
  error_code         = 502
  response_page_path = "/custom_error_pages/502.html"
}

# Behavior for the API (default): forwards all viewer headers
default_cache_behavior {
  target_origin_id         = "api"
  origin_request_policy_id = aws_cloudfront_origin_request_policy.AllViewerExceptHostHeader.id
}

# Behavior for the error pages: ALSO forwards all viewer headers
ordered_cache_behavior {
  path_pattern             = "/custom_error_pages/*"
  target_origin_id         = "error-pages"
  origin_request_policy_id = aws_cloudfront_origin_request_policy.AllViewerExceptHostHeader.id
  # ↑ THIS IS THE PROBLEM
}

The AllViewerExceptHostHeader policy tells CloudFront: "forward every header the viewer sent (Authorization, Cookie, X-Custom-Whatever) to the origin, except Host." That's correct for the API behavior, where your backend needs those headers. It's catastrophically wrong for the error page behavior, where the origin is an S3 bucket serving static HTML.

Three ways the token leaks

The error page path isn't linked anywhere. No browser navigates to /custom_error_pages/502.html during normal use. Three practical routes still reach it.

The path is trivially discoverable by scanners. It's in CloudFront's own documentation examples, and tools like ffuf, feroxbuster, and nuclei include /error/, /custom_error_pages/, and /502.html in their default wordlists. A scanner that hits the path and sees <Code>InvalidArgument</Code> with <ArgumentName>Authorization</ArgumentName> in the XML response can identify the misconfiguration from the shape alone. Our pentest tooling found this one that way.

A production outage can leak the token without any scanner. When the origin returns 502/503, CloudFront intercepts the error and fetches the S3 error page with the original request's Authorization header attached. The user didn't navigate to the error page; CloudFront sent them there. Their token leaks through a normal deployment, container restart, or scaling event, visible in the response body to any browser extension, corporate proxy, or compromised CDN edge watching that window.

Path	Trigger	Result
Direct access	A GET request with an `Authorization` header reaches `/custom_error_pages/502.html`.	S3 reflects the header in XML. No origin failure is needed.
Origin failure	An API request carries a JWT, then the origin returns 502 or 503.	CloudFront fetches the S3 error page with the original headers, and the token comes back to the user.
Plain HTTP	The error-page behavior uses `viewer_protocol_policy = "allow-all"`.	A network attacker can read the token without breaking its cryptography.

Most of the distributions we audited had the plain-HTTP setting. The API behavior used redirect-to-https; the error-page behavior was a separate block with separate settings.

The Terraform Template Problem

We found this in a client's infrastructure codebase. We audited every CloudFront distribution across their staging and production accounts. The same ordered_cache_behavior block for /custom_error_pages/* showed up on distribution after distribution, all pointing to the same S3 error pages bucket, all using AllViewerExceptHostHeader.

The block was written once for the API distribution, where it was correct. Then it was copied to every other distribution (SPAs, admin dashboards, static sites, upload CDNs) because it "worked" and nobody questioned whether a static error page bucket needed to receive the viewer's Authorization header.

Terraform's copy-paste turned one mistake into a pattern across the whole account.

A handful of the affected distributions served authenticated traffic (APIs and legacy apps). Those were fully exploitable: every authenticated request that hit a 502 leaked the caller's JWT. The rest served static frontends that don't normally receive Authorization headers, but the direct-access path still worked on all of them.

The Chain That Made It Critical

The JWT leak alone is a HIGH: you can capture a token and replay it for the remaining access-token lifetime. But during this engagement, we found it chained with something else that pushed it to CRITICAL.

The API signs session tokens with a JWT secret. On staging, that secret was hardcoded as a trivially guessable value in the ECS task definition's plaintext environment block. Not in Secrets Manager, not pulled from SSM Parameter Store, just a short, dictionary-word string sitting in Terraform.

We confirmed the secret remotely without any internal access. The API returns different error messages depending on whether a token's signature verifies:

# Token signed with the guessable secret (correct):
curl -X POST https://api.staging.example.com/api/session/token \
  -H "Cookie: session=<forged-token-signed-with-guessable-secret>"
# → "Session has been revoked" (signature PASSED, DB hash lookup failed)

# Token signed with wrong secret:
curl -X POST https://api.staging.example.com/api/session/token \
  -H "Cookie: session=<token-with-wrong-signature>"
# → "Invalid session" (signature FAILED)

That error difference acts as an oracle. "Session has been revoked" means the JWT signature verified successfully; the server moved past cryptographic verification to the database lookup phase. "Invalid session" means the signature check failed. The two code paths are distinguishable remotely.

We proved the full chain end-to-end

We registered a test account and signed in. We captured the session token, leaked it through the error page, and validated it offline. Then we replayed it and had the account. All with curl.

Step 1: Sign in. We registered a test account and signed in. The server set an httpOnly session cookie. It was a long-lived JWT signed with the guessable secret.

Step 2: Leak the token. We sent a GET request to the error page path. The session token was in the Authorization header. The S3 XML error returned it verbatim. Byte for byte, it matched the original.

Step 3: Validate offline. We computed the HMAC-SHA256 signature using the guessable secret. It matched the token's signature exactly. No server interaction needed. The attacker now knows the token is real, who it belongs to, and when it expires.

Step 4: Replay. We sent the leaked session token to the token endpoint. The server verified the JWT signature (passed) and looked up the token hash (found a real active session). It minted a fresh access token.

Step 5: Account takeover. We used the fresh access token to call the user-profile endpoint. The server returned the full profile: ID, email, name, role, all PII fields. Full account access with no password. All from a token that was never meant to leave the httpOnly cookie jar.

The leak becomes account takeover when the staging JWT secret is guessable.

We ran the entire chain with curl against a live staging environment. We didn't need source code access to exploit it. We only needed it to discover the hardcoded secret in Terraform. An external attacker reaches the same outcome by guessing the secret or by intercepting a token through the error page leak.

The public bucket

The bucket is also enumerable:

curl https://your-error-pages-bucket.s3.amazonaws.com/

Returns a full ListBucketResult XML with every file, size, ETag, and last-modified timestamp, no authentication required.

It's public because the error page origin uses an S3 website endpoint, which forces public read access. OAC (Origin Access Control) only works with the REST API endpoint. The pages themselves are benign HTML, but the bucket is one IAM policy drift away from attacker-writable: a phishing form in a 502 page would be served by your own domain, from your own CloudFront distribution, with your own TLS certificate.

The Fix

The error page behavior serves static HTML from S3. It doesn't need the viewer's auth token, cookies, query strings, or any other header.

When you omit origin_request_policy_id entirely, CloudFront sends only the minimum required headers to the origin. That is enough for an S3 website endpoint to fetch a file by path, and there is nothing left for S3 to reflect back.

The simplest fix is to delete the origin_request_policy_id line from every error page behavior. It shouldn't have been there in the first place.

But if you want to be explicit about the intent (and make it harder for someone to re-add a permissive policy later thinking it was accidentally removed), define a policy that forwards nothing:

resource "aws_cloudfront_origin_request_policy" "ErrorPagesNoAuth" {
  name    = "ErrorPages-NoAuth"
  comment = "Error pages are static HTML, no viewer headers needed"

  headers_config {
    header_behavior = "none"
  }
  cookies_config {
    cookie_behavior = "none"
  }
  query_strings_config {
    query_string_behavior = "none"
  }
}

Then in every ordered_cache_behavior for /custom_error_pages/*:

ordered_cache_behavior {
  path_pattern             = "/custom_error_pages/*"
  target_origin_id         = "error-pages"
  origin_request_policy_id = aws_cloudfront_origin_request_policy.ErrorPagesNoAuth.id
  viewer_protocol_policy   = "redirect-to-https"
  # ... rest unchanged
}

Verification:

# Before fix:
curl https://your-api.com/custom_error_pages/502.html \
  -H "Authorization: Bearer TEST"
# → <ArgumentValue>Bearer TEST</ArgumentValue>  ← LEAKED

# After fix:
curl https://your-api.com/custom_error_pages/502.html \
  -H "Authorization: Bearer TEST"
# → <!DOCTYPE html><html>...(your 502 page HTML)...  ← SAFE

For defense in depth:

Switch the S3 origin from a public website endpoint to a private bucket with OAC. This closes the public listing vulnerability and eliminates the unencrypted HTTP origin protocol.
Set error_caching_min_ttl = 0 so error responses aren't cached at all.
Set viewer_protocol_policy = "redirect-to-https" on the error page behavior so the path can't be accessed over plain HTTP.

Forwarding headers to error pages is valid when the origin needs the viewer's identity: a Lambda@Edge function rendering personalized error pages, a Cognito-gated bucket, or a debugging service logging which user hit the error. In those cases the origin is an application that can safely consume auth headers. If your error page origin is S3, the answer is always header_behavior = "none".

What we shipped

Sixteen distributions closed. The JWT reflection path is gone across the client's account, and HTTP-to-HTTPS is enforced on every error-page behavior. The one residual risk (the public S3 website endpoint) is on the follow-up list.

How to Check If You're Vulnerable

Step 1: Find your error page path. Look for custom_error_response blocks in your CloudFront distribution. The response_page_path tells you where the error pages live.

Step 2: Send a request with an auth header.

curl -s https://your-domain.com/custom_error_pages/502.html \
  -H "Authorization: Bearer CHECK_THIS_TOKEN"

Step 3: Check the response. If you see <ArgumentValue>Bearer CHECK_THIS_TOKEN</ArgumentValue> in the XML, you're vulnerable. If you see your HTML error page, you're fine. A 403 means the path doesn't exist or the behavior doesn't allow GET, which is also fine.

Step 4: Check all your distributions. If you use the same Terraform module or copy-paste pattern across distributions, check every one. In one engagement we found the majority of distributions sharing the same misconfiguration. The one you check might be fine while a dozen others are leaking.

Check every distribution

Nobody asked whether a static HTML bucket needed the same headers as an authenticated API. If you're running CloudFront with custom S3 error pages, spend sixty seconds running the curl command above against each distribution.

A note on responsible testing: every test account created during this engagement was registered with a non-existent @example.com address, used only for curl-based proof-of-concept, and cleaned up after the findings were documented. No real user sessions were intercepted or replayed. The full chain was proven end-to-end using our own credentials against a staging environment with explicit authorization from the infrastructure owner.

Iftar at Apetit: The Sarajevo Team Picks a Menu

2026-08-04T11:00:00+02:00

From time to time, the Sarajevo Clearview team meets in the same room. "Same room" takes work and dedication. Most days, the team is stretched across the globe in apartments in Sarajevo, Malang, and Cape Town, with a laptop on US East Coast time for a client.

The week before Ramadan, as it is the company's custom, Amina booked us a table for iftar at Apetit, the food spot everyone in Sarajevo recommends. It sits on Vilsonovo šetalište, the riverside promenade locals call Vils, five minutes from most of our apartments.

svejedno: whatever, fine with me
može: works for me
ekipa: the crew, our people
fakat: honestly, for real
meni: for me / put me down for
pa kad je iftar: well, whenever iftar is (the punchline)

Here is the thread that got us there, translated and lightly compressed:

Amina Kadrić

@here Iftar in Sarajevo next week 🌙 React with the day that works — I'll find us a spot. 🍽️

Amina Kadrić

@nedim @Aid @Ahmed @Adnan — Booked Apetit for Tuesday. Pick a menu, ping me by tomorrow. 🙏

nedim

Hadžijski one for me. 🥩

Aid

Same. ➕

Adnan

Number three, the baked chicken. 🍗

Ahmed

Same. Chicken. 🍗

Redzy

🍗 Same as those two ⬆️

Amina

Plot twist 🌀 — corporate menu needs 4+ people. Waiting on confirmation. What's your backup?

Aid

Svejedno. 🤷

nedim

Svejedno. Bump me to Menu 2. 🤷

Ahmed

I'll switch to corporate — but only if @nedim takes my sahan 😄 Dolma and sarma, not my crowd.

Amina

Still short one 👀 I'm on fish or chicken.

nedim

Genuinely svejedno. Put whatever on my plate. 🍽️

Amina

I'll be late — client call till 5:30. You'll all beat me there. Reservation's under my name.

nedim

What time?

Amina

pa kad je iftar 😄 (whenever iftar is)

A few things in that thread stayed with me.

"Svejedno."

How to say it: sveh-YED-noh. Stress the middle.

Half the thread is people saying svejedno. In this team, that word means trust. Amina booked the restaurant, picked the menu options, chased the head-count. Nobody in the thread felt the need to second-guess any of it. When one of us takes on the coordination work, the rest of us get out of the way. And we take any excuse to sit at the same table — Clearview picks up the tab, and a distributed team never says no to a paid-for meal.

The same pattern shows up on client work. When someone picks up a task, the rest of the team stays out of the way. We ask when we have something to add, and we say svejedno — or the English equivalent, "sounds good, go for it" — when we don't.

Sometimes the person out front is the senior engineer on a client sprint. This time it was Amina, the one who knows every food spot in Sarajevo.

"Pa kad je iftar."

Amina's reply to my what-time question assumes that everyone at the table already knows when iftar is. A non-Bosnian reader would probably miss it.

Iftar is the meal Muslims eat to break the daily fast during the month of Ramadan. It happens at sunset, which means the exact time slides ten to fifteen minutes later every week as the days get longer. In Sarajevo in mid-March, sunset was somewhere around 6:15 pm. If you knew the calendar, you knew the time. If you did not, you asked, and Amina answered. Because she is the boss.

The cultural detail did not have to be spelled out in the thread. Everyone on the invite already got it.

Time for meza

The corporate menu did land in the end. The five of us who made it — Amina, Aid, Adnan, Ahmed, and me — ate at Apetit as the sun set behind the hills over the Miljacka. Redzy joined for part of it. There were the standard meza, the Bosnian word for the little starter plates other cuisines call mezze or mezes. In Bosnia, meza is the part of the meal where the real talking happens, where a table settles in and stops being in a hurry. Then came the mains, and then someone ordered Bosanska kafa and the conversation wandered, as it always does, into a mix of shop talk and family news.

Nobody talked about "team building."

Iftar at Apetit, Sarajevo, March 17, 2026.

We had not all sat at the same table since the last one, and we would probably not sit at the same table for another couple of months. That is the rhythm of a distributed team — long stretches of async work broken by a shared meal every few months.

Why the dinner is worth it

For a distributed team, in-person meals matter more than they look. The grand summit in Bali counts — Clearview gathered in Ubud in January — but the small local dinner counts too, and it happens more often. The one where Amina's message goes out, five people RSVP within the day, and the reservation goes under her name because she is the one who called the restaurant. Typical Amina. She knows the best places to eat in Sarajevo, and the rest of us are smart enough to show up where she tells us to.

Every time we do one of these, the async work that follows for the next few weeks is smoother. Threaded replies come with more benefit of the doubt, less second-guessing, nobody a stranger. If you have never worked distributed, that is the reason to spend real money on getting people in a room. It makes the rest of the year work better.

We are lucky, in Sarajevo, that most of the Clearviewers here can be in the same city inside of an hour. Having enough of us in one city to fill a table on short notice is a real gift. When Ahmed messages that he doesn't like dolma and sarma, or Amina jokes that we should all know when iftar is, that only feels normal because we have eaten together before.

Nedim · Sarajevo · March 2026

How a JWT Audience Map Saved a CORS Mistake: A Defense-in-Depth Case Study

2026-07-22T11:00:00+02:00

While pen-testing a client's API, we found a CORS policy that accepted any subdomain of the client's main domain, with credentials, on every kind of HTTP request. Textbook misconfiguration. We were drafting it as a HIGH severity finding when we tried to run the exploit ourselves, and discovered a second, independent control that stopped the attack at a different layer.

Two terms before we go further:

CORS — the browser's rule about which other websites are allowed to send authenticated requests to your API. Mis-set, it lets any site act on a logged-in user's behalf.
JWT — JSON Web Token. The short signed string the API hands a user after they log in. Every subsequent request carries it as proof of who the user is. The token has an audience field that names which application the token was minted for, and the API can refuse a token presented to the wrong audience.

Two independent controls — the stricter one held while the weaker one was broken.

The CORS Misconfiguration

The Express CORS middleware allowed any subdomain through a regex:

// cors.loader.ts
const pattern = /^https:\/\/([a-z0-9-]+\.)?client\.example\.org$/;

This matches any subdomain, including ones that do not exist yet, ones that could be registered by attackers, and ones that could be taken over via a dangling DNS record. We confirmed the breadth with a small matrix:

# Evil subdomain — accepted
curl -sI "$API/api/config" \
  -H "Origin: https://evil.client.example.org"
# access-control-allow-origin: https://evil.client.example.org
# access-control-allow-credentials: true

# Punycode subdomain — accepted
curl -sI "$API/api/config" \
  -H "Origin: https://xn--evil.client.example.org"
# access-control-allow-origin: https://xn--evil.client.example.org

# Numeric prefix — accepted
curl -sI "$API/api/config" \
  -H "Origin: https://123evil.client.example.org"
# access-control-allow-origin: https://123evil.client.example.org

CORS with credentials: true, wildcard subdomains, and DELETE listed in the preflight Access-Control-Allow-Methods. Every box on the checklist of "trivially exploitable CORS misconfiguration."

We were about to write HIGH — any subdomain can steal authenticated session data. We tried to actually exploit it first.

The Exploit That Failed

For the exploit to work in a real browser, the attacker needs to make authenticated requests from their evil subdomain. With JWT auth (Authorization header, not cookies), the attacker first has to obtain a token from the evil origin — either by signing in there, or by stealing a token from the legitimate frontend's localStorage and replaying it.

We tried signing in from the evil origin:

curl -s -X POST "$API/api/auth/signin" \
  -H "Content-Type: application/json" \
  -H "Origin: https://evil.client.example.org" \
  -d '{"email":"test@example.org","password":"Secret123"}'

{ "success": false, "message": "Unknown origin" }

CORS accepted the origin. The API rejected the sign-in. Something independent of CORS was checking.

The JWT Audience Map

Digging into jwt.config.ts surfaced this:

const audienceMap: Record<string, string> = {
  // Production
  'https://app.client.example': 'main-app',
  'https://admin.client.example':         'admin-app',
  'https://partners.client.example':  'partner-portal',
  // Staging
  'https://app.staging.client.example': 'main-app',
  'https://partners.staging.client.example':  'partner-portal',
  'https://admin.staging.client.example':         'admin-app',
  // API-internal calls
  'https://api.staging.client.example': 'api-internal',
};

export function resolveAudience(origin: string | undefined): string | null {
  if (!origin) return 'api-internal';
  return audienceMap[origin] ?? null;  // null = rejected
}

Seven explicit origins. Anything else returns null, which the calling code translates into a 401 Unknown origin. The CORS policy says "any subdomain is fine." The JWT system says "only these seven specific origins get tokens." Two access-control systems with conflicting policies — and the stricter one wins.

Testing Every Authenticated Path

We needed to know whether the audience check covers every authenticated operation or just sign-in:

# 1. Get a token from a legitimate origin.
TOKEN=$(curl -s -X POST "$API/api/auth/signin" \
  -H "Origin: https://partners.staging.client.example" \
  -d '{"email":"test@example.org","password":"Secret123"}' \
  | jq -r '.data.access_token')

# 2. Try the token from the evil origin.
curl -s "$API/api/auth/me" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Origin: https://evil.client.example.org"
# → "Unknown origin"

# 3. Try with no Origin header at all.
curl -s "$API/api/auth/me" \
  -H "Authorization: Bearer $TOKEN"
# → "Audience mismatch"
#   (token has aud=partner-portal, no origin resolves to api-internal, mismatch)

# 4. Try with a different legitimate origin.
curl -s "$API/api/auth/me" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Origin: https://app.staging.client.example"
# → "Audience mismatch"
#   (partner-portal ≠ main-app)

Every path blocked. The audience check fires on every authenticated request, not just sign-in. A token minted for partner-portal only works when sent with the partners.staging.client.example Origin header — any other origin (evil, no-origin, different legitimate origin) returns 401.

The Refresh Token Path

Token refresh has the same check:

curl -s -X POST "$API/api/auth/refresh-token" \
  -H "Origin: https://evil.client.example.org" \
  -d "{\"refresh_token\": \"$REFRESH\"}"
# → "Unknown origin"

The attacker cannot refresh a stolen token from an evil origin either.

What Is Still Vulnerable

The audience check only runs on authenticated endpoints. Public endpoints do not need it, and do not get it:

# Public config — readable from any CORS-accepted origin.
curl -s "$API/api/config" \
  -H "Origin: https://evil.client.example.org"
# 200 OK — feature flags returned

# Public catalog — same shape.
curl -s "$API/api/catalog/" \
  -H "Origin: https://evil.client.example.org"
# 200 OK

Public data is readable from any origin that the CORS policy lets through. The data itself is not sensitive — feature flags and public catalog entries — so the impact is low. The client's product owner confirmed this explicitly.

The real remaining risk is XSS on a legitimate subdomain. If an attacker achieves XSS on, say, partners.staging.client.example, they can read the JWT out of localStorage and use it from script running on that legitimate origin. Neither control fires. The audience check sees a legitimate origin and CORS sees an approved one. The two controls only stop arbitrary origins, not legitimate origins compromised via a different bug.

The Severity Downgrade

We originally drafted the CORS wildcard as HIGH. After the exploitation attempt, we downgraded it to MEDIUM, with the residual risk path documented explicitly:

Attack Scenario	CORS	JWT Audience	Net Result
Sign in from an evil subdomain	allowed	blocked	blocked
Use a stolen token from an evil origin	allowed	blocked	blocked
Refresh a stolen token from evil origin	allowed	blocked	blocked
Read public data from an evil subdomain	allowed	n/a	succeeds (not sensitive)
Token use from XSS'd legitimate origin	allowed	allowed	vulnerable

We still recommended the CORS fix, and it landed in the same sprint. Defense in depth means not relying on one independent control to do the work of two. Without the audience map, the CORS bug would have been trivially exploitable. With it, an attacker first needs script execution on a legitimate origin (XSS on partners.staging.client.example or similar), which is a much higher bar.

Why Two Independent Controls Worked

The CORS policy and the JWT audience map were written at different times by different people for different reasons. CORS was set up early in the project, when the team was thinking about subdomain flexibility. The audience map was added later, when multi-portal authentication came in. Neither was deliberately a "backup" for the other.

The two controls do not need to know about each other. CORS asks whether the browser is allowed to send the request; the audience check asks whether the token is valid for this origin. Different questions, enforced in different places. When one is weaker, the other holds — which is what happened here.

Two things this engagement changed for us

First: try to exploit a finding end to end before assigning severity. The path from "this check is missing" to "I can steal user data" often has unexpected obstacles, and the severity should reflect the real impact, not the impact of the broken control in isolation.

Second: unify the CORS allowlist and the JWT audience list into one source-of-truth file, two readers. When the two lists drift, defense in depth becomes defense in name only. On this engagement, we consolidated both into a single origins.config.ts at boot; the CORS middleware and the audience resolver now read from the same list.

The residual risk (XSS on a legitimate subdomain) went into the report explicitly, because a layered defense that holds against arbitrary origins still needs the reader to know what would open the attack.

From DevOps to DevSecOps: How I Became Clearview's In-House Pen-Tester

2026-07-11T11:00:00+02:00

It took me a decade of shipping things reliably before I started spending my nights trying to break them.

For a long time I was the person who shipped it. CI/CD pipelines, Terraform modules, the monitoring stack, the "why is staging down?" Slack messages at midnight. I have been doing that work for nearly two decades, even before it was called DevOps.

Earlier this year I took on a different job inside Clearview Team. Now, before any new feature ships, somebody spends a day trying to break it. Often, that somebody is me.

The Night the Job Description Changed

It was March 2026. I could not sleep. Too much coffee, too many open browser tabs, and a Hacker News thread I could not stop reading about a breach that turned out to be a misconfiguration — a public-read S3 bucket exposing millions of records.

"That could never happen to us," I thought. "We have policies. We check these things." Then I got up, opened my laptop, and started checking anyway.

Four hours later I had found three public S3 buckets in our own infrastructure. None of them held customer data — they held logs and config files, enough information for an attacker to draw a map of the system.

The Terraform that created those buckets had been written in the early days, before we had standards, before we had a review process, before a single checkov rule ran in the pipeline. Nobody had ever gone back. They had terraform apply-ed once and moved on.

Those buckets were not somebody else's mistake. They were my Terraform, my missing review process. Finding them was less a security win than a critique of how I had been working for years.

I fixed them that morning, wrote up the incident, told the team. And then I could not stop.

What an Engineer Reads at Night When They Cannot Sleep

I started spending evenings on a different reading list:

JWT vulnerabilities — specifically how stateless tokens make revocation almost impossible
SQL injection patterns that still work in 2026 — parameterised queries handle the textbook case, but ORMs like TypeORM ship their own foot-guns (raw queries, addSelect on relations, QueryBuilder with string interpolation)
CORS bypasses — wildcard subdomain matching, reflected origins, preflight caching abuse
What trust proxy actually does in Express — and why getting the hop count wrong means your rate limiter trusts attacker-supplied IPs, which matters a great deal if you are sitting behind CloudFront and an ALB

I am not claiming to be a research-level expert in any of these. I am claiming curiosity. I went deep enough on each to spot the same patterns in our own codebases and to ask sharper questions during reviews. That is different from being a full pen tester, and I try not to confuse the two.

Slowly, I started looking at our code differently. Every endpoint became a question about what happens when somebody sends weird input; every configuration became a suspicion about whether it was actually secure or only looked secure.

I was running curl against staging at strange hours, creating test invoices, reading the error messages line by line, mapping out what existed. I was becoming paranoid, in a mostly-good way.

The hard part about paranoia is knowing when to put it down. I have lost sleep over things that turned out to be fine. I have filed false positives with embarrassing confidence — I once shipped a critical report about a writable S3 bucket because I had tested with aws s3 cp --dryrun, which validates syntax but not IAM. The actual upload failed with AccessDenied. I had to correct my own report the next morning.

Paranoia is useful, but it needs guardrails: a checklist, a second opinion, the willingness to say I was wrong out loud and in writing.

What Tools Catch and What They Miss

People often ask why I do so much of this by hand instead of running Burp Suite, OWASP ZAP, or one of the big enterprise scanners.

I do use those tools. They are excellent at coverage. A DAST scanner will methodically poke every parameter on every endpoint. A dependency scanner will pick up known CVEs in your package-lock.json. These are table stakes — you should run them no matter who is on your team.

But automated tools find known patterns. What I find by hand is different:

The CORS rule that accepts any subdomain — technically valid regex, but allows attacker-controlled origins
The scientific-notation invoice for one million dollars — 1e6 passes IsNumber() validation because it is a number, but the business logic never expected six figures
The internal fields that nobody should be allowed to set, but anyone can — Object.assign(user, req.body) happily merges isDeleted and role along with everything else
The webhook that accepts whatever you send it because validation was deferred to an async queue, which also does not validate — a design tradeoff that quietly became a security gap

None of these have a CVE number. They are business-logic vulnerabilities — the consequence of how this specific code, this specific ORM configuration, and this specific architectural decision interact. Scanners do not know a client's business rules; the person who reviews the diff does.

That said, manual testing does not scale. I cannot personally curl every endpoint before every release for every Clearview client. So we run both — automated tools for breadth, a human for depth — and every manual finding becomes a custom rule that catches the next instance automatically.

The Role We Wrote Around It

At some point my late-night reading turned into part of the job. It happened gradually.

First I added security checks inside our Terraform modules — automated S3 bucket policy validation, IAM policy scanning, the kind of guard that would have caught those public buckets before they ever reached production.

Then I started doing quiet pre-deployment reviews. Not formal audits. Just "let me look at this endpoint before it goes live."

Then, on a real client project, I found a vulnerability that would have let anyone create invoices without authentication. No login. No rate limiting. Just curl -X POST and a JSON payload, every minute, forever.

That was the inflection point. Not because it was the scariest finding I have ever shipped a report on, but because there was a dollar sign attached. The team — and the client — could see exactly what an attacker would have done with it. Abstract security risk is easy to deprioritise. A vulnerability that prints fake invoices in your accounting system is not.

After that, the role was real.

Today, at Clearview Team, every new feature we ship to a client gets sent through a pen-tester pass before it lands in production. That pass is a known step in the sprint — scheduled and time-boxed. It is the same engineer (often me) sitting down with the diff and asking, on purpose, how do I break this?

That is what an in-house pen tester gets you that an external annual audit does not. The feedback loop is short, and the fix usually lands in the same sprint that introduced the risk, not nine months later in a remediation backlog.

What I Catch on the Pen-Tester Pass

Here is the shortlist of categories that show up over and over again across stacks and clients:

Auth holes on new endpoints. Somebody adds a route, forgets the auth decorator, ships. I catch this before the route hits production by asserting every endpoint either carries @Authorized() or sits on an explicit public-endpoint allowlist.
Mass assignment via ORM helpers. Object.assign(entity, dto) looks innocent. It is not, when the entity has fields the user is never supposed to set. I run a curl against the endpoint with a payload that sets every internal field and see what survives.
IDOR / BOLA on numeric IDs. Anything in a URL of the form /invoices/:id. I swap in another tenant's id and read the response.
Input validation that is actually type checking. IsNumber() lets 1e9 through. IsString() lets a 50 MB payload through. I send the obvious edge cases and read the error messages.
Rate limits that count the wrong IP. If trust proxy is misconfigured, every request looks like it came from the load balancer. I send ten thousand requests with rotating X-Forwarded-For headers and watch what the limiter thinks.
Infrastructure drift since the last engagement. A bucket that was private six months ago is public now because someone toggled a setting in the console. I re-run checkov and tfsec against the live Terraform plan, not just the repo.

None of this is novel. The novelty is that it happens every sprint, by the same person, on the same engagement. Familiarity with the codebase is the hidden multiplier — an external auditor will never find the business-logic invoice bug, because they will not know what an invoice is supposed to look like for this client.

What It Gets the Client

Clearview Team takes on web and mobile build engagements for startups and scaling companies. The teams we work with are usually small, usually shipping fast, and usually one configuration drift away from a story they would rather not be in.

What the client gets:

A feature is never the first thing to find a vulnerability in production. A peer reading the diff with an attacker mindset is.
They don't pay extra for the security review. It is not a separate procurement, a separate vendor, or a separate quarter. It rides on the same sprint that ships the feature.
When we find something, we hand them the fix. I am an engineer. They do not get a thirty-page report — they get a pull request.
The codebase gets harder to break the longer we work on it. Every finding becomes a CI rule that catches the next instance automatically. The boundary tightens by itself.
No surprise certifications, no surprise breach. When a SOC 2 auditor or a customer due-diligence questionnaire shows up, the answers are already true.

I will not put a percentage on what this prevents — I do not know what would have shipped without it, and Clearview does not invent numbers for sales copy. In the months since this role has been real, no Clearview client has shipped a production-incident-class security bug from a feature that went through the pen-tester pass. The ones we have found in production were inherited from before our engagement, and we wrote up exactly what we found and exactly what we changed.

What This Costs Us

Adding this role to every engagement is not free.

A sprint takes a little longer. Time-boxed, but real.
Some features get re-scoped. When we hit something structural rather than local during the review, the fix is sometimes an architectural change. That conversation is uncomfortable, and we have it anyway.
The pen tester needs a peer. I cannot review my own work alone — I have been wrong before and I will be again. The second opinion is part of the process, not an add-on.

We treat all three as features of the model rather than bugs in it. A feature ships a little slower and holds up better in production.

The Mindset Shift, for Anyone Considering the Move

If you are a DevOps engineer reading this and wondering whether you should make the same shift — you do not need a certification. You do not need to become a full-time pen tester. You need to start asking what if? about the systems you already own.

Five things that helped me, in order:

Audit your own infrastructure first. Run checkov on your Terraform. Check your S3 bucket policies. Review your IAM roles. You will probably find something, and it will motivate everything that follows.
Learn one attack vector deeply. Mass assignment if you use an ORM with auto-mapping. BOLA/IDOR if your API uses sequential IDs. Authentication bypass if you run your own auth flow. SSRF if your app makes outbound HTTP calls based on user input. Pick one. Learn it well enough to spot it in the wild. Then pick another.
Read the detailed breach post-mortems. Not the sensational ones. Cloudflare's. GitLab's. The ones that explain the root cause, the timeline, and what changed. That is where real attacks live.
Break something — in staging, with permission. Find a vulnerability. Exploit it responsibly. Document the steps. See how it feels. That feeling — the mix of excitement and dread — is how you know you are thinking like an attacker.
Turn every finding into a check. Every manual finding should become a rule that runs automatically next time. Found a public S3 bucket? Add a checkov rule. Found a missing auth decorator? Add a CI test that verifies every route is on the auth list. The goal is to make yourself unnecessary for the easy stuff, so you can spend your time on the hard stuff.

The paranoia helps, but the checklists help more.

An Event-Driven Architecture Case Study: Lessons from a Real-World Application

2025-04-07T19:32:53+02:00

Over the past decade, Clearview Team has built and maintained a pair of interconnected platforms for a confidential awards group. This is the event-driven AWS architecture that keeps both systems observable.

Prefer a downloadable version Read the case study as a PDF Download

The Client

A Confidential Non-profit and Awards Group *Membership Operations, Automated and Simplified *

At one time managed primarily through Excel spreadsheets and mail-in forms, the client’s membership system was functional but time-intensive for its administrators. With the benefit of an automated, online membership management system, client’s Board of Directors wisely contracted Clearview to combine member-related functions into one web-based platform.

More than a decade later, we’re still working in close partnership with the client to maintain and streamline member processes and relations through a secure, custom-built online portal. We’re continually finding new ways to empower administrators and members alike with rich tools and features, supporting this 501(c)(3) non-profit’s continued growth.

Purpose-built member dashboard with quick access to useful member functions including online signup, membership upgrade, and renewal; info & address changes; member-only services and perks; automated password reset & forgotten login recovery; notification preferences.
Powerful administrator dashboard, complete with tiered admin privileges, quick member lookup & sorting, bulk member management, numerous import/export options, and detailed reports.
Secure, multi-method payment processing with in-dashboard payment logs, receipts, and refund processing.
Newsletter and notification management including automated synchronization with MailChimp newsletter and SendGrid transactional email platforms.
Scheduled self-maintenance and member relations tasks including proactive address validation and expiration notifications.

The Digital Workflow Platform

Annual Intake and Review Workflow

The client also runs an annual intake-and-review cycle that once involved sorting through over a thousand hand-mailed physical packages. This veritable mountain of material was then shuffled through very extensive multi-stage review and selection processes by hand.

Since joining the client’s team in 2010, Clearview has systematically brought each of these processes together in a single online platform.

Fully web-based intake process, including secure online payment; upload of video clips, documents, and signoff forms; followup intake for supplementary materials.
Automated media verification, storage, and transcoding, utilizing Vimeo API integration for high-quality, secure, and familiar playback for reviewers. All media is secured against unauthorized download.
Advanced partner dashboard, enabling organizations with many moving parts to delegate intake tasks to multiple designated contributors and divide labor.
Review management, including secure reviewer application and selection process; a dedicated review portal with high-quality material review, shortlisting, issue flagging, and discussion features; selection and notification workflows.
Selection management, including application and eligibility processes and a secure selection interface.
Tight integration with the organization’s membership database for permissions verification.
Extensive administration dashboard, including tiered admin roles; organization and intake management; category and field management; payment, receipt, and transaction management; flexible sponsor-credits system; deadline management, late fee rules, and exceptions; numerous import/export tools.
Secure, triple-redundant data archival following each year’s cycle.
On-call support, including email support for contributors, reviewers, and participants.

The Project

Clearview Team's engineers and clients are spread across time zones, so monitoring and incident response carry real coordination overhead. This engagement is where we chose Event-driven architecture.

Challenges

The challenges we aim to solve with this approach:

Decoupling of services: To improve the agility of the application we designed our system not to be tightly coupled.
Real-time responsiveness: The application services operate on event-driven communication, which means the services can produce an event in real time. This enables our on-call engineers to react to important events that are produced by either user actions or system events.
Improved observability: The event-driven approach improves system observability. By analyzing event flows, we constantly monitor and improve the system behavior, identify bottlenecks, and locate the root cause of issues more effectively.

Deployment

The application is deployed on the AWS Cloud platform as AWS already provides all the communication buses that we need to produce, broker, and consume events.

Event Types We have covered the most important event types in the app, such as:

Finance events
Membership events
Scheduled job events
Health check events.

Goals

This project aims to solve the client’s requirements to create and continuously improve the membership and awards system while implementing a modern system architecture. More details about what kind of problems the application is designed to solve can be seen in the diagram below.

System Architecture

This application is built around separate services, each running inside its own container. Services communicate through events — state changes that trigger actions elsewhere in the system.

Event-driven Architecture

An event is a state change — a user placing an order, a payment failing, a scheduled job firing. Services produce events and other services consume them, without needing to know about each other directly.

Event-driven architecture is often called EDA for short.

The rest of this post covers how we wired the event bus for job scheduling and monitoring — the piece that lets our team get notified in real-time and react before downtime reaches users.

Event-driven Architecture Showcase

Below is a diagram showing what a typical EDA system looks like and how it functions.

AWS based Event-driven architecture

The Application

The application is built on NodeJS with an API service serving multiple front-end domains. This approach has served us well. We have been running the app as a container on AWS.

The diagram shows API and the DB secured and isolated in the VPC. Frontend is served to the client via CloudFront which pulls the website assets from the S3 bucket.

Please keep in mind that the client communicates with the API after it has been loaded on the client-side rendered website.

Event Producers

Events are generated from various services and can be produced either by a client action or by a scheduled job.

API Event Producer

The best example of a client-produced event is any type of event related to credit card payments. The app uses the “@aws-sdk/client-eventbridge” library to communicate with the AWS Event Bridge service.

API Events List

The core services of the app produce events. Below is a list of services with a description of what each does.

The diagram shows API Services and Controllers that send events.

API Produced Sample Events

Finance Related Events Below you can find payment processing failed sample events

Health Check Events

Mailing List Events

Scheduler Event Producer

As it was not necessary to re-implement the wheel and create our scheduling service, we relied on the well-tested and proven AWS EventBridge Scheduler for the scheduling system. For this purpose a custom and dedicated Event Bus was created and named Client Production Scheduler Bus.

In this case, the Event Producer is the Scheduler itself.

The diagram shows events that are sent from the first event producer which is the Event Bridge Scheduler received and processed by The Production Event Bus, parsed by AWS Lambda, and at the end sent to a Slack webhook.

As per the diagram every cron job has the following flow:

a scheduler of recurring type that produces an event and sends it to the custom scheduler bus,
an endpoint that is defined as part of the api_destination event rule.

Evaluated Jobs

The complete list can be found below.

The diagram shows the list of Scheduled jobs, their endpoints, and json payload event they generate which gets sent to the Event Bus.

Company Statistic Scheduled Job

The diagram shows events that are produced the Event Bridge Scheduler, received and processed by The Production Event Bus, and forwarded to the target which is unique endpoint via API Destination.

Event Scheduler Job Notes:

Recurring schedule type
Cron expression style
Runs every 8 hours
Target: Scheduler Bus
API PutEvents

Event Rule

The bus receives the event and triggers the api cron company statistic rule.

API Destination

The Infrastructure

In this section, we will cover how the AWS infrastructure was set up, the services that send events to the default Event Bus, and how we process and consume those events.

The AWS Account

We used AWS Organizations to separate Production and Development environments. This is important as infrastructure-related events are sent to the default event bus. We wanted production and development events not to trigger the same rules as this could lead to unexpected behaviors.

The diagram shows how we separated production and development accounts while managing both with a single management account.

It makes it easier to have two environments not mixing up as we know for a fact that development events will not end up on the production Event Bus causing potential problems.

AWS RDS

The database service’s health and availability are very important for the app’s reliability. Therefore, we have a setup of monitoring database events with Slack notifications.

Every time there is an event such as the database service backing up an event is created and sent to the Event Bus. Such an event will end up as a Slack notification and be read by our team.

The diagram shows events that are sent from the MariaDB service, received and processed by AWS EventBridge, parsed by AWS Lambda, and at the end sent to a Slack webhook.

We decided that the following event categories will be forwarded to Slack: Notification, availability, backup, failure, low storage, maintenance and recovery.

The EventBridge Rule

AWS Health

The diagram shows events that are sent from the AWS Health service, received and processed by AWS EventBridge, parsed by AWS Lambda, and at the end sent to a Slack webhook.

Events type can be one of the following categories:

accountNotification
issue
scheduledChange

AWS Health events are sent to the default Event Bus. We want to react when we receive an event from the AWS Health service, therefore we create rules. For example, you can use AWS Health to receive email notifications if you have AWS resources in your AWS account scheduled for updates, such as EC2 instances. Below you can find examples of such rules.

EC2 service

Sample rules

Trigger on Every Event

Trigger on Specific Service-related Event

Rule for a specific service and event type category.

In this example, we will create a rule so that EventBridge reacts to the following.

Multiple Services and Event Type Categories

The examples in the previous procedure show you how to create a rule for a single service and event type category. You can also create a rule for multiple services and event-type categories. This means that you don’t have to create a separate rule for each service and category that you want to monitor. To do so, you must edit the event pattern as per following example

Example Rule

Migrating EC2 to AWS Graviton (ARM): A Cost-Reduction Case Study

2024-02-11T05:59:43+01:00

The world of cloud computing is changing fast, that is a fact. Some changes have a more significant impact, as was the case with the rise of AWS Graviton processors.

The whole tech industry is experiencing the growing adoption of ARM architecture, so why should it not be the case with the Cloud as well?

This article focuses on the impact of AWS Graviton on the Cloud offering, the shift towards the ARM architecture, and how Clearview is helping clients jump on the Graviton train and save on costs up to 20%.

Key points on AWS Graviton

AWS Graviton processors are becoming popular because of their superior performance and energy efficiency in cloud environments which reduces your carbon footprint,
Adapting the ARM architecture requires data migration or possible code improvements before benefiting from the superior improvements in both cost and performance.

The Rise of AWS Graviton

Understanding AWS Graviton Processors

AWS Graviton processors are powered by ARM architecture, enabling performance and cost optimizations, which represent a significant advancement in cloud computing, compared to traditional x86 processors.

Graviton processors are optimized for Cloud workloads with the following benefits:

Large L1 and L2 caches for every virtual central processing unit (vCPU), meaning that most of your workload never burdens RAM,
Every virtual CPU is a physical core, meaning more isolation between virtual CPUs,
Cores are connected in a fast mesh with ~2TB/s of bisection bandwidth, which allows applications to move very quickly from core to core,
Graviton’s RAM architecture means you don’t need to worry about application memory allocation, or which cores are running the application.

Generations of AWS Graviton

Graviton processors come in various generations with each generation offering various advances in processing power and efficiency.

Graviton

The first generation ARM architecture-based, known as A1 type, Graviton-powered EC2 instances were launched in 2018. Not every region is still providing this first-generation instance type. Use the following awscli command to check if a region supports the A1 instance type.

aws ec2 describe-instance-type-offerings \
    --location-type "availability-zone" \
    --filters Name=location,Values=us-east-2a \
    --region us-east-2 \
    --query "InstanceTypeOfferings[*].[InstanceType]" \
    --output text \
    --profile default | sort | grep a1

This processor features:

64-bit ARM Neoverse cores,
The instances are up to 40% less expensive than the same number of vCPUs and DRAM available in other instance types.

Graviton 2

Launched in 2019 it represents the second generation of AWS Graviton processes. Graviton2-based instance types offer up to 40% better price performance than fifth-generation instances.

Graviton 2 Instance types

We now have 12 instance families (M6g, M6gd, C6g, C6gd, C6gn, R6g, R6gd, T4g, X2gd, Im4gn, Is4gen, and G5g) that are powered by AWS Graviton2 processors that provide significant price performance benefits for a wide range of workloads.

Graviton 3

Launched in 2022, it is the third and latest generation that is generally available. When compared to AWS Graviton2 processors it provides:

up to 25% better computing performance,
up to 2x better floating-point performance,
up to 2x faster crypto performance, and
up to 3x better ML performance,
support for bfloat16,
and features the latest DDR5 memory, which provides 50% more memory bandwidth compared to DDR4.
up to 60% less energy usage for the same performance than comparable EC2 instances.

Graviton 3 instance types

All 7th generation instance types containing “g” means it is based on AWS Graviton 3, the silicon designed by AWS. Examples:

C7g, C7gd, C7gn: the “C” instance family is designed for compute-intensive workloads,
M7g, M7gd: “G” instance family is designed for general-purpose workloads with balanced computing, memory, and networking,
R7g, R7gd: The “R” instance family is designed for memory-intensive workloads.

Graviton 4

Launched in late November 2023 Graviton 4 represents the fourth generation and it is still in preview only, with general availability planned in Q1 of 2024. When compared to Graviton3 processors it provides:

up to 30% better computing performance,
50% more cores,
75% more memory bandwidth.

Graviton4 will be available in memory-optimized Amazon EC2 R8g instances, which are currently in preview only. R8g instances offer larger instance sizes with up to 3x more vCPUs and 3x more memory than current generation R7g instances. To learn more about Graviton4-based R8g instances, visit this link.

Switch your AWS Services to Graviton

Here we will describe which services we have switched over to AWS Graviton.

Some services can be switched with a few clicks, while others require a full data migration following best practices.

AWS EC2

Following is a tutorial on how we switched EC2 instances. Please note that t4g is run on Graviton 2. Migrating our NodeJScode to be compatible with ARM architecture meant upgrading a few packages only.

Note: do not forget to set your awscli profile as in all of these examples the default one is used.

Find ARM LTS Ubuntu Images

For the latest LTS which is 22.04 LTS (Jammy Jellyfish), currently.

aws ec2 describe-images \
    --filters "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-arm64-server-*" "Name=architecture,Values=arm64" \
    --query "sort_by(Images, &CreationDate)[].[Name, ImageId, Architecture]"  \
    --output text \
    --region us-east-1 \
    --profile=default

This command will show you the IDs of the images that support the ARM architecture. Choose one and set it at the next step.

Set AMI ID

As the AMI ID has been found in the previous command, we need to save it to a variable.

AMI_ID=ami-02ddaf75821f25213

List EC2 Key Pairs

The following commands will show keys that were added previously. If you get an empty result you need to add your public SSH key.

aws ec2 describe-key-pairs \
  --region us-east-1 \
  --profile=default

Find the Availability Zone

The zone is directly connected to the subnet.

aws ec2 describe-instance-type-offerings \
--location-type availability-zone \
    --filters Name=instance-type,Values=t4g.small \
    --region us-east-1 \
    --profile=default

Note: This step is very important as you need to find the what availability zone supports your instance size.

Find the Subnet and Security Group

We need to find the correct Subnet and SG.

List Security Groups

aws ec2 describe-security-groups \
    --query "SecurityGroups[*].{Name:GroupName,ID:GroupId}" \
    --region us-east-1 \
    --profile=default

List subnets

aws ec2 describe-subnets \
    --region us-east-1 \
    --profile=default

Note: Find the correct subnet that is used by your old EC2 instance.

Create the EC2 Instance

Please note that the following command is an example only and that you need to change the values for:

the key name,
the subnet ID,
the security group ID,
set your tags,
use your profile name.

aws ec2 run-instances --image-id $AMI_ID --instance-type t4g.small  \
    --key-name nedim \
    --subnet-id subnet-000000000000000 \
    --security-group-ids sg-00000000000000000 \
    --associate-public-ip-address \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ubuntu-test-graviton}]' \
    --region us-east-1 \
    --profile=default

Prepare the image

Connect to the VM, and run the system upgrade.

New Server Configuration with Ansible

Since we used Ansible to set up the old server the process of setting up an identically configured server was a piece of cake. All we had to do was change the hostname in the host inventory file add the new server and rerun the playbooks.

Volume size

Make sure the new disk volume is identical to the size of the old server’s disk volume as we will copy over data from the old server to the new one.

Data Migration

Using RSYNC, copy over the data from the old server to the new Graviton-based one, using the SSH protocol.

rsync -avz -e ssh /var/www/ root@enew-server.compute-1.amazonaws.com:/var/www/

Note: This data migration does not guarantee that the code will work 100%, you will need to deploy and build the code again to be sure it works as expected. In our case, we needed to upgrade several NodeJS packages.

AWS RDS

With RDS instances no migration is necessary and you can take the easy way of just modifying the instance class of your instances if your database engine version supports it.

Determine if the current database version meets the minimum required version for moving to Graviton2.

Version supporting Graviton:

MySQL: 8.0.17 and higher,
PostgreSQL: 12.3, 13 and higher,
MariaDB: 10.4.13, 10.5 and higher.

If your RDS instance isn’t at a version supported by Graviton2, you need to upgrade to a supported version.

AWS ElastiCache for Redis

Migrating Amazon Elasticache for Redis and Memcached to Graviton2-based instances offers a smooth transition, only a change of instance type is required, with a performance improvement of up to 45%, all at a 20% lower cost compared to similar x86-based instances.

The Beginning of a New Age in Cloud Computing

In conclusion, the rise of AWS Graviton and ARM architecture represents a change in cloud computing that impacts the whole IT industry. These technologies bring together improved performance, cost-effectiveness, and energy efficiency. As more organizations adopt these technologies, we can expect a future where the flexibility of ARM architecture and the efficiency of Graviton processors set the standard for cloud infrastructure.

Therefore, do not miss the chance to make the switch and be part of the new standard.

Deploying Node Containers on AWS App Runner with Terraform

2024-01-11T07:01:42+01:00

The Smart Way to Deploy Containers on The Cloud

Say hello to AWS AppRunner

Containers on AWS AppRunner

Why AppRunner?

It is easy to deploy and manage

If you tried to deploy containers on the cloud you have experienced anything but an easy process. It has been easy to deploy containers on your local machine but deploying them on the Cloud has not been an easy process.

That was true until AWS AppRunner was introduced back in 2021.

It is a great solution for startups

Here at Clearview, we have found that AppRunner can be of very benefit to startup projects as it is cost-effective because of its on-demand pricing model and it is a fully managed service.

This will result in less time spent on setting up the infrastructure which saves us time and money.

It has an attractive pricing model

What is great about AppRunner is that when the container is in an idle state you only pay for the memory provisioned in each container instance. When a container is in the idle state it is considered to be a Provisioned container instance. Once the container starts to use CPU resources, it will be considered an Active container instance and you will be charged for the CPU resources used.

As you can pause and resume the service you can as well additionally save on costs.

In summary

Provisioned container instances — charged for memory ($0.007 / GB-hour),
Active container instances — charged for both memory and CPU resources used ($0.064 / vCPU-hour, $0.007 / GB-hour),
Paused instances— CPU and memory are not charged.

What can AppRunner do?

Since AWS App Runner is a fully managed service it will

automatically deploy web applications by detecting a new container image that has been pushed to the container registry,
automatically scale and load balance to meet the traffic needs.

Automatic deployment, scaling, and load balancing make it a desirable technology.

Easily Managed by Terraform

By adding Terraform to the mix we can provision this technology even faster. Later in this article, we will provide examples of code on how to set up AppRunner with Terraform.

History

Back when it was introduced it was still an early tech and still not developed and mature but in the last two years, it has grown significantly. Here are some improvements that we have witnessed while using the tech:

Added support for more regions across the world,
Reduced duration for deploying applications using container images, which is about a 30–40% reduction in deployment time depending on the container image size,
Increased instance startup time from one to a maximum of five minutes which enables slower instances that need more time of startup to be used,
Added dual stack support for incoming traffic through public endpoints, both IPv4 and IPv6 endpoints, simultaneously,
Reduced the time taken for image-based service deployment, Added support for immediate deployment failure if App Runner couldn’t pull an image.

Provisioning AppRunner with Terraform

In this demonstration, we will provision AWS services with Terraform as follows:

ECR for hosting container images,
NodeJS API service on AppRunner,
NuxtJS FrontEnd service on AppRunner.

Local variables are defined at the top of the .tf file to be straightforward about what values should be changed depending on the project.

Diagram

In the diagram, it is clear that the publicly exposed service is the FrontEnd service, which then connects to the API service. To point and connect one service to the other environment variables are used.

Provision ECR

This is an example of how to provision the ECR for API.

aws_apprunner_service_with_terraform/ecr.tf at e4c96bc5e856753515c2c640e853f0deeef5e71f · clearview/aws_apprunner_service_with_terraform

# my_api
resource "aws_ecr_repository" "my_ecr_api" {
  name                 = "my_ecr/my_api"
  image_tag_mutability = "MUTABLE"
  image_scanning_configuration {
    scan_on_push = true
  }
}

resource "aws_ecr_lifecycle_policy" "my_ecr_api" {
  repository = aws_ecr_repository.my_ecr_api.name

  policy = jsonencode({
    rules = [{
      rulePriority = 1
      description  = "keep last 10 images"
      action = {
        type = "expire"
      }
      selection = {
        tagStatus   = "any"
        countType   = "imageCountMoreThan"
        countNumber = 10
      }
    }]
  })
}

output "my_ecr_api_registry_id" {
  value       = aws_ecr_repository.my_ecr_api.registry_id
  description = ""
}
output "my_ecr_api_repository_url" {
  value       = aws_ecr_repository.my_ecr_api.repository_url
  description = ""
}

Provision of the API Service

In this example, the health check is performed against a custom endpoint available on the/api/healtcheck URN, but that might be different in your case.

To be safe and if you do not have a health check endpoint yet just use / .

This instance is provisioned with the following configuration:

1024 CPU units,
2048 MB of memory,
The health check endpoint is located at /api/healthcheck ,
Container port running on 3000 .

aws_apprunner_service_with_terraform/api_service.tf at e4c96bc5e856753515c2c640e853f0deeef5e71f · clearview/aws_apprunner_service_with_terraform

locals {
  api_port                   = 3000
  api_domain                 = "api.mydomain.team"
  api_git_repo               = "my/my_api"
  api_development_git_branch = "dev"
  api_ecr_image_development  = "${aws_ecr_repository.my_api_ecr.repository_url}:dev"
}
resource "aws_apprunner_service" "my_api" {

  depends_on = [
    aws_ecr_repository.my_api_ecr,
    aws_iam_role.my_app_runner_roles,
    aws_apprunner_vpc_connector.my_vpc_connector
  ]

  service_name = "my_api"

  source_configuration {
    authentication_configuration {
      access_role_arn = aws_iam_role.my_app_runner_roles.arn
    }

    image_repository {
      image_identifier      = local.api_ecr_image_development
      image_repository_type = "ECR"
      image_configuration {
        port = 3000
        runtime_environment_variables = {
          # Server Port
          PORT = "3000"
          #
          NODE_ENV       = "development"
          LOG_LEVEL      = "debug"
          PORT           = local.api_port
          CONTAINER_PORT = local.api_port
          HOST_PORT      = local.api_port
        }
        runtime_environment_secrets = {
        }
      }
    }
    auto_deployments_enabled = true
  }

  health_check_configuration {
    path                = "/api/healthcheck"
    healthy_threshold   = 1
    interval            = 5
    protocol            = "HTTP"
    timeout             = 20
    unhealthy_threshold = 20
  }

  instance_configuration {
    cpu    = "2048"
    memory = "4096"
  }

  network_configuration {
    egress_configuration {
      egress_type       = "VPC"
      vpc_connector_arn = aws_apprunner_vpc_connector.my_vpc_connector.arn
    }
  }

  tags = {
    Name = "my-my_api-apprunner-service"
  }
}

output "my_api_apprunner_my_api_service_url" {
  value       = aws_apprunner_service.my_api.service_url
  description = ""
}

Provision of the FrontEnd service

This instance is provisioned with the following configuration:

1024 CPU units,
2048 MB of memory,
The health check endpoint is located at /ping .

aws_apprunner_service_with_terraform/fe_service.tf at e4c96bc5e856753515c2c640e853f0deeef5e71f · clearview/aws_apprunner_service_with_terraform

locals {
  my_fe_port                   = 3001
  my_fe_domain                 = "frontend.my-domain.org"
  my_fe_apprunner_domain       = "members.development.my-domain.org"
  my_fe_ecr_image_development  = "${aws_ecr_repository.my_fe_ecr.repository_url}:dev"
  my_fe_git_repo               = "clearview/my-members-fe"
  my_fe_development_git_branch = "dev"
}

resource "aws_apprunner_service" "my_fe" {
  depends_on = [
    aws_ecr_repository.my_fe_ecr
  ]

  service_name = "my_fe"

  source_configuration {
    authentication_configuration {
      access_role_arn = aws_iam_role.my_app_runner_roles.arn
    }

    image_repository {
      image_identifier      = local.my_fe_ecr_image_development
      image_repository_type = "ECR"
      image_configuration {
        port          = local.my_fe_port
        start_command = "yarn dev"
        runtime_environment_variables = {
          HOST           = "0.0.0.0"
          HOSTNAME       = "0.0.0.0"
          NITRO_HOST     = "0.0.0.0"
          ENV            = "development"
          NODE_ENV       = "development"
          PORT           = local.my_fe_port
          NITRO_PORT     = local.my_fe_port
          CONTAINER_PORT = local.my_fe_port
          HOST_PORT      = local.my_fe_port
          # API
          PROXY_TARGET      = "https://${local.api_domain}"
          NUXT_API_BASE_URL = "https://${local.api_domain}"
          # Nuxt
          NUXT_PUBLIC_DEV      = true
          NUXT_PUBLIC_DEBUG    = true
          NUXT_PUBLIC_SITE_URL = "http://localhost:${local.my_fe_port}"
          NUXT_PORT            = local.my_fe_port
          NUXT_HOST            = "0.0.0.0"
          # Debug
          LOG_LEVEL = "debug"
        }
      }
    }
    auto_deployments_enabled = true
  }

  health_check_configuration {
    path                = "/ping"
    healthy_threshold   = 1
    interval            = 20
    protocol            = "HTTP"
    timeout             = 19
    unhealthy_threshold = 20
  }

  instance_configuration {
    cpu    = "1024"
    memory = "2048"
  }

  network_configuration {
    egress_configuration {
      egress_type       = "VPC"
      vpc_connector_arn = aws_apprunner_vpc_connector.my_vpc_connector.arn
    }

    ingress_configuration {
      is_publicly_accessible = true
    }

  }

  tags = {
    Name = "my-my_fe-apprunner-service"
  }
}

output "my_fe_apprunner_my_fe_service_url" {
  value       = aws_apprunner_service.my_fe.service_url
  description = ""
}

Conclusion

In this article, we covered AppRunner from its humble beginnings to the powerful technology it has proven to be today. We provided useful diagrams and Terraform code for easy provisioning of service.

If you have any questions please do not hesitate to contact our team.

AWS Budgets and Alerts with Terraform: A Cost-Monitoring Case Study

2024-01-03T13:47:03+01:00

Monitor and Control Amazon Cloud Costs with Terraform

For every Cloud project, you need to set budgets and alerts with Terraform

AWS Budgets with Terraform

Why set up a budget for your cloud account?

If you wonder why most startups fail it is simple, they run out of money and there have been many horror stories about crazy big cloud bills.

At Clearview, when we put a new client on the Cloud, we know they will get a bill eventually. It’s our responsibility to make sure that the bill will not be a big and sudden surprise.

Setting up the budgets

Setting the AWS Cloud budget

The first step is to set budgets and alerts. We usually set:

The Account wide budget,
The Most used service budget,
The Most important tag budget.

Every budget sends out an email notification when the budget cost threshold is reached, in these two cases:

The ACTUAL cost which will be triggered once that amount has already been spent,
The FORECASTED cost notification will be triggered earlier as the cost is based on prediction based on your past usage.

As we are a tech company the budgets are set using Terraform and in the next section, you can find diagrams and code samples that explain how budgets and notifications work.

Monthly Account Budget

Set your Monthly AWS Budget.

This budget tracks account-wide costs.

resource "aws_budgets_budget" "monthly_account_budget" {
  name         = "Monthly Budget for my Account"
  budget_type  = "COST"
  limit_amount = "500"
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 90
    threshold_type             = "PERCENTAGE"
    notification_type          = "ACTUAL"
    subscriber_email_addresses = ["email@my.team"]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 100
    threshold_type             = "PERCENTAGE"
    notification_type          = "FORECASTED"
    subscriber_email_addresses = ["email@my.team"]
  }
}

aws_budgets_notifications_with_terraform/monthly_account_budget.tf at 0b8af27933b32c27c7b09c4b633724ed17fe82b9 · clearview/aws_budgets_notifications_with_terraform

The notification block arguments

Each notification block inside aws_budgets_budget controls one alert. The five arguments the AWS provider expects are:

comparison_operator — one of GREATER_THAN, LESS_THAN, or EQUAL_TO. For a spend budget you almost always use GREATER_THAN.
threshold — a number. Compared against the budgeted amount.
threshold_type — PERCENTAGE or ABSOLUTE_VALUE. Percentage is easier to reason about because you don't have to rewrite the block when the budget limit changes.
notification_type — ACTUAL (the current spend) or FORECASTED (AWS's prediction based on past usage). Set one of each per budget so you get both an early warning and a real breach alert.
subscriber_email_addresses — a list of email strings. This is where the alert lands. Every address in the list receives a copy. Note: it is a Terraform list, not a comma-separated string, so use ["a@x", "b@x"] shape not "a@x, b@x".

There are two adjacent arguments that this example does not use but you might reach for:

subscriber_sns_topic_arns — send the notification to an SNS topic instead of (or in addition to) email. Useful when you want the alert to fan out into Slack via an SNS→Lambda hop.
subscriber_type on the older notification shape has been replaced by the three subscriber_* list arguments; you should not need it on modern provider versions (hashicorp/aws >= 4.0).

Full provider docs: aws_budgets_budget#notification.

Budgets by Service

AWS Budgets illustration.

EC2 Monthly Budget

This budget will track costs for EC2 services only.

resource "aws_budgets_budget" "ec2_monthly_budget" {
  name         = "My EC2 Monthly Budget"
  budget_type  = "COST"
  limit_amount = "400"
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  cost_filter {
    name = "Service"
    values = [
      "Amazon Elastic Compute Cloud - Compute",
    ]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 90
    threshold_type             = "PERCENTAGE"
    notification_type          = "ACTUAL"
    subscriber_email_addresses = ["email@my.team"]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 100
    threshold_type             = "PERCENTAGE"
    notification_type          = "FORECASTED"
    subscriber_email_addresses = ["email@my.team"]
  }
}

aws_budgets_notifications_with_terraform/ec2_monthly_account_budget.tf at 0b8af27933b32c27c7b09c4b633724ed17fe82b9 · clearview/aws_budgets_notifications_with_terraform

S3 Monthly Budget

AWS S3 Monthly Budget.

This budget will track costs for S3 buckets only.

resource "aws_budgets_budget" "s3_monthly_budget" {
  name         = "My S3 Monthly Budget"
  budget_type  = "COST"
  limit_amount = "100"
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  cost_filter {
    name = "Service"
    values = [
      "Amazon Simple Storage Service",
    ]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 90
    threshold_type             = "PERCENTAGE"
    notification_type          = "ACTUAL"
    subscriber_email_addresses = ["email@my.team"]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 110
    threshold_type             = "PERCENTAGE"
    notification_type          = "FORECASTED"
    subscriber_email_addresses = ["email@my.team"]
  }
}

aws_budgets_notifications_with_terraform/s3_monthly_account_budget.tf at 0b8af27933b32c27c7b09c4b633724ed17fe82b9 · clearview/aws_budgets_notifications_with_terraform

AppRunner Monthly Budget

This budget will track costs for AppRunner services only.

resource "aws_budgets_budget" "apprunner_monthly_budget" {
  name         = "My AppRunner Monthly Budget"
  budget_type  = "COST"
  limit_amount = "100"
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  cost_filter {
    name = "Service"
    values = [
      "AWS App Runner",
    ]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 90
    threshold_type             = "PERCENTAGE"
    notification_type          = "ACTUAL"
    subscriber_email_addresses = ["email@my.team"]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 110
    threshold_type             = "PERCENTAGE"
    notification_type          = "FORECASTED"
    subscriber_email_addresses = ["email@my.team"]
  }
}

aws_budgets_notifications_with_terraform/apprunner_monthly_account_budget.tf at 0b8af27933b32c27c7b09c4b633724ed17fe82b9 · clearview/aws_budgets_notifications_with_terraform

Budgets based on tag filtering

Production Tag Monthly Budget

This budget will group multiple types of service based on the tag, in this case, the production tag is used.

production_tag_monthly_budget.tf on GitHub

Closing

This article explained the importance of setting a cloud budget and then demonstrated how to do it programmatically. Not only that, we went a step further and created nice diagrams for each budget so that it is easier for every reader to imagine the setup.

We hope this article finds its way and helps you avoid unnecessary cloud costs.

Nedim Hadzimahmutovic · Remote Since Forever

Closing the Rate-Limit Bypass: A CloudFront + ALB Case Study

The Client Architecture

What We Caught

Why the ALB Was Reachable At All

The Fix Isn't Dropping trust proxy

Defense in Depth — What We Added Alongside

1. CloudFront custom-header validation at the app layer

2. Account-based rate limiting on auth endpoints

What We Shipped Back

The Checklist We Carry Forward

How To Check Your Own Setup

Common AWS WAF rate-limit questions

How do I rate-limit per hostname on AWS WAF?

Can AWS WAF rate-limit on a 1-minute window instead of 5?

How do I rate-limit by URL path segment?

How do I exclude a known good IP from an AWS WAF rate rule?

Replacing a Wildcard CORS Policy: An Express Allowlist Case Study

The Configuration We Found

How We Demonstrated the Problem

The Preflight Was Also Verbose

Why This Happens — The Regex Trap

The Fix We Shipped — An Explicit Allowlist

What We Had To Fix Alongside CORS

The Feature-Branch Conversation

Verifying After the Change

What We Added Alongside CORS — The Layered Story

Five things we now check on every CORS engagement

Cutting the File-Upload Bill: A Cognito + S3 Case Study

What the Frontend Had to Do

The Architecture

The Three Layers of Permission Evaluation

The Terraform — Identity Pool

The Terraform — IAM Role

The Terraform — Bucket and Bucket Policy

Verifying With curl

What We Added Back From the Managed Service

1. File-size limits and content-type validation, properly

2. Anti-abuse for guest uploads

3. Bucket abuse limits

Migration Plan (No Downtime)

What the Client Walked Away With

Two gotchas the tutorials skip

How a CloudFront Custom Error Page Leaked JWTs to S3: A Case Study

The One-Line Proof

Why did CloudFront forward the Authorization header to S3?

Three ways the token leaks

The Terraform Template Problem

The Chain That Made It Critical

We proved the full chain end-to-end

The public bucket

The Fix

What we shipped

How to Check If You're Vulnerable

Check every distribution

Iftar at Apetit: The Sarajevo Team Picks a Menu

"Svejedno."

"Pa kad je iftar."

Time for meza

Why the dinner is worth it

How a JWT Audience Map Saved a CORS Mistake: A Defense-in-Depth Case Study

The CORS Misconfiguration

The Exploit That Failed

The JWT Audience Map

Testing Every Authenticated Path

The Refresh Token Path

What Is Still Vulnerable

The Severity Downgrade

Why Two Independent Controls Worked

Two things this engagement changed for us

From DevOps to DevSecOps: How I Became Clearview's In-House Pen-Tester

The Night the Job Description Changed

What an Engineer Reads at Night When They Cannot Sleep

What Tools Catch and What They Miss

The Role We Wrote Around It

What I Catch on the Pen-Tester Pass

What It Gets the Client

What This Costs Us

The Mindset Shift, for Anyone Considering the Move

An Event-Driven Architecture Case Study: Lessons from a Real-World Application

The Fix Isn't Dropping `trust proxy`

Verifying With `curl`

Why did CloudFront forward the `Authorization` header to S3?