The team had two constraints. "Do not block the rest of the work" and "do not break production." On a service that deploys every day, with a long backlog of feature work that the product team had committed to, an upgrade that needed the whole team to stop and migrate together would have been the wrong shape. We had to find the order that let the migration happen alongside the feature work, with the test suite green at every commit and production unaware that anything had changed.

The order we ended up with: a staging branch that absorbed the breaking changes, the test-harness migration ahead of the runtime, the Docker image pinned in parallel, and the eight or so smaller follow-ups that landed after the main upgrade so nobody had to read a five-thousand-line PR.

Two terms before we go further

• TypeORM is a TypeScript ORM for SQL databases. It maps decorated TypeScript classes to tables and gives you a query builder, an entity manager, and a migration runner. The 0.x to 1.0 upgrade is the kind of release that touches every file in the codebase that talks to the database.
• Test harness, in this post, is the infrastructure around the test suite: the factories that build entities, the per-test database setup and teardown, the mocked services, the rate-limit mocks, the fixtures. None of it ships to production. All of it has to be green for CI to pass.

Four places the upgrade broke

The breaking surface on the upgrade landed in four places.

Factory APIs. TypeORM's setSeederFactory shape changed. Every factory we had (about thirty of them, one per major entity) needed a new signature.

Repository methods. A handful of methods that had a "first one wins" behaviour in 0.x became "explicit single or throw" in 1.0. findOne() without an options argument is no longer valid; you have to pass findOne({ where: { id } }) or use findOneBy({ id }). This was a global codemod.

Entity manager transactions. The transaction API tightened. Some callsites that had been relying on implicit transaction propagation needed explicit manager.withRepository(...) calls.

Decorator metadata. @PrimaryGeneratedColumn got stricter about its options. A handful of entities with custom configurations needed a small rewrite.

None of those were hard individually. All four at once, on a codebase with hundreds of entities, was the upgrade.

The order we ran it in

Each step below has its own commit (or branch) and was reviewable on its own.

One. Pin the new Docker image and Node version on a parallel branch.

TypeORM 1.0 requires Node 22. Our production image was on Node 20. Before any code changed, the new image had to be built and tested. We landed Node 22.17.1 + production stage yarn setup for typeorm@1.0.0 as the first commit on the upgrade branch:

# Dockerfile (shortened)
FROM node:22.17.1-bookworm-slim AS builder
WORKDIR /app
COPY package.json yarn.lock .yarnrc.yml ./
COPY .yarn .yarn
RUN yarn install --immutable

FROM node:22.17.1-bookworm-slim AS production
WORKDIR /app
COPY --from=builder /app/node_modules ./node_modules
COPY . .
RUN yarn install --immutable --production
CMD ["node", "dist/server.js"]

The build target uses the production-stage yarn setup because the new TypeORM brings transitive dependencies that the old --immutable flag was tolerant of and the new one is not. We ran the new image against the old code for a week in staging to make sure the runtime change itself was inert.

Two. Migrate the test harness before the runtime.

This is the step that decided whether the upgrade would block the team or not. The test harness is what runs in CI on every PR. If the test harness is half-migrated, nobody can ship anything. So the test harness goes first, in its own PR, with the old TypeORM still in production:

// database/factories/address.factory.ts (before)
import { define } from "typeorm-seeding";
import { Address } from "../../modules/address/address.entity";

define(Address, (faker) => {
  const address = new Address();
  address.street = faker.location.street();
  address.city = faker.location.city();
  return address;
});

// database/factories/address.factory.ts (after)
import { setSeederFactory } from "typeorm-extension";
import { Address } from "../../modules/address/address.entity";

export default setSeederFactory(Address, (faker) => {
  const address = new Address();
  address.street = faker.location.street();
  address.city = faker.location.city();
  return address;
});

About thirty factories, all the same shape, all migrated in one commit. We landed the test harness work as a series of "complete TypeORM 1.0 test-harness migration" commits. Runtime was still on 0.x, test setup was ready for 1.0. The two coexisted because the factories are dev-time only.

Three. Bump the runtime on a staging branch.

The actual package.json change. Roughly fifteen lines changed in the manifest, a hundred or so callsites changed across the source for the findOne → findOneBy codemod and the transaction-API tightening. We landed it on a staging branch that lived for about a week.

{
  "dependencies": {
    "typeorm": "^1.0.0",
    // ... transitive updates for things that wanted Node 22
  }
}

During that week, every other PR rebased onto the staging branch as it was merged. Conflicts happened (entity decorators and query builders both got touched), but they stayed bounded. We had a Slack channel for "my PR is breaking on the staging rebase, who else has touched this file" and it never had more than one or two messages a day.

Four. Fix the small tests that quietly broke.

Once the runtime was on 1.0, a small set of tests broke for reasons the code diff itself did not show. The framework was behaving slightly differently now, and the tests were catching it. A rate-limit mock had assumed the old library called it at one specific moment; the new library called it a moment later. A recaptcha mock had relied on a method name that had been renamed. Some test suites we had not touched in a long time were quietly assuming old data-object shapes and stopped compiling once the underlying types changed.

We landed these as a series of small commits with names like "test: align auth assertions and complete rate-limit mock" and "test: fix constructor drift in the compile-failing test suites." Each one was a few hours of work and a small PR. Nothing dramatic.

Five. Ship the security fixes we had queued up.

This one was a bonus. The TypeORM upgrade gave us a single PR window where every service was already being touched. We pulled in a handful of unrelated security findings (bugs where an attacker could swap in another user's ID, and write endpoints that let one user modify another user's data) and shipped them in the same staging branch. "While we are in here" is a real shipping pattern; we used it once, deliberately, on the boundary the upgrade had opened anyway.

Two conflicts we planned for and didn't need

Two conflicts I want to call out because they are the ones I was expecting and did not get.

Production rollback. We had a plan. The plan was "deploy the new image, watch the error rate, roll back if anything spikes." We did not need it. The week of staging burn-in caught everything that would have spiked.

Migration runner. TypeORM 1.0 has a different migrations API than 0.x. We were prepared to rewrite our migration files. We did not need to. The existing migrations are run-once historical records that the new runner reads cleanly without changes. The new migrations going forward use the new API, but the catalogue of historical migrations is untouched.

What we'd do again on the next one

1. Test harness first, runtime second. If the harness is broken, every other PR is blocked. If the runtime is broken, only the upgrade PR is blocked.
2. Pin the new base image on a parallel branch. Node version upgrades that come bundled with the main one should be tested as their own change before the code change lands on top.
3. Codemod the global API changes in one commit. findOne() → findOneBy() is hundreds of callsites. One mechanical commit beats fifty thoughtful ones.
4. Keep the staging branch alive for a week. Let other PRs rebase onto it. The conflicts are real but they are bounded if the staging branch is fresh.
5. Pull in the "while we are in here" fixes deliberately. Big PRs are scary, but a major version upgrade is the one PR window where every file gets touched anyway. The marginal cost of an extra security fix is much lower in that window than in any other.

Sixteen distributions shared the same copy-pasted Terraform block.

CloudFront is AWS's content delivery network (CDN). A JSON Web Token (JWT) is the signed session string the API gives a user after login. curl is the command-line tool we use to send HTTP requests by hand. Three terms are enough to follow the finding.

CloudFront was handling TLS at the edge, HTTP/2 and HTTP/3 connections, DDoS protection, and branded error pages from a separate S3 bucket. The staging setup had been running for weeks. The distributions were stable, the error pages rendered correctly, and the TLS grades looked good.

The One-Line Proof

curl https://your-api.example.com/custom_error_pages/502.html \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.YOUR_ACTUAL_TOKEN"

If your response looks like this, you're vulnerable:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>InvalidArgument</Code>
  <Message>Unsupported Authorization Type</Message>
  <ArgumentName>Authorization</ArgumentName>
  <ArgumentValue>Bearer eyJhbGciOiJIUzI1NiJ9.YOUR_ACTUAL_TOKEN</ArgumentValue>
</Error>

Your full JWT (user ID, email, role, expiry, everything in the payload) reflected back in an S3 XML error response. The path exists on practically every CloudFront distribution with custom error pages. Any GET request with an Authorization header triggers it, no authentication and no origin failure required.

S3 reflects any Authorization scheme, not just Bearer. Basic auth credentials (Basic dXNlcjpwYXNzd29yZA==), API tokens (Token API_TOKEN_VALUE), anything non-AWS gets dumped into the XML. If your API uses any of these, the same leak applies.

Cookies, X-API-Key, and custom headers are forwarded to S3 too but not reflected in the XML. S3 only complains about the one header it tries to parse.

Why did CloudFront forward the Authorization header to S3?

A CloudFront distribution sits between users and one or more origins (the actual servers that hold your content). Each origin gets a behavior that tells CloudFront which requests to route there and what to forward along.

In this pattern, there are two origins behind one distribution:

The API origin is your application: an Application Load Balancer (ALB) in front of Elastic Container Service (ECS) containers, a Lambda function, or an EC2 instance. It needs the viewer's Authorization header to authenticate requests, Content-Type to parse bodies, and cookies for sessions. Forwarding everything to this origin is correct.

The error page origin is an S3 bucket containing three static HTML files: 502.html, 503.html, and 404.html. It serves the same branded "We'll be right back" page to every user regardless of who they are.

It needs nothing from the viewer: not their identity, not their cookies, not their query strings. It just needs CloudFront to ask for a file by path, and it returns the HTML.

The problem is that both origins were given the same origin request policy: the one designed for the API. So when CloudFront routes a request to the S3 error page bucket, it sends along everything the viewer included, including Authorization: Bearer <JWT>.

S3 is not your API. It doesn't understand Bearer tokens. It tries to interpret the Authorization header as AWS Signature Version 4, fails, and returns an XML error that includes the header value it couldn't parse. Your JWT is now in the response body.

The setup looks like this in Terraform (CloudFormation and the console follow the same pattern):

# Origin 1: Your API (ALB, ECS, Lambda, etc.)
origin {
  domain_name = aws_lb.api.dns_name
  origin_id   = "api"
}

# Origin 2: S3 bucket with static error pages
origin {
  domain_name = aws_s3_bucket.error_pages.website_endpoint
  origin_id   = "error-pages"
}

# When the API returns 502, serve the S3 error page instead
custom_error_response {
  error_code         = 502
  response_page_path = "/custom_error_pages/502.html"
}

# Behavior for the API (default): forwards all viewer headers
default_cache_behavior {
  target_origin_id         = "api"
  origin_request_policy_id = aws_cloudfront_origin_request_policy.AllViewerExceptHostHeader.id
}

# Behavior for the error pages: ALSO forwards all viewer headers
ordered_cache_behavior {
  path_pattern             = "/custom_error_pages/*"
  target_origin_id         = "error-pages"
  origin_request_policy_id = aws_cloudfront_origin_request_policy.AllViewerExceptHostHeader.id
  # ↑ THIS IS THE PROBLEM
}

The AllViewerExceptHostHeader policy tells CloudFront: "forward every header the viewer sent (Authorization, Cookie, X-Custom-Whatever) to the origin, except Host." That's correct for the API behavior, where your backend needs those headers. It's catastrophically wrong for the error page behavior, where the origin is an S3 bucket serving static HTML.

Three ways the token leaks

The error page path isn't linked anywhere. No browser navigates to /custom_error_pages/502.html during normal use. Three practical routes still reach it.

The path is trivially discoverable by scanners. It's in CloudFront's own documentation examples, and tools like ffuf, feroxbuster, and nuclei include /error/, /custom_error_pages/, and /502.html in their default wordlists. A scanner that hits the path and sees <Code>InvalidArgument</Code> with <ArgumentName>Authorization</ArgumentName> in the XML response can identify the misconfiguration from the shape alone. Our pentest tooling found this one that way.

A production outage can leak the token without any scanner. When the origin returns 502/503, CloudFront intercepts the error and fetches the S3 error page with the original request's Authorization header attached. The user didn't navigate to the error page; CloudFront sent them there. Their token leaks through a normal deployment, container restart, or scaling event, visible in the response body to any browser extension, corporate proxy, or compromised CDN edge watching that window.

Most of the distributions we audited had the plain-HTTP setting. The API behavior used redirect-to-https; the error-page behavior was a separate block with separate settings.

The Terraform Template Problem

We found this in a client's infrastructure codebase. We audited every CloudFront distribution across their staging and production accounts. The same ordered_cache_behavior block for /custom_error_pages/* showed up on distribution after distribution, all pointing to the same S3 error pages bucket, all using AllViewerExceptHostHeader.

The block was written once for the API distribution, where it was correct. Then it was copied to every other distribution (SPAs, admin dashboards, static sites, upload CDNs) because it "worked" and nobody questioned whether a static error page bucket needed to receive the viewer's Authorization header.

Terraform's copy-paste turned one mistake into a pattern across the whole account.

A handful of the affected distributions served authenticated traffic (APIs and legacy apps). Those were fully exploitable: every authenticated request that hit a 502 leaked the caller's JWT. The rest served static frontends that don't normally receive Authorization headers, but the direct-access path still worked on all of them.

The Chain That Made It Critical

The JWT leak alone is a HIGH: you can capture a token and replay it for the remaining access-token lifetime. But during this engagement, we found it chained with something else that pushed it to CRITICAL.

The API signs session tokens with a JWT secret. On staging, that secret was hardcoded as a trivially guessable value in the ECS task definition's plaintext environment block. Not in Secrets Manager, not pulled from SSM Parameter Store, just a short, dictionary-word string sitting in Terraform.

We confirmed the secret remotely without any internal access. The API returns different error messages depending on whether a token's signature verifies:

# Token signed with the guessable secret (correct):
curl -X POST https://api.staging.example.com/api/session/token \
  -H "Cookie: session=<forged-token-signed-with-guessable-secret>"
# → "Session has been revoked" (signature PASSED, DB hash lookup failed)

# Token signed with wrong secret:
curl -X POST https://api.staging.example.com/api/session/token \
  -H "Cookie: session=<token-with-wrong-signature>"
# → "Invalid session" (signature FAILED)

That error difference acts as an oracle. "Session has been revoked" means the JWT signature verified successfully; the server moved past cryptographic verification to the database lookup phase. "Invalid session" means the signature check failed. The two code paths are distinguishable remotely.

We proved the full chain end-to-end

We registered a test account and signed in. We captured the session token, leaked it through the error page, and validated it offline. Then we replayed it and had the account. All with curl.

Step 1: Sign in. We registered a test account and signed in. The server set an httpOnly session cookie. It was a long-lived JWT signed with the guessable secret.

Step 2: Leak the token. We sent a GET request to the error page path. The session token was in the Authorization header. The S3 XML error returned it verbatim. Byte for byte, it matched the original.

Step 3: Validate offline. We computed the HMAC-SHA256 signature using the guessable secret. It matched the token's signature exactly. No server interaction needed. The attacker now knows the token is real, who it belongs to, and when it expires.

Step 4: Replay. We sent the leaked session token to the token endpoint. The server verified the JWT signature (passed) and looked up the token hash (found a real active session). It minted a fresh access token.

Step 5: Account takeover. We used the fresh access token to call the user-profile endpoint. The server returned the full profile: ID, email, name, role, all PII fields. Full account access with no password. All from a token that was never meant to leave the httpOnly cookie jar.

We ran the entire chain with curl against a live staging environment. We didn't need source code access to exploit it. We only needed it to discover the hardcoded secret in Terraform. An external attacker reaches the same outcome by guessing the secret or by intercepting a token through the error page leak.

The public bucket

The bucket is also enumerable:

curl https://your-error-pages-bucket.s3.amazonaws.com/

Returns a full ListBucketResult XML with every file, size, ETag, and last-modified timestamp, no authentication required.

It's public because the error page origin uses an S3 website endpoint, which forces public read access. OAC (Origin Access Control) only works with the REST API endpoint. The pages themselves are benign HTML, but the bucket is one IAM policy drift away from attacker-writable: a phishing form in a 502 page would be served by your own domain, from your own CloudFront distribution, with your own TLS certificate.

The Fix

The error page behavior serves static HTML from S3. It doesn't need the viewer's auth token, cookies, query strings, or any other header.

When you omit origin_request_policy_id entirely, CloudFront sends only the minimum required headers to the origin. That is enough for an S3 website endpoint to fetch a file by path, and there is nothing left for S3 to reflect back.

The simplest fix is to delete the origin_request_policy_id line from every error page behavior. It shouldn't have been there in the first place.

But if you want to be explicit about the intent (and make it harder for someone to re-add a permissive policy later thinking it was accidentally removed), define a policy that forwards nothing:

resource "aws_cloudfront_origin_request_policy" "ErrorPagesNoAuth" {
  name    = "ErrorPages-NoAuth"
  comment = "Error pages are static HTML, no viewer headers needed"

  headers_config {
    header_behavior = "none"
  }
  cookies_config {
    cookie_behavior = "none"
  }
  query_strings_config {
    query_string_behavior = "none"
  }
}

Then in every ordered_cache_behavior for /custom_error_pages/*:

ordered_cache_behavior {
  path_pattern             = "/custom_error_pages/*"
  target_origin_id         = "error-pages"
  origin_request_policy_id = aws_cloudfront_origin_request_policy.ErrorPagesNoAuth.id
  viewer_protocol_policy   = "redirect-to-https"
  # ... rest unchanged
}

Verification:

# Before fix:
curl https://your-api.com/custom_error_pages/502.html \
  -H "Authorization: Bearer TEST"
# → <ArgumentValue>Bearer TEST</ArgumentValue>  ← LEAKED

# After fix:
curl https://your-api.com/custom_error_pages/502.html \
  -H "Authorization: Bearer TEST"
# → <!DOCTYPE html><html>...(your 502 page HTML)...  ← SAFE

For defense in depth:

• Switch the S3 origin from a public website endpoint to a private bucket with OAC. This closes the public listing vulnerability and eliminates the unencrypted HTTP origin protocol.
• Set error_caching_min_ttl = 0 so error responses aren't cached at all.
• Set viewer_protocol_policy = "redirect-to-https" on the error page behavior so the path can't be accessed over plain HTTP.

Forwarding headers to error pages is valid when the origin needs the viewer's identity: a Lambda@Edge function rendering personalized error pages, a Cognito-gated bucket, or a debugging service logging which user hit the error. In those cases the origin is an application that can safely consume auth headers. If your error page origin is S3, the answer is always header_behavior = "none".

What we shipped

Sixteen distributions closed. The JWT reflection path is gone across the client's account, and HTTP-to-HTTPS is enforced on every error-page behavior. The one residual risk (the public S3 website endpoint) is on the follow-up list.

How to Check If You're Vulnerable

Step 1: Find your error page path. Look for custom_error_response blocks in your CloudFront distribution. The response_page_path tells you where the error pages live.

Step 2: Send a request with an auth header.

curl -s https://your-domain.com/custom_error_pages/502.html \
  -H "Authorization: Bearer CHECK_THIS_TOKEN"

Step 3: Check the response. If you see <ArgumentValue>Bearer CHECK_THIS_TOKEN</ArgumentValue> in the XML, you're vulnerable. If you see your HTML error page, you're fine. A 403 means the path doesn't exist or the behavior doesn't allow GET, which is also fine.

Step 4: Check all your distributions. If you use the same Terraform module or copy-paste pattern across distributions, check every one. In one engagement we found the majority of distributions sharing the same misconfiguration. The one you check might be fine while a dozen others are leaking.

Check every distribution

Nobody asked whether a static HTML bucket needed the same headers as an authenticated API. If you're running CloudFront with custom S3 error pages, spend sixty seconds running the curl command above against each distribution.

A note on responsible testing: every test account created during this engagement was registered with a non-existent @example.com address, used only for curl-based proof-of-concept, and cleaned up after the findings were documented. No real user sessions were intercepted or replayed. The full chain was proven end-to-end using our own credentials against a staging environment with explicit authorization from the infrastructure owner.

Probably not often because unsubscription is now something that you have to legally think of, especially if you send marketing emails.

So how do you implement an unsubscribe link that is highly accessible, doesn't require any login, secure and unspoofable, and respect the RFC 8058 so that most email clients can automatically unsubscribe the user using your link? An email that respect your user so that they won't tap on that "Mark as spam" instead; and an email that won't trigger accidental unsubscribe because the email client crawls through every link to check for viruses.

JWT

JSON Web Token. A signed string with three parts (header, payload, signature) separated by dots.

HMAC · HS256

A keyed hash. A shared secret plus the payload produces a signature that only holders of the secret can create.

RFC 8058

The "one-click unsubscribe" standard that Gmail and Apple Mail support via the `List-Unsubscribe` and `List-Unsubscribe-Post` headers.

kid

A JWT header field naming which signing key was used, so a verifier can support key rotation — try current, then previous.

Possible Approaches

Random Token In The Database

The simplest approach would be to create a table that contains a token associated with an email that the user can use to unsubscribe.

CREATE TABLE unsubscribe_tokens (
  token uuid PRIMARY KEY,
  user_id uuid NOT NULL,
  email varchar NOT NULL,
  created_at timestamp NOT NULL
)

Generate a UUID per email, write the row, put the token in the URL. Endpoint gets hit, looks up the token, finds the user.

The problem with this approach is that you have to create a lot of them, and if you want to keep track of where the user unsubscribed from, you will create one token per email. For email that are sent 4 times a month for 100.000 users, that's 400.000 inserts on this table. By the end of the year, you will have 4.8M rows.

Signed Token In The Email

The alternative is to use the email itself as the storage of the token. A signed token that can't be spoofed and by default "distributed".

JWT is a good candidates for this, because we can sign it and then encode it as a link on the email.

The minimum fields that we need is just the email. That's it — of course you can add additional fields for tracking and other verifications. But you just need an email associated with the token and sign it and you're good to go.

We sign it with a shared HMAC secret — or if you're distributed and do key-pair verification, you can always do it too:

export function signEmailSubscriptionToken(
  payload: EmailSubscriptionTokenPayload,
): string {
  return jwt.sign(payload, EMAIL_SUBSCRIPTION_SECRET, {
    algorithm: 'HS256',
  });
}

And the URL that goes into the email footer looks like:

https://example.com/manage-email-subscriptions?token=<jwt-here>

Ideally, this will open a page that will allow the user to manage their subscriptions, and only execute the unsubcription via user interactions and not on page load. Otherwise, crawler might accidentally trigger unsubscriptions when this URL is accessed.

So you will need another endpoint, that accepts the token as well, but called with POST method.

Is This Safe?

There are three real threats to an unsubscribe URL:

Someone forges a URL to unsubscribe a stranger. The signature stops this. You cannot produce a valid token without our secret. Try to change the sub claim on a token you already have, and the JWT verification fails immediately. There is no way to forge a token that verifies without the secret. That is the whole point of signing.

A crawler enumerates URLs and unsubscribes everyone. The token is a random-looking base64 string that is different for every user. You cannot walk from one URL to the next. There is no ?userId=1234 you can increment to find the next one.

Someone steals a token from a leaked email and tries to escalate — use it to change the user's password, or log in as them. The thing that stops it is not the token — it is the endpoint.

The unsubscribe endpoint only knows how to do one thing, and that thing is "change this user's email subscription preferences." It does not accept requests to change the password. It does not accept requests to issue a session cookie. It does not accept requests to update the email address. So, even if the token leaks — say the email got forwarded, or someone got access to an old inbox — the worst outcome is that a stranger unsubscribes them from an email.

And the only line of code you need to write to verify the token is just this:

payload = jwt.verify(token, EMAIL_SUBSCRIPTION_SECRET);

RFC 8058

Gmail and Apple Mail have supported RFC 8058 for a while — the "one-click unsubscribe" standard. You set two headers on your outgoing email:

List-Unsubscribe: <https://example.com/one-click/unsubscribe?token=abc123>
List-Unsubscribe-Post: List-Unsubscribe=One-Click

The URL above can be the POST endpoint that you also use for the actual unsubscription instead of the manage subscription page.

And Gmail renders a small "Unsubscribe" button next to the sender name. When the user taps it, Gmail POSTs to your URL on the user's behalf. The user never opens the email. They just tap once and the mail client confirms "unsubscribed."

A few things worth knowing

Be careful with the expiration of the token, some jurisdiction might require you to have specific expiration date. The easiest would just to not set any expiration date. I know it feels natural to set expiresIn: '30d' on the sign call.

Keep the payload small. URLs in emails are already long. Do not stuff extra fields into the token unless you have to.

Do log the unsubscribe. Not for legal reasons (although also for those) — for debugging. If a user complains that they were unsubscribed and did not remember doing it, having a log of when the token was verified and from what IP is very useful.

Rotate your secret. The signing key is a secret. If it leaks, every unsubscribe URL you have ever sent is compromisable. A kid field in the JWT header lets you support multiple keys, and the verifier can try current, then previous.

Feel free to share if you have a different approach to unsubscribing users from emails without asking them to log in. This pattern has worked for us but there is definitely more than one way to do it.

We split the file into four sub-services behind a stable facade, landed the whole thing in one PR without changing a test, and wrote the procedure up as a reusable Claude skill for the next time.

Two terms before we go further

• Facade, in this post, is a class that exposes the same public methods as the old service did, but delegates each method to the right one of the new sub-services. The old import path keeps working; the internal shape moves.
• Sub-service is one of the new files the mega-service is split into. Each sub-service owns one slice of responsibility (querying, mutating, membership-state, stats) and is named after that slice.

The split we shipped

src/modules/user/
├── user.service.ts                   #  facade — was 1,307 lines, now ~120
├── user-query.service.ts             #  349 lines — read paths
├── user-mutation.service.ts          #  460 lines — write paths
├── user-membership.service.ts        #  288 lines — membership-state changes
├── user-stats.service.ts             #  307 lines — derived counts and aggregates
└── user.service.password-reset.spec.ts

The split ran along responsibility inside the service layer, not up through the controller and model layers. Reading a user and updating a user are different jobs on the same domain object, and they grow at different rates. On this service, the read methods stayed small and stable while the write methods kept picking up business rules.

Query — anything that returned a user (or a list of users) without mutating state. findById, findByEmail, findByCompany, searchByKeyword. The query service does not write.

Mutation — anything that wrote a user row (create, update, updatePreferences, softDelete, restore). The mutation service loads a row, mutates it, saves it.

Membership — methods that flip the user's membership state: upgrade tier, downgrade tier, transfer membership, attach to company, detach from company. The state transitions are tangled enough to justify their own file.

Stats — derived counts and aggregates. "How many users in this company are active," "how many memberships expire this month," "how many users by role." Reading across many rows is a different job from finding a single user.

A few methods did not fit any of the four. Some were doing two things and got split before they moved. Some were orchestration — they touched two or more sub-services in a single flow — and stayed on the facade.

The facade — keep the public API stable

The facade is the file every other module imports. Its surface area is identical to the old service: every method other modules called still exists, with the same signature.

// user.service.ts — the facade, after the split
@Service()
export class UserService {
  constructor(
    private readonly query: UserQueryService,
    private readonly mutation: UserMutationService,
    private readonly membership: UserMembershipService,
    private readonly stats: UserStatsService,
  ) {}

  // ─── Read methods delegate to query ─────────────────────────────────
  findById = this.query.findById.bind(this.query);
  findByEmail = this.query.findByEmail.bind(this.query);
  findByCompany = this.query.findByCompany.bind(this.query);
  searchByKeyword = this.query.searchByKeyword.bind(this.query);

  // ─── Write methods delegate to mutation ─────────────────────────────
  create = this.mutation.create.bind(this.mutation);
  update = this.mutation.update.bind(this.mutation);
  updatePreferences = this.mutation.updatePreferences.bind(this.mutation);
  softDelete = this.mutation.softDelete.bind(this.mutation);
  restore = this.mutation.restore.bind(this.mutation);

  // ─── Membership ─────────────────────────────────────────────────────
  upgradeMembership = this.membership.upgrade.bind(this.membership);
  downgradeMembership = this.membership.downgrade.bind(this.membership);
  attachMemberToCompany = this.membership.attachToCompany.bind(this.membership);

  // ─── Stats ──────────────────────────────────────────────────────────
  countActiveByCompany = this.stats.countActiveByCompany.bind(this.stats);
  countExpiringMemberships = this.stats.countExpiringMemberships.bind(this.stats);

  // ─── Orchestration methods stay here ────────────────────────────────
  // Methods that genuinely coordinate across two or more sub-services.
  async passwordResetFlow(email: string, newPassword: string) {
    const user = await this.query.findByEmail(email);
    if (!user) return { success: false };

    await this.mutation.updatePassword(user.id, newPassword);
    await this.stats.recordSecurityEvent(user.id, "password-reset");
    return { success: true };
  }
}

The bind-and-assign shape looks unusual on first read, but each method on the facade is the same function object as the one on the sub-service — no wrapper indirection, no extra logging. userService.findById(42) runs with the same overhead it always had.

Orchestration methods stay on the facade because they touch several sub-services in one flow. passwordResetFlow above hits the query, mutation, and stats services in sequence, so it belongs to none of them alone.

How the split landed in one PR

The PR added around sixteen hundred lines and deleted around twelve hundred. Most of it was file moves. The tests did not need to change: because the facade's public API is byte-identical to the old service, every test that imported UserService kept passing untouched. That is the safety net the whole refactor rides on.

Three disciplines made the PR reviewable.

One sub-service per commit. Four code commits, one per sub-service, plus a fifth that thins the facade. The reviewer reads one sub-service at a time.

Method-by-method migration. Each method moved with its tests. If a test in the suite was targeting a single method, that test stayed near the method, in the same commit as the move.

No behavioural changes. Every method moved is byte-identical to what was in the old service. Behavioural refactors — N+1 fixes, added logging, tighter authorization — go in follow-up PRs after the split lands.

The reusable skill — service-facade-refactor

The skill file lives at .claude/skills/service-facade-refactor/SKILL.md in this codebase and in our internal skill library. The next mega-service does not need to re-derive any of this.

A Claude skill is a small Markdown file documenting a reusable procedure in enough detail that an LLM-assisted contributor can apply it without having lived through the original work.

The skill in full:

# Service Facade Refactor

A pattern for splitting a single mega-service file into multiple
responsibility-tagged sub-services behind a stable public facade,
without breaking any caller.

## When to use this skill

A single service file has crossed roughly 800 lines, the team is
avoiding it on PRs, and the file mixes more than one of:
read paths, write paths, state-machine transitions, derived stats.

If the file is large but coherent (one responsibility, just verbose),
this is not the right skill — use a method-extraction refactor instead.

## Inputs

- The mega-service file (path).
- The full test suite (must be green before starting).
- One hour of uninterrupted reading time, before any code moves.

## Steps

1. **Tag every method on the existing service.** Read top-to-bottom.
   Each method gets exactly one tag:
   - **Read** — returns the entity, no writes.
   - **Write** — mutates the entity, may read first.
   - **State machine** — tier upgrades, status transitions, lifecycle.
   - **Stats** — counts, aggregates, derived values across many entities.
   - **Orchestration** — coordinates across two or more of the above.

   If a method does not fit one tag, it is doing two things; split
   the method *before* you split the service.

2. **Each non-orchestration tag becomes a sub-service.**
   File naming: `<domain>-<tag>.service.ts`. Example: `user-query.service.ts`.

3. **Move methods one tag at a time, in their own commit, with
   their tests.** The test file moves with the method, or, if the
   tests are integration-style, stay in place and reference the
   sub-service through the facade.

4. **Rewrite the original service file as a facade.**
   - For each method on a sub-service, `bind`-and-assign on the facade:
     `findById = this.query.findById.bind(this.query);`
   - For orchestration methods (touch two or more sub-services), keep
     the implementation on the facade.

5. **Run the full test suite. It should pass with no test changes.**
   The facade's public API is byte-identical to the original service.

6. **Land as a single PR with one commit per sub-service plus one
   for the facade.** Behavioural changes (N+1 fixes, new logging,
   tightened authz) are *follow-up PRs*, not part of the split.

## Output shape

- One facade file, exposing the same methods the original did.
- Four (or fewer) sub-service files, each named by its tag.
- Zero test changes.
- A single PR, multi-commit, readable one sub-service at a time.

## What this is not

- Not a layered split (controller / service / model). The split is
  by responsibility within the service layer, not by layer.
- Not a behavioural change. Save the perf and authz fixes for after.
- Not appropriate for files under ~500 lines. The overhead of the
  split is worse than the readability win at that size.

Step 1 is where the split gets decided. The tagging walks top-to-bottom on five yes/no questions, and the first "yes" wins.

The skill works across codebases because it does not name a specific domain. Any TypeScript codebase with service classes and a mega-service problem can apply it the same way.

The judgement calls in the skill took longer to write than the steps. The "if the file is large but coherent, this is not the right skill" line at the top and the "what this is not" section at the bottom are what keep the skill from getting misapplied. A skill that only describes the happy path gets used wrong the first time somebody reaches for it.

The next mega-service on this codebase — finance.service.ts, around eleven hundred lines — is queued up for the same treatment.

The week before Ramadan, as it is the company's custom, Amina booked us a table for iftar at Apetit, the food spot everyone in Sarajevo recommends. It sits on Vilsonovo šetalište, the riverside promenade locals call Vils, five minutes from most of our apartments.

svejedno

whatever, fine with me

može

works for me

ekipa

the crew, our people

fakat

honestly, for real

meni

for me / put me down for

pa kad je iftar

well, whenever iftar is (the punchline)

Here is the thread that got us there, translated and lightly compressed:

A few things in that thread stayed with me.

"Svejedno."

Half the thread is people saying svejedno. In this team, that word means trust. Amina booked the restaurant, picked the menu options, chased the head-count. Nobody in the thread felt the need to second-guess any of it. When one of us takes on the coordination work, the rest of us get out of the way. And we take any excuse to sit at the same table — Clearview picks up the tab, and a distributed team never says no to a paid-for meal.

The same pattern shows up on client work. When someone picks up a task, the rest of the team stays out of the way. We ask when we have something to add, and we say svejedno — or the English equivalent, "sounds good, go for it" — when we don't.

Sometimes the person out front is the senior engineer on a client sprint. This time it was Amina, the one who knows every food spot in Sarajevo.

"Pa kad je iftar."

Amina's reply to my what-time question assumes that everyone at the table already knows when iftar is. A non-Bosnian reader would probably miss it.

Iftar is the meal Muslims eat to break the daily fast during the month of Ramadan. It happens at sunset, which means the exact time slides ten to fifteen minutes later every week as the days get longer. In Sarajevo in mid-March, sunset was somewhere around 6:15 pm. If you knew the calendar, you knew the time. If you did not, you asked, and Amina answered. Because she is the boss.

The cultural detail did not have to be spelled out in the thread. Everyone on the invite already got it.

Time for meza

The corporate menu did land in the end. The five of us who made it — Amina, Aid, Adnan, Ahmed, and me — ate at Apetit as the sun set behind the hills over the Miljacka. Redzy joined for part of it. There were the standard meza, the Bosnian word for the little starter plates other cuisines call mezze or mezes. In Bosnia, meza is the part of the meal where the real talking happens, where a table settles in and stops being in a hurry. Then came the mains, and then someone ordered Bosanska kafa and the conversation wandered, as it always does, into a mix of shop talk and family news.

Nobody talked about "team building."

We had not all sat at the same table since the last one, and we would probably not sit at the same table for another couple of months. That is the rhythm of a distributed team — long stretches of async work broken by a shared meal every few months.

Why the dinner is worth it

For a distributed team, in-person meals matter more than they look. The grand summit in Bali counts — Clearview gathered in Ubud in January — but the small local dinner counts too, and it happens more often. The one where Amina's message goes out, five people RSVP within the day, and the reservation goes under her name because she is the one who called the restaurant. Typical Amina. She knows the best places to eat in Sarajevo, and the rest of us are smart enough to show up where she tells us to.

Every time we do one of these, the async work that follows for the next few weeks is smoother. Threaded replies come with more benefit of the doubt, less second-guessing, nobody a stranger. If you have never worked distributed, that is the reason to spend real money on getting people in a room. It makes the rest of the year work better.

We are lucky, in Sarajevo, that most of the Clearviewers here can be in the same city inside of an hour. Having enough of us in one city to fill a table on short notice is a real gift. When Ahmed messages that he doesn't like dolma and sarma, or Amina jokes that we should all know when iftar is, that only feels normal because we have eaten together before.

Nedim  ·  Sarajevo  ·  March 2026

Two terms before we go further:

• CORS — the browser's rule about which other websites are allowed to send authenticated requests to your API. Mis-set, it lets any site act on a logged-in user's behalf.
• JWT — JSON Web Token. The short signed string the API hands a user after they log in. Every subsequent request carries it as proof of who the user is. The token has an audience field that names which application the token was minted for, and the API can refuse a token presented to the wrong audience.

Two independent controls — the stricter one held while the weaker one was broken.

The CORS Misconfiguration

The Express CORS middleware allowed any subdomain through a regex:

// cors.loader.ts
const pattern = /^https:\/\/([a-z0-9-]+\.)?client\.example\.org$/;

This matches any subdomain, including ones that do not exist yet, ones that could be registered by attackers, and ones that could be taken over via a dangling DNS record. We confirmed the breadth with a small matrix:

# Evil subdomain — accepted
curl -sI "$API/api/config" \
  -H "Origin: https://evil.client.example.org"
# access-control-allow-origin: https://evil.client.example.org
# access-control-allow-credentials: true

# Punycode subdomain — accepted
curl -sI "$API/api/config" \
  -H "Origin: https://xn--evil.client.example.org"
# access-control-allow-origin: https://xn--evil.client.example.org

# Numeric prefix — accepted
curl -sI "$API/api/config" \
  -H "Origin: https://123evil.client.example.org"
# access-control-allow-origin: https://123evil.client.example.org

CORS with credentials: true, wildcard subdomains, and DELETE listed in the preflight Access-Control-Allow-Methods. Every box on the checklist of "trivially exploitable CORS misconfiguration."

We were about to write HIGH — any subdomain can steal authenticated session data. We tried to actually exploit it first.

The Exploit That Failed

For the exploit to work in a real browser, the attacker needs to make authenticated requests from their evil subdomain. With JWT auth (Authorization header, not cookies), the attacker first has to obtain a token from the evil origin — either by signing in there, or by stealing a token from the legitimate frontend's localStorage and replaying it.

We tried signing in from the evil origin:

curl -s -X POST "$API/api/auth/signin" \
  -H "Content-Type: application/json" \
  -H "Origin: https://evil.client.example.org" \
  -d '{"email":"test@example.org","password":"Secret123"}'

{ "success": false, "message": "Unknown origin" }

CORS accepted the origin. The API rejected the sign-in. Something independent of CORS was checking.

The JWT Audience Map

Digging into jwt.config.ts surfaced this:

const audienceMap: Record<string, string> = {
  // Production
  'https://app.client.example': 'main-app',
  'https://admin.client.example':         'admin-app',
  'https://partners.client.example':  'partner-portal',
  // Staging
  'https://app.staging.client.example': 'main-app',
  'https://partners.staging.client.example':  'partner-portal',
  'https://admin.staging.client.example':         'admin-app',
  // API-internal calls
  'https://api.staging.client.example': 'api-internal',
};

export function resolveAudience(origin: string | undefined): string | null {
  if (!origin) return 'api-internal';
  return audienceMap[origin] ?? null;  // null = rejected
}

Seven explicit origins. Anything else returns null, which the calling code translates into a 401 Unknown origin. The CORS policy says "any subdomain is fine." The JWT system says "only these seven specific origins get tokens." Two access-control systems with conflicting policies — and the stricter one wins.

Testing Every Authenticated Path

We needed to know whether the audience check covers every authenticated operation or just sign-in:

# 1. Get a token from a legitimate origin.
TOKEN=$(curl -s -X POST "$API/api/auth/signin" \
  -H "Origin: https://partners.staging.client.example" \
  -d '{"email":"test@example.org","password":"Secret123"}' \
  | jq -r '.data.access_token')

# 2. Try the token from the evil origin.
curl -s "$API/api/auth/me" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Origin: https://evil.client.example.org"
# → "Unknown origin"

# 3. Try with no Origin header at all.
curl -s "$API/api/auth/me" \
  -H "Authorization: Bearer $TOKEN"
# → "Audience mismatch"
#   (token has aud=partner-portal, no origin resolves to api-internal, mismatch)

# 4. Try with a different legitimate origin.
curl -s "$API/api/auth/me" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Origin: https://app.staging.client.example"
# → "Audience mismatch"
#   (partner-portal ≠ main-app)

Every path blocked. The audience check fires on every authenticated request, not just sign-in. A token minted for partner-portal only works when sent with the partners.staging.client.example Origin header — any other origin (evil, no-origin, different legitimate origin) returns 401.

The Refresh Token Path

Token refresh has the same check:

curl -s -X POST "$API/api/auth/refresh-token" \
  -H "Origin: https://evil.client.example.org" \
  -d "{\"refresh_token\": \"$REFRESH\"}"
# → "Unknown origin"

The attacker cannot refresh a stolen token from an evil origin either.

What Is Still Vulnerable

The audience check only runs on authenticated endpoints. Public endpoints do not need it, and do not get it:

# Public config — readable from any CORS-accepted origin.
curl -s "$API/api/config" \
  -H "Origin: https://evil.client.example.org"
# 200 OK — feature flags returned

# Public catalog — same shape.
curl -s "$API/api/catalog/" \
  -H "Origin: https://evil.client.example.org"
# 200 OK

Public data is readable from any origin that the CORS policy lets through. The data itself is not sensitive — feature flags and public catalog entries — so the impact is low. The client's product owner confirmed this explicitly.

The real remaining risk is XSS on a legitimate subdomain. If an attacker achieves XSS on, say, partners.staging.client.example, they can read the JWT out of localStorage and use it from script running on that legitimate origin. Neither control fires. The audience check sees a legitimate origin and CORS sees an approved one. The two controls only stop arbitrary origins, not legitimate origins compromised via a different bug.

The Severity Downgrade

We originally drafted the CORS wildcard as HIGH. After the exploitation attempt, we downgraded it to MEDIUM, with the residual risk path documented explicitly:

We still recommended the CORS fix, and it landed in the same sprint. Defense in depth means not relying on one independent control to do the work of two. Without the audience map, the CORS bug would have been trivially exploitable. With it, an attacker first needs script execution on a legitimate origin (XSS on partners.staging.client.example or similar), which is a much higher bar.

Why Two Independent Controls Worked

The CORS policy and the JWT audience map were written at different times by different people for different reasons. CORS was set up early in the project, when the team was thinking about subdomain flexibility. The audience map was added later, when multi-portal authentication came in. Neither was deliberately a "backup" for the other.

The two controls do not need to know about each other. CORS asks whether the browser is allowed to send the request; the audience check asks whether the token is valid for this origin. Different questions, enforced in different places. When one is weaker, the other holds — which is what happened here.

Two things this engagement changed for us

First: try to exploit a finding end to end before assigning severity. The path from "this check is missing" to "I can steal user data" often has unexpected obstacles, and the severity should reflect the real impact, not the impact of the broken control in isolation.

Second: unify the CORS allowlist and the JWT audience list into one source-of-truth file, two readers. When the two lists drift, defense in depth becomes defense in name only. On this engagement, we consolidated both into a single origins.config.ts at boot; the CORS middleware and the audience resolver now read from the same list.

The residual risk (XSS on a legitimate subdomain) went into the report explicitly, because a layered defense that holds against arbitrary origins still needs the reader to know what would open the attack.

Our first onboarding didn't treat the permission with that much respect. We did it the way most apps do, at some point in the flow we called the API, the iOS dialog popped up with no context, and the user had to decide on the spot why an app they installed five minutes ago wants to send them things. Predictably, a lot of them tapped "Don't Allow".

We assumed we could ask again later, once the user had seen enough of the app to change their mind. That assumption was wrong, and it's the reason this post exists. On iOS the native prompt is a one-shot, once the user says no, calling requestPermissionsAsync() again just returns "denied" without showing anything. The only recovery is the user going to Settings, Notifications, Your App and flipping the switch by hand, and very few people who decline ever do that.

So every "Don't Allow" was a permanent opt-out from a core part of the product, which made the decline rate the most expensive metric in the whole onboarding.

Let me show you what we did instead, the screen we put in front of the prompt, the hook that guards the request, and what we're still measuring.

What the iOS docs don't tell you

The native prompt is a one-shot, that part is documented. What isn't documented as prominently is when it becomes a one-shot. It's not the first time you call requestPermissionsAsync(), it's the first time the user sees it. Once they've seen the prompt and declined, you can't show it again from your app code.

You can detect the state with getPermissionsAsync():

import * as Notifications from "expo-notifications";

const { status } = await Notifications.getPermissionsAsync();
// "granted" | "denied" | "undetermined"

"undetermined" is the only state where calling requestPermissionsAsync() will actually show the system prompt. "denied" means the user already saw it and said no, and your only recovery is the Settings deep link. "granted" means you're clear to send notifications.

The whole pattern in this post is built on one rule: never call request while the status is undetermined unless you're confident the user will say yes.

Timing: a lesson from game design

Mobile games solved this years ago. A well-designed game never asks for notification permission on first launch, it waits until you've finished the first level, earned your first reward, or started a building that takes an hour to complete. At that moment the notification has an obvious job, "we'll tell you when it's done", and the player has already felt the value, so the ask answers a question they actually have.

The worst time to ask is before the user has experienced anything, at that point the permission prompt is a cost with no visible benefit and "Don't Allow" is the rational answer. The best time is right after a moment of value, when the notification is clearly in service of something the user just chose.

That's why our screen sits where it does in the onboarding. The user has already entered their name, picked their goals and invested a few minutes in the flow, so the notification ask is framed as the thing that makes those goals happen. Same principle as the game asking after the first level, earn the moment first, then ask.

The screen we show before the prompt

The screen we shipped is one step in the onboarding flow, right after the user has entered their name, picked their goals and invested a few minutes in the flow. It's deliberately simple, a title, a short description and two buttons:

<NotificationExplainerScreen
  title="Stay on track"
  description="We'll send you gentle reminders for your daily habits, lab result updates the moment they're ready, and progress milestones worth celebrating. One nudge at the right time, never spam. You're in control, turn off or customize every notification type anytime in Settings."
  primaryAction={{
    label: "Turn on notifications",
    onPress: requestNotificationPermission,
  }}
  secondaryAction={{ label: "Skip for now", onPress: advanceOnboarding }}
/>

Every line of that copy is doing a job, and most of the jobs come straight from game design.

"Gentle reminders for your daily habits" names the concrete thing the user just set up, the goals from two screens ago, so the notification is in service of something they chose. "Progress milestones worth celebrating" is the reward loop, games learned a long time ago that people say yes to notifications about wins, a level completed, a streak kept, much more readily than to notifications about obligations. "Lab result updates the moment they're ready" gives the notification an obvious job, the same way a game asks for permission when your building has an hour left, we'll tell you when it's done.

And "you're in control, turn off or customize anytime in Settings" lowers the perceived cost of saying yes. Good games always let you toggle notification types individually, and telling the user up front that they keep that control makes "Turn on notifications" feel reversible instead of a commitment.

Then there's "Skip for now", and this one needs explaining because it looks like a mistake. It isn't. Skipping our screen costs nothing, the permission status stays undetermined, so we can ask again later at a better moment, after the user's first completed habit or their first lab result. Declining the OS prompt costs everything, it burns the one-shot permanently. So the skip button is there on purpose, we'd much rather collect a cheap "not now" on our own screen than push an unsure user into an expensive, irreversible "Don't Allow" on Apple's.

The hook

The screen calls a single hook on continue:

// hooks/useNotificationSetup.ts — abridged
import * as Notifications from "expo-notifications";

export function useNotificationSetup() {
  const requestNotificationPermission = useCallback(async () => {
    const current = await Notifications.getPermissionsAsync();

    if (current.status === "granted") {
      // Already on. Skip the prompt; advance the onboarding.
      return "granted";
    }

    if (current.status === "denied") {
      // Already declined. Show the "open Settings" sheet instead of the prompt.
      showSettingsSheet();
      return "denied";
    }

    // status === "undetermined" — the one case where the prompt will actually fire.
    const result = await Notifications.requestPermissionsAsync({
      ios: {
        allowAlert: true,
        allowBadge: true,
        allowSound: true,
        allowAnnouncements: false,
      },
    });

    if (result.status === "granted") {
      await registerForPushToken();
    }

    return result.status;
  }, []);

  return { requestNotificationPermission };
}

Three guards in three lines. "Granted" skips the prompt entirely, "denied" opens the Settings deep-link sheet so the user can recover, and "undetermined" is the only path that calls the OS request.

Asking for the four iOS flags explicitly is worth doing. We didn't enable allowAnnouncements, that's the Siri "read this aloud through AirPods" permission and the product doesn't need it. Asking for permissions you don't need just nudges the user toward decline.

The Settings sheet, for the users who already said no

We show a separate small sheet for users whose status is already "denied":

<BottomSheet
  title="Notifications are off"
  body="You said no the first time, which is fine. The only way to turn them on now is in your phone's Settings."
  primaryAction={{
    label: "Open Settings",
    onPress: () => Linking.openSettings(),
  }}
  secondaryAction={{ label: "Maybe later", onPress: dismiss }}
/>

The copy matters here. "You said no the first time, which is fine" respects the user's earlier choice, and "the only way to turn them on now is in your phone's Settings" is the honest mechanic, not a sales pitch. The share of declined users who open Settings from this sheet is meaningfully higher than the share who responded to the previous "Notifications are off, enable them" banner inside the app shell.

The measurement

We're tracking three numbers, weekly.

Our screen's tap rate. What share of users who land on our screen tap "Turn on notifications".

Permission grant rate after the CTA. Of the users who tap our CTA, how many then tap "Allow" on the system prompt that follows.

Net grant rate. Our screen's tap rate times the system-prompt grant rate. The lift over the bare system-prompt baseline is the metric the product team watches, and on this product it's meaningful, meaningful enough that showing our own screen first is now the model for every new permission ask, not just notifications.

What I would tell another engineer setting this up

1. Never call requestPermissionsAsync without checking the status first. The one-shot rule on iOS is unforgiving, the three-branch guard is one of those patterns where the lines you don't call matter more than the ones you do.
2. Write the screen copy around concrete, positive jobs. Habit reminders the user just set up, lab results the moment they're ready, milestones worth celebrating. Specific and rewarding beats "stay engaged with the app", the same reason game notifications lead with wins, not obligations.
3. Let users skip your screen, never let them skip into the OS prompt. A "Skip for now" on your own screen is cheap, the status stays undetermined and you can ask again at a better moment. A "Don't Allow" on the system prompt is permanent. Design the screen so the unsure user takes the cheap no.
4. Settings deep link for the declined cohort. Most users who declined the first time won't come back, some will, so make the recovery a one-tap path.
5. Ask for the iOS flags you actually need. allowAnnouncements isn't free, every permission you ask for raises the bar to "yes".

For a long time I was the person who shipped it. CI/CD pipelines, Terraform modules, the monitoring stack, the "why is staging down?" Slack messages at midnight. I have been doing that work for nearly two decades, even before it was called DevOps.

Earlier this year I took on a different job inside Clearview Team. Now, before any new feature ships, somebody spends a day trying to break it. Often, that somebody is me.

The Night the Job Description Changed

It was March 2026. I could not sleep. Too much coffee, too many open browser tabs, and a Hacker News thread I could not stop reading about a breach that turned out to be a misconfiguration — a public-read S3 bucket exposing millions of records.

"That could never happen to us," I thought. "We have policies. We check these things." Then I got up, opened my laptop, and started checking anyway.

Four hours later I had found three public S3 buckets in our own infrastructure. None of them held customer data — they held logs and config files, enough information for an attacker to draw a map of the system.

The Terraform that created those buckets had been written in the early days, before we had standards, before we had a review process, before a single checkov rule ran in the pipeline. Nobody had ever gone back. They had terraform apply-ed once and moved on.

Those buckets were not somebody else's mistake. They were my Terraform, my missing review process. Finding them was less a security win than a critique of how I had been working for years.

I fixed them that morning, wrote up the incident, told the team. And then I could not stop.

What an Engineer Reads at Night When They Cannot Sleep

I started spending evenings on a different reading list:

• JWT vulnerabilities — specifically how stateless tokens make revocation almost impossible
• SQL injection patterns that still work in 2026 — parameterised queries handle the textbook case, but ORMs like TypeORM ship their own foot-guns (raw queries, addSelect on relations, QueryBuilder with string interpolation)
• CORS bypasses — wildcard subdomain matching, reflected origins, preflight caching abuse
• What trust proxy actually does in Express — and why getting the hop count wrong means your rate limiter trusts attacker-supplied IPs, which matters a great deal if you are sitting behind CloudFront and an ALB

I am not claiming to be a research-level expert in any of these. I am claiming curiosity. I went deep enough on each to spot the same patterns in our own codebases and to ask sharper questions during reviews. That is different from being a full pen tester, and I try not to confuse the two.

Slowly, I started looking at our code differently. Every endpoint became a question about what happens when somebody sends weird input; every configuration became a suspicion about whether it was actually secure or only looked secure.

I was running curl against staging at strange hours, creating test invoices, reading the error messages line by line, mapping out what existed. I was becoming paranoid, in a mostly-good way.

The hard part about paranoia is knowing when to put it down. I have lost sleep over things that turned out to be fine. I have filed false positives with embarrassing confidence — I once shipped a critical report about a writable S3 bucket because I had tested with aws s3 cp --dryrun, which validates syntax but not IAM. The actual upload failed with AccessDenied. I had to correct my own report the next morning.

Paranoia is useful, but it needs guardrails: a checklist, a second opinion, the willingness to say I was wrong out loud and in writing.

What Tools Catch and What They Miss

People often ask why I do so much of this by hand instead of running Burp Suite, OWASP ZAP, or one of the big enterprise scanners.

I do use those tools. They are excellent at coverage. A DAST scanner will methodically poke every parameter on every endpoint. A dependency scanner will pick up known CVEs in your package-lock.json. These are table stakes — you should run them no matter who is on your team.

But automated tools find known patterns. What I find by hand is different:

• The CORS rule that accepts any subdomain — technically valid regex, but allows attacker-controlled origins
• The scientific-notation invoice for one million dollars — 1e6 passes IsNumber() validation because it is a number, but the business logic never expected six figures
• The internal fields that nobody should be allowed to set, but anyone can — Object.assign(user, req.body) happily merges isDeleted and role along with everything else
• The webhook that accepts whatever you send it because validation was deferred to an async queue, which also does not validate — a design tradeoff that quietly became a security gap

None of these have a CVE number. They are business-logic vulnerabilities — the consequence of how this specific code, this specific ORM configuration, and this specific architectural decision interact. Scanners do not know a client's business rules; the person who reviews the diff does.

That said, manual testing does not scale. I cannot personally curl every endpoint before every release for every Clearview client. So we run both — automated tools for breadth, a human for depth — and every manual finding becomes a custom rule that catches the next instance automatically.

The Role We Wrote Around It

At some point my late-night reading turned into part of the job. It happened gradually.

First I added security checks inside our Terraform modules — automated S3 bucket policy validation, IAM policy scanning, the kind of guard that would have caught those public buckets before they ever reached production.

Then I started doing quiet pre-deployment reviews. Not formal audits. Just "let me look at this endpoint before it goes live."

Then, on a real client project, I found a vulnerability that would have let anyone create invoices without authentication. No login. No rate limiting. Just curl -X POST and a JSON payload, every minute, forever.

That was the inflection point. Not because it was the scariest finding I have ever shipped a report on, but because there was a dollar sign attached. The team — and the client — could see exactly what an attacker would have done with it. Abstract security risk is easy to deprioritise. A vulnerability that prints fake invoices in your accounting system is not.

After that, the role was real.

Today, at Clearview Team, every new feature we ship to a client gets sent through a pen-tester pass before it lands in production. That pass is a known step in the sprint — scheduled and time-boxed. It is the same engineer (often me) sitting down with the diff and asking, on purpose, how do I break this?

That is what an in-house pen tester gets you that an external annual audit does not. The feedback loop is short, and the fix usually lands in the same sprint that introduced the risk, not nine months later in a remediation backlog.

What I Catch on the Pen-Tester Pass

Here is the shortlist of categories that show up over and over again across stacks and clients:

1. Auth holes on new endpoints. Somebody adds a route, forgets the auth decorator, ships. I catch this before the route hits production by asserting every endpoint either carries @Authorized() or sits on an explicit public-endpoint allowlist.
2. Mass assignment via ORM helpers. Object.assign(entity, dto) looks innocent. It is not, when the entity has fields the user is never supposed to set. I run a curl against the endpoint with a payload that sets every internal field and see what survives.
3. IDOR / BOLA on numeric IDs. Anything in a URL of the form /invoices/:id. I swap in another tenant's id and read the response.
4. Input validation that is actually type checking. IsNumber() lets 1e9 through. IsString() lets a 50 MB payload through. I send the obvious edge cases and read the error messages.
5. Rate limits that count the wrong IP. If trust proxy is misconfigured, every request looks like it came from the load balancer. I send ten thousand requests with rotating X-Forwarded-For headers and watch what the limiter thinks.
6. Infrastructure drift since the last engagement. A bucket that was private six months ago is public now because someone toggled a setting in the console. I re-run checkov and tfsec against the live Terraform plan, not just the repo.

None of this is novel. The novelty is that it happens every sprint, by the same person, on the same engagement. Familiarity with the codebase is the hidden multiplier — an external auditor will never find the business-logic invoice bug, because they will not know what an invoice is supposed to look like for this client.

What It Gets the Client

Clearview Team takes on web and mobile build engagements for startups and scaling companies. The teams we work with are usually small, usually shipping fast, and usually one configuration drift away from a story they would rather not be in.

What the client gets:

• A feature is never the first thing to find a vulnerability in production. A peer reading the diff with an attacker mindset is.
• They don't pay extra for the security review. It is not a separate procurement, a separate vendor, or a separate quarter. It rides on the same sprint that ships the feature.
• When we find something, we hand them the fix. I am an engineer. They do not get a thirty-page report — they get a pull request.
• The codebase gets harder to break the longer we work on it. Every finding becomes a CI rule that catches the next instance automatically. The boundary tightens by itself.
• No surprise certifications, no surprise breach. When a SOC 2 auditor or a customer due-diligence questionnaire shows up, the answers are already true.

I will not put a percentage on what this prevents — I do not know what would have shipped without it, and Clearview does not invent numbers for sales copy. In the months since this role has been real, no Clearview client has shipped a production-incident-class security bug from a feature that went through the pen-tester pass. The ones we have found in production were inherited from before our engagement, and we wrote up exactly what we found and exactly what we changed.

What This Costs Us

Adding this role to every engagement is not free.

• A sprint takes a little longer. Time-boxed, but real.
• Some features get re-scoped. When we hit something structural rather than local during the review, the fix is sometimes an architectural change. That conversation is uncomfortable, and we have it anyway.
• The pen tester needs a peer. I cannot review my own work alone — I have been wrong before and I will be again. The second opinion is part of the process, not an add-on.

We treat all three as features of the model rather than bugs in it. A feature ships a little slower and holds up better in production.

The Mindset Shift, for Anyone Considering the Move

If you are a DevOps engineer reading this and wondering whether you should make the same shift — you do not need a certification. You do not need to become a full-time pen tester. You need to start asking what if? about the systems you already own.

Five things that helped me, in order:

1. Audit your own infrastructure first. Run checkov on your Terraform. Check your S3 bucket policies. Review your IAM roles. You will probably find something, and it will motivate everything that follows.
2. Learn one attack vector deeply. Mass assignment if you use an ORM with auto-mapping. BOLA/IDOR if your API uses sequential IDs. Authentication bypass if you run your own auth flow. SSRF if your app makes outbound HTTP calls based on user input. Pick one. Learn it well enough to spot it in the wild. Then pick another.
3. Read the detailed breach post-mortems. Not the sensational ones. Cloudflare's. GitLab's. The ones that explain the root cause, the timeline, and what changed. That is where real attacks live.
4. Break something — in staging, with permission. Find a vulnerability. Exploit it responsibly. Document the steps. See how it feels. That feeling — the mix of excitement and dread — is how you know you are thinking like an attacker.
5. Turn every finding into a check. Every manual finding should become a rule that runs automatically next time. Found a public S3 bucket? Add a checkov rule. Found a missing auth decorator? Add a CI test that verifies every route is on the auth list. The goal is to make yourself unnecessary for the easy stuff, so you can spend your time on the hard stuff.

The paranoia helps, but the checklists help more.

Prefer a downloadable version Read the case study as a PDF Download ↓

The Client

A Confidential Non-profit and Awards Group *Membership Operations, Automated and Simplified *

At one time managed primarily through Excel spreadsheets and mail-in forms, the client’s membership system was functional but time-intensive for its administrators. With the benefit of an automated, online membership management system, client’s Board of Directors wisely contracted Clearview to combine member-related functions into one web-based platform.

More than a decade later, we’re still working in close partnership with the client to maintain and streamline member processes and relations through a secure, custom-built online portal. We’re continually finding new ways to empower administrators and members alike with rich tools and features, supporting this 501(c)(3) non-profit’s continued growth.

• Purpose-built member dashboard with quick access to useful member functions including online signup, membership upgrade, and renewal; info & address changes; member-only services and perks; automated password reset & forgotten login recovery; notification preferences.
• Powerful administrator dashboard, complete with tiered admin privileges, quick member lookup & sorting, bulk member management, numerous import/export options, and detailed reports.
• Secure, multi-method payment processing with in-dashboard payment logs, receipts, and refund processing.
• Newsletter and notification management including automated synchronization with MailChimp newsletter and SendGrid transactional email platforms.
• Scheduled self-maintenance and member relations tasks including proactive address validation and expiration notifications.

The Digital Workflow Platform

Annual Intake and Review Workflow

The client also runs an annual intake-and-review cycle that once involved sorting through over a thousand hand-mailed physical packages. This veritable mountain of material was then shuffled through very extensive multi-stage review and selection processes by hand.

Since joining the client’s team in 2010, Clearview has systematically brought each of these processes together in a single online platform.

• Fully web-based intake process, including secure online payment; upload of video clips, documents, and signoff forms; followup intake for supplementary materials.
• Automated media verification, storage, and transcoding, utilizing Vimeo API integration for high-quality, secure, and familiar playback for reviewers. All media is secured against unauthorized download.
• Advanced partner dashboard, enabling organizations with many moving parts to delegate intake tasks to multiple designated contributors and divide labor.
• Review management, including secure reviewer application and selection process; a dedicated review portal with high-quality material review, shortlisting, issue flagging, and discussion features; selection and notification workflows.
• Selection management, including application and eligibility processes and a secure selection interface.
• Tight integration with the organization’s membership database for permissions verification.
• Extensive administration dashboard, including tiered admin roles; organization and intake management; category and field management; payment, receipt, and transaction management; flexible sponsor-credits system; deadline management, late fee rules, and exceptions; numerous import/export tools.
• Secure, triple-redundant data archival following each year’s cycle.
• On-call support, including email support for contributors, reviewers, and participants.

The Project

Clearview Team's engineers and clients are spread across time zones, so monitoring and incident response carry real coordination overhead. This engagement is where we chose Event-driven architecture.

Challenges

The challenges we aim to solve with this approach:

• Decoupling of services: To improve the agility of the application we designed our system not to be tightly coupled.
• Real-time responsiveness: The application services operate on event-driven communication, which means the services can produce an event in real time. This enables our on-call engineers to react to important events that are produced by either user actions or system events.
• Improved observability: The event-driven approach improves system observability. By analyzing event flows, we constantly monitor and improve the system behavior, identify bottlenecks, and locate the root cause of issues more effectively.

Deployment

The application is deployed on the AWS Cloud platform as AWS already provides all the communication buses that we need to produce, broker, and consume events.

Event Types We have covered the most important event types in the app, such as:

• Finance events
• Membership events
• Scheduled job events
• Health check events.

Goals

This project aims to solve the client’s requirements to create and continuously improve the membership and awards system while implementing a modern system architecture. More details about what kind of problems the application is designed to solve can be seen in the diagram below.

System Architecture

This application is built around separate services, each running inside its own container. Services communicate through events — state changes that trigger actions elsewhere in the system.

Event-driven Architecture

An event is a state change — a user placing an order, a payment failing, a scheduled job firing. Services produce events and other services consume them, without needing to know about each other directly.

Event-driven architecture is often called EDA for short.

The rest of this post covers how we wired the event bus for job scheduling and monitoring — the piece that lets our team get notified in real-time and react before downtime reaches users.

Event-driven Architecture Showcase

Below is a diagram showing what a typical EDA system looks like and how it functions.

The Application

The application is built on NodeJS with an API service serving multiple front-end domains. This approach has served us well. We have been running the app as a container on AWS.

Please keep in mind that the client communicates with the API after it has been loaded on the client-side rendered website.

Event Producers

Events are generated from various services and can be produced either by a client action or by a scheduled job.

API Event Producer

The best example of a client-produced event is any type of event related to credit card payments. The app uses the “@aws-sdk/client-eventbridge” library to communicate with the AWS Event Bridge service.

API Events List

The core services of the app produce events. Below is a list of services with a description of what each does.

API Produced Sample Events

Finance Related Events Below you can find payment processing failed sample events

Scheduler Event Producer

As it was not necessary to re-implement the wheel and create our scheduling service, we relied on the well-tested and proven AWS EventBridge Scheduler for the scheduling system. For this purpose a custom and dedicated Event Bus was created and named Client Production Scheduler Bus.

In this case, the Event Producer is the Scheduler itself.

As per the diagram every cron job has the following flow:

• a scheduler of recurring type that produces an event and sends it to the custom scheduler bus,
• an endpoint that is defined as part of the api_destination event rule.

Evaluated Jobs

The complete list can be found below.

Company Statistic Scheduled Job

Event Scheduler Job Notes:

• Recurring schedule type
• Cron expression style
• Runs every 8 hours
• Target: Scheduler Bus
• API PutEvents

Event Rule

The bus receives the event and triggers the api cron company statistic rule.

The Infrastructure

In this section, we will cover how the AWS infrastructure was set up, the services that send events to the default Event Bus, and how we process and consume those events.

The AWS Account

We used AWS Organizations to separate Production and Development environments. This is important as infrastructure-related events are sent to the default event bus. We wanted production and development events not to trigger the same rules as this could lead to unexpected behaviors.

It makes it easier to have two environments not mixing up as we know for a fact that development events will not end up on the production Event Bus causing potential problems.

AWS RDS

The database service’s health and availability are very important for the app’s reliability. Therefore, we have a setup of monitoring database events with Slack notifications.

Every time there is an event such as the database service backing up an event is created and sent to the Event Bus. Such an event will end up as a Slack notification and be read by our team.

We decided that the following event categories will be forwarded to Slack: Notification, availability, backup, failure, low storage, maintenance and recovery.

The EventBridge Rule

AWS Health

Events type can be one of the following categories:

• accountNotification
• issue
• scheduledChange

AWS Health events are sent to the default Event Bus. We want to react when we receive an event from the AWS Health service, therefore we create rules. For example, you can use AWS Health to receive email notifications if you have AWS resources in your AWS account scheduled for updates, such as EC2 instances. Below you can find examples of such rules.

EC2 service

Sample rules

Trigger on Every Event

Rule for a specific service and event type category.

In this example, we will create a rule so that EventBridge reacts to the following.

Multiple Services and Event Type Categories

The examples in the previous procedure show you how to create a rule for a single service and event type category. You can also create a rule for multiple services and event-type categories. This means that you don’t have to create a separate rule for each service and category that you want to monitor. To do so, you must edit the event pattern as per following example

Example Rule

I remember the joy when I first deployed my web application using cPanel FTP. I right clicked on my folder | compress to zip, then upload it to cPanel, uncompress everything, and setup the credentials — no .env file, everything is hardcoded on the config file.

It took me multiple years of learning and experience to be able to deploy a software correctly. I will write a summary on how to do it, one for my personal reference, and two so people can learn about it as well and doesn’t have to fall into the pit of having their software hacked.

Assumption

This article assume that we will be deploying these software systems:

• Server (VM)
• Database
• Backend API
• Frontend
• Nginx
• SSL

We won’t go deep into horizontal scaling or multiservice architecture as most of the time, you won’t need it.

We assumed that you have experience on interacting with server, writing code, and using Linux.

Setting Up The Server

We will go with the traditional route of using a virtual machine / virtual private server — instead of ready to use system like AWS AppRunner or Google App Engine.

You will have to pick an OS — most providers support Windows, but unless you are deploying Microsoft based software like .NET — it’s always a good idea to use Linux.

Go ahead and set up your server using Digital Ocean, Google Compute Engine, or any other providers.

Securing The Server

The first step that you need to do once you got your server up and running is to secure it.

The main gate of your server is most likely an SSH server, so we will secure it first.

1. Create an SSH key if you don’t have already
2. Create a new user with strong password, use your SSH key for this user
3. Add the user to sudoers
4. Configure SSH to only allow this user to log in
5. Configure SSH to prevent password login
6. Configure SSH to disallow root login

The > means you write it as content of the file, not an actual character that you type.

# Run on your local
ssh-keygen -t rsa -b 4096

# Connect to your server
adduser new_username
mkdir /home/new_username/.ssh
chmod 700 /home/new_username/.ssh
nano /home/new_username/.ssh/authorized_keys

usermod -aG sudo new_username

nano /etc/ssh/sshd_config
> AllowUsers new_username
> PasswordAuthentication no
> PermitRootLogin no

Then we need to configure the firewall to shutdown everything.

1. Enable firewall
2. Disallow all port except SSH

# firewall
ufw enable
ufw default deny incoming
ufw allow ssh

We will whitelist our port later.

Next, use the OS package manager (apt, yum, etc) to update all existing softwares.

apt update
apt upgrade

Setting Up Database

For our database server, we’ll be using PostgreSQL, a battle-tested and feature-rich open-source database management system. MySQL is a good alternative but I kept forgetting how to setup Postgres properly so here I am writing about it.

Installing PostgreSQL on our Ubuntu server is a straightforward process. First, we’ll update the package index and install the PostgreSQL package:

sudo apt update
sudo apt install postgresql

After the installation, PostgreSQL automatically creates a default database cluster. However, we’ll create a new cluster with our preferred settings to ensure optimal performance and configuration.

Switch to the PostgreSQL user and initialize a new cluster with the desired locale and encoding settings:

sudo su - postgres
initdb -D /path/to/data/directory --locale=en_US.UTF-8 --encoding=UTF8

Next, we’ll configure PostgreSQL by editing the postgresql.conf file located in the data directory we specified during cluster initialization. Here, we can adjust settings such as listen addresses, port numbers, maximum connections, shared buffers, and memory allocation for various operations.

For authentication, we’ll edit the pg_hba.conf file. During development, we can use peer or ident authentication for local connections and MD5 for remote connections. However, in a production environment, it's recommended to use MD5 or certificate-based authentication for enhanced security.

To create a dedicated PostgreSQL user and database for our application, we’ll execute the following commands as the postgres user:

createuser --pwprompt app_user
createdb --owner=app_user app_database

This creates a new user (app_user) and a dedicated database (app_database) owned by that user.

By default, PostgreSQL doesn’t allow remote connections. To enable remote access, we’ll update the listen_addresses parameter in postgresql.conf and add a line in pg_hba.conf to allow remote connections with appropriate authentication methods.

Securing the PostgreSQL server matters. We’ll disable the PostgreSQL user’s ability to log in via password, use strong passwords for database users, consider SSL/TLS encryption for connections, and regularly update PostgreSQL to the latest version for security patches.

Regular backups are essential for data integrity and disaster recovery. We’ll set up backup procedures using tools like pg_dump or pg_basebackup, storing backups in a secure off-site or cloud location. Testing backup and restore processes regularly is also a best practice.

Performance monitoring and tuning are ongoing tasks. We’ll use tools like pgBadger or pg_stat_statements to monitor PostgreSQL's performance, tune database settings based on workload and hardware resources, implement indexing strategies, and consider partitioning large tables for better management and performance.

With our PostgreSQL database server set up, configured, and secured according to best practices, we’re ready to connect our application and begin development and deployment processes.

Setting up your server timezone so your server, dabase, and application uses the same timezone settings will save you headache in the future. Use UTC if you are serving international customers, or set it to your local time if you are sure that it’s only going to be used internally or specific to your region.

Preparing Your Backend Service

The first step that you need to do is to ensure that you don’t have any secret credentials in any of your version controller file. Use environment variables and make sure your backend service use the environment value instead of hard coding it on your code.

We won’t talk about how you version control or how you managed to get your code into the server. A quick info on this, you can create an SSH key on your server and use it for deploy keys on GitHub.

Once you securely move all of your secret information into environment variables, it’s time to get it up and running

The Daemon

While you can just npm start — it will run in foreground and once your session ends, your backend service will die as well.

Systemd is a popular daemon system that we can use to ensure our backend service will keep running even when we close our SSH session, or when it fails and need to restart.

First, we need to have a dedicated user to run our service. So create a new user and define a daemon configuration that uses this user.

sudo adduser backend-app-user

nano /etc/systemd/system/backend-app.service

> [Unit]
> Description=Backend Application
> After=network.target

> [Service]
> User=backend-app-user
> Group=backend-app-user
> WorkingDirectory=/path/to/backend/app
> Environment=NODE_ENV=production
> Environment=PORT=3000
> ExecStart=/usr/bin/node /path/to/backend/app/app.js
> Restart=always
> RestartSec=10

> [Install]
> WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl start backend-app
sudo systemctl status backend-app
sudo systemctl enable backend-app

Now your backend service is up and running, it’s time to set up the frontend.

Deploying The Frontend

Most frontend is just a static files, even if you use framework like React, Vue, Angular — in the end it will be compiled into a static file.

We won’t talk about server-side frontend like Next, you’d better of writing a fullstack application using Rails or Laravel — trust me.

Because front end is just static files, we will just need to make sure that we can bring our frontend artifacts into our server.

Nginx

Nginx is fast, small, and easy to configure. So let’s use it.

First, let’s install Nginx on our server:

sudo apt update
sudo apt install nginx

After the installation, Nginx will start automatically, and you can verify its status with the following command:

sudo systemctl status nginx

Next, we’ll configure Nginx to serve our frontend application’s static files. Create a new configuration file (e.g., frontend.conf) in the /etc/nginx/conf.d/ directory:

sudo nano /etc/nginx/conf.d/frontend.conf

Paste the following configuration into the file, replacing /path/to/frontend/dist with the actual path to your frontend application's built or compiled static files:

server {
    listen 80;
    server_name your_domain.com www.your_domain.com;
    root /path/to/frontend/dist;
    index index.html;
    location / {
        try_files $uri $uri/ /index.html;
    }
}

This configuration tells Nginx to listen on port 80 (the default HTTP port) for requests to your_domain.com and www.your_domain.com. It sets the document root to /path/to/frontend/dist, which is where your frontend application's static files are located.

The try_files directive ensures that Nginx will first try to serve the requested file or directory. If neither exists, it will serve the index.html file, enabling client-side routing for single-page applications.

But How Do I Call Backend From My Frontend?

We managed to run our backend service, but it running locally on a local port. We want to have a reverse-proxy that act as a gate. So it will become like FE -> Nginx Proxy -> Backend.

Modify our nginx website conf,

server {
    listen 80;
    server_name your_domain.com www.your_domain.com;

    root /path/to/frontend/dist;
    index index.html;

    location / {
        try_files $uri $uri/ /index.html;
    }

    location /api/ {
        proxy_pass http://localhost:3000/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

Now your_domain.com/api become the base endpoint for your API.

Securing Nginx with SSL/TLS

It’s 2024, if you try to open your_domain.com — browser will shame you publicly by saying your website is dangerous/insecure/badly written in React.

Before we can obtain our SSL certificate, we need a domain name that points to our server. You can open your domain provider and setup an A record that points to your server IP.

Being 2024 means we have Let’s Encrypt to help us obtain SSL certificate. Let’s start with installing certbot.

sudo apt update
sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx

Certbot will automatically configure everything for you. Make sure that everything is good, test your nginx configuration and reload it.

sudo nginx -t
sudo systemctl reload nginx

Let’s Encrypt SSL has a short lifetime, make sure to run sudo certbot renew every 90 days to make sure that your certificate is valid. Most of the time this will be done automatically.

Now, enable port 80 and 443 on your firewall and you’re all set. You probably want to do more future proofing like setting backups, setting monitoring, compressions, caching, etc. But, enjoy your first step for now 🎉