What a high-stakes marketing launch taught me about boring infrastructure

At CoStar I built event-driven enrollment pipelines (Lambda, API Gateway, SQS, Kafka) for the Homes.com launch, and worked on the marketing-automation platform that sends to millions of contacts a day across seven-plus marketplaces. The work that mattered most was rarely the part that got demoed. It was the plumbing that made sure a user got one email instead of two.

Here is a simple version of the risk. Someone sets up property alerts on Apartments.com. Later they also create a Homes.com profile with the same email and a slightly different name. A campaign fires on listings that match their alert. If the system treats them as two people, they get the email twice. At the volume these platforms send, even a tiny duplicate rate is a lot of annoyed people.

Below is the boring-infrastructure list I keep coming back to. It isn't complete. It's the set of things I've learned to look for first.

Idempotency keys on every write path.

On the campaign-send path, one piece I owned was a strict idempotency key. I didn't design the whole pipeline, but this part was mine. Every send command carries a deterministic key derived from campaign, contact, and send window. The service checks the key against a short-lived store, and a duplicate gets acknowledged and dropped.

It's the least interesting line of code in the service. It's also what makes upstream retries harmless. Queues redeliver, partitions get replayed, and a consumer that assumes "exactly once" will eventually be wrong. With the key in place, a retry is a no-op instead of a second email.

If your service writes anything a user can observe (sends, charges, notifications, state changes), it needs an idempotency key. Build it first and treat everything else as a fallback.

Audit logs as a typed schema, not "log everything."

The easy approach to an audit log is to write a row for every decision and figure out the shape later. That holds up until several services write to it. Each one ends up with its own idea of what a field like "campaign source" means, and reconciling them becomes a recurring analytics chore.

The fix isn't a bigger logging library. It's a typed, versioned schema with a clear owner, and a CI check that rejects any write that doesn't validate. It's dull work and it's much cheaper on day one than after the definitions have drifted.

Dedup against a canonical record, not a hash of inputs.

The naive design is to hash the email and campaign ID and deduplicate on that. It works until the same person exists as two records, like a Homes.com user who is also an Apartments.com user.

The sturdier approach resolves each contact to a canonical record first and dedups on that. A contact's canonical ID rarely changes, so the lookup caches well, and the miss path can go to the warehouse. Give the reconciliation an explicit latency budget on the write path so it can't quietly become the slowest step.

Let the queue push back.

When ingestion gets a spike, say a homepage feature or a partner replaying events, the cheap answer is to raise Lambda concurrency and trust AWS to absorb it. The better answer is for the queue to push back on the producer.

Backpressure looks bad on a dashboard because it shows up as fewer events processed per second. That's the point. A system that refuses to ingest at twice its capacity is recoverable. A system that ingests twice its capacity and then takes down two services downstream is not.

The shape: SQS visibility timeouts tuned to downstream latency, Lambda concurrency capped below what the downstream can absorb in a burst, and a producer that treats "queue is full" as a normal response. None of this is clever. All of it keeps a launch quiet.

Rehearse failure before launch, not during it.

Before a launch, write down the failure modes that worry you most. For each one, note what it looks like in your monitoring, the first command to run, and how to roll back. Then actually rehearse some of them, because a runbook nobody has run is a guess.

Rehearsal won't stop an incident. What it does is prevent the second one, where people improvise the recovery under pressure and make things worse. The gaps you find in a calm rehearsal are much cheaper than the ones you find live.

The principle.

Boring infrastructure is what lets the interesting parts of a product stay interesting. The campaign engine gets the demo. The idempotency keys and dedup get the launch through. You don't ship the first without the second.

When I join a new codebase, the first things I look for are idempotency keys, an audit-log schema, and runbooks that someone has actually tested. When they're missing, I budget time to add them before an outage makes the case for me.

Owen