You Don't Need Five Databases (Yet)

It's Monday morning, and you're looking at an architecture diagram.

On one side sit five boxes: Redis for the queue, Elasticsearch for search, Pinecone for the embeddings, a managed cron service for the scheduled jobs, Kafka for the event log. On the other side sits one box - Postgres, quietly doing all five jobs.

Which one do you build first? Honestly, neither answer is wrong. But only one of them is right for where you actually are right now.

Five boxes on day one

Go with the five boxes, and you're not making a bad call exactly. Every one of those tools is genuinely good at its job. Redis is fast. Elasticsearch is good at search. Kafka is built for real throughput. Nobody's arguing with any of that.

What's easy to lose track of is everything you're signing up for underneath those good intentions. Five things to provision. Five things to monitor. Five sets of credentials to rotate. Five vendors, any one of which can independently page you at three in the morning. And you're taking all of that on for load you don't have yet - forty customers or a hundred, it barely matters, the number is still small enough that every one of those five systems is running at roughly zero percent of what it was built for.

You didn't buy capacity. You bought five waiting rooms, and you start paying rent on all of them today.

What one box can actually do

Go with the one box instead, and here's roughly what that looks like once you get into it.

A job queue turns out to be one query:

SELECT id, payload
FROM jobs
WHERE status = 'pending'
ORDER BY created_at
FOR UPDATE SKIP LOCKED
LIMIT 10;

Multiple workers can pull jobs off the same table without stepping on each other, because SKIP LOCKED tells Postgres to hand each worker a different set of rows instead of making them queue up behind the same lock. No broker to run, nothing extra to keep alive at 3 AM.

Search works out almost as cleanly. Full-text search is a tsvector column and a GIN index, with the pg_trgm extension layered on top when you also want fuzzy, typo-tolerant matching:

ALTER TABLE articles ADD COLUMN search_vector tsvector
  GENERATED ALWAYS AS (to_tsvector('english', title || ' ' || body)) STORED;

CREATE INDEX articles_search_idx ON articles USING GIN (search_vector);

SELECT title FROM articles
WHERE search_vector @@ plainto_tsquery('english', 'connection pool')
ORDER BY ts_rank(search_vector, plainto_tsquery('english', 'connection pool')) DESC;

That gets you ranked search on data you already have, inside the same transaction as the write that created it - no separate index to keep in sync, no second system that can drift out of date with the first.

The rest of the list holds up the same way. Vector similarity is the pgvector extension: you store the embedding right next to the row it describes, index it with HNSW, and query it with regular SQL. Scheduled jobs are pg_cron, so the cron table lives in the same database as the data the job actually touches, instead of sitting in a separate scheduler that has never heard of your schema. And an event log is just an append-only table, or a proper outbox pattern feeding logical replication downstream if other services need to react to what happened.

None of this is a hack you're getting away with. Every one of these is a maintained extension or a documented pattern, and people run all of it in production, at real scale, today.

So on day one, the one-box path isn't the cautious choice you settle for. It's genuinely the correct one.

Where the one-box path starts costing you

It does have an expiration date, though, and it's worth walking through what actually happens once you reach it, because the failure modes aren't vague - they're specific enough to watch for.

The connection pool problem

Say max_connections is sitting at the default, roughly a hundred - most teams never touch that number, because why would you until something forces the question. Your app servers hold maybe twenty of those connections at any given time. Then your queue grows to fifty workers, because someone doubled the worker count last year just to clear a backlog faster. Each of those workers holds a connection open while it polls, so that's seventy connections spoken for before a single customer even opens your checkout page.

Then the summer clearance sale kicks off. Traffic triples. Checkout tries to open the connections it needs, and there simply aren't any left in the pool. Your customers don't get a friendly "the database is a bit slow today" - they get a 500, or worse, a request that just hangs until it times out. Nobody touched the checkout code. The queue quietly ate the connections checkout needed, and nobody noticed until the graphs did.

The vacuum problem

Say your queue processes fifty thousand jobs a day: insert a row, work the job, delete the row, fifty thousand times over. Every deleted row doesn't vanish the moment you delete it - it becomes a dead tuple, sitting there until autovacuum gets around to reclaiming the space. Under normal conditions that's completely fine; autovacuum exists for exactly this pattern.

But say someone kicks off a long analytics query against that same database - a report that runs for forty minutes. While it's running, Postgres can't safely clean up dead tuples that the still-open transaction might still need to see, so for those forty minutes, vacuum just waits its turn. Do that with a handful of reports a day, and your queue table, which should weigh in at a few megabytes, quietly balloons to twenty gigabytes of mostly dead rows. Every query against it gets slower - not because the queue itself grew, but because the table underneath it never actually shrank back down.

The buffer cache problem

This is the one that catches people off guard, because nothing about it looks like a mistake. Say you run a nightly job that re-embeds anything a user edited that day and rebuilds part of your HNSW index. It kicks off at 2 AM, scans a few million vectors, and pulls a large chunk of that index into memory to get through the work.

Postgres's buffer cache isn't infinite, so something has to make room for all of that, and what usually gets evicted is whatever was least recently touched (LRU). At 2 AM, that's often the orders table your checkout flow has been reading from constantly all day. By the time your first customers show up in the morning, those pages have gone cold, and the first checkout queries of the day quietly pay for a disk read instead of getting the memory hit they're used to. Nobody deployed anything overnight. Nobody touched checkout. Your vector search just rearranged the cache while everyone was asleep.

Put the three together and you can see the actual cost of one box doing five jobs. It's not that any single job is too heavy for Postgres to carry - it's that all five of them are quietly competing for the same connections, the same vacuum cycles, and the same slice of memory, and none of them can see the others doing it.

Where the paths reconverge

Here's the reassuring part, though: the two paths aren't nearly as far apart as the diagram makes them look, because the decision to split something out of the one-box path was never supposed to be a feeling. It's a number, and you can go look for it.

• Queue: is vacuum lag on that table climbing week over week, even after you've tuned autovacuum specifically for it?
• Search: does your checkout p99 measurably get worse during the minutes your heaviest search queries run?
• Event log: is replica lag spiking every time a downstream consumer catches up?

There's a sharper version of that last one worth knowing about, too. If a downstream consumer (say, Debezium relaying to Kafka) goes down entirely, the logical replication slot doesn't just quietly fall behind - it holds onto every WAL segment since the last one that consumer actually read, because it has no way of knowing when it's coming back: could be ten minutes, could be dead for weeks. Those segments pile up on disk, and left alone, they will fill it, quietly, well before anyone notices the consumer is gone at all - so it's worth monitoring this one too.

Any one of those signals, with a real number behind it, is a very strong argument for peeling that one feature out - not all five at once, just the one that's actually hurting you, and only onto the system built specifically for that job.

And because you started with plain SQL and a couple of extensions rather than a bespoke integration, that move is a data pipeline problem, not a rewrite. You already know the exact shape of the data. You're just giving it a better home once it's earned one. That's the two paths meeting in the middle - except this way, you only ever paid for the part of the five-box path you actually ended up needing.

Start with one

So, which box do you build first? Start with one. Add a specialized system when a specific, measured cost tells you to - not when a diagram tells you to, and not when a conference talk tells you to.

The five-box architecture was never wrong, exactly. It's just an answer to a question nobody's asked you yet. Six months from now, if that vacuum lag actually starts climbing, that's the day to open a ticket for Redis. Not before.