Postgres LISTEN/NOTIFY actually scales
Posted by KraftyOne 6 hours ago
Comments
Comment by jerf 6 hours ago
The ceiling of LISTEN/NOTIFY is small enough that you need to pay attention, and I personally like to have at least an order of magnitude of slack left over even after my most pessimistic load numbers are accounted for, but it's still plenty for a lot of projects, and the integration with the rest of the DB, its availability, its not being another service you have to devops, it's definitely not something that should be simply dismissed out of hand as an option. Even the original 2K/s number they cite is a lot of messages for some systems that are more properly measured in seconds per message.
Comment by mlyle 3 hours ago
IMO one should design for actual anticipated scale with moderate margin, only exceeding this when it's relatively "free" to do so. (If you can buy bigger hardware for a few K, or if solutions are equivalent other than scalability, pick the bigger solution).
Comment by zbentley 6 hours ago
Nitpick on an otherwise good post, but I don’t think there are very many 6billion RPS systems out there, and those that do exist are almost certainly using bespoke, purpose-built tools
Comment by odo1242 5 hours ago
Comment by znpy 3 hours ago
Comment by 10000truths 3 hours ago
Comment by monster_truck 3 hours ago
Comment by ndriscoll 3 hours ago
Comment by jerf 5 hours ago
Comment by nzoschke 6 hours ago
Once you start down the "durable workflows" path, you start seeing them everywhere.
My latest experiments are treating individual emails as durable workflows, where you, the people you're communicating with, agents and tools like GitHub or Attio all take turns in the flow.
https://housecat.com/blog/gmail-durable-workflows-sandbox-vm
Comment by dang 3 hours ago
Postgres LISTEN/NOTIFY does not scale - https://news.ycombinator.com/item?id=44490510 - July 2025 (321 comments)
Comment by vhiremath4 1 hour ago
In practice, this eventually ended up being very awkward because extending the functionality (since we had "built" it) and had to work around internal pg semantics (we should have just moved off much sooner). It also did not scale well. We ended up getting a ton of disk contention on our RDS instance in non-obvious ways, and the vacuum runs on that table was a nightmare. Additionally, it was hard to get other engineers to really debug and take ownership of the system because they automatically viewed a queue (very easy to understand) implemented in a foreign way (off pg internals) as something "scary". It was emotional, not rational, but we are emotional beings, and I do not blame them. These were good engineers with a lot of other things on their plates.
Obviously, this is all hand-wavy without discussing the internal schema, indeces, etc. that we had set up, but my main takeaway with core technology from this experience was to always reach for the dumb, expected, simple thing. Even if it adds another moving piece in the infra stack. Unless I need very strong data consistency guarantees, it's always better to use something like SQS, Redis queues, etc. where the understanding is that it is just a queue (or at least the API contract suggests simplicity), and then everything needs to work around it.
The fewer mechanistic responsibilities per core data store, the better in my experience.
Comment by hmaxdml 48 minutes ago
It seems to me this can be boiled down to "using things without understanding how they work doesn't scale". Yes, vanilla listen/notify doesn't scale. But the OP actually figured out how to make it scale. So your engineers don't have to.
As the community builds, over time, distributed systems, it also understands which brick can do what, and it turns out that starting with less bricks and adding some when you actually need them makes for healthier systems.
Comment by vhiremath4 43 minutes ago
Comment by zmmmmm 53 minutes ago
Comment by vhiremath4 46 minutes ago
> It is so rare that a new moving part beats other factors
Fair! I will restate that we were growing very fast, so that's probably an outlier environmental factor that I am potentially overly discounting. If that's not the case (likely for most startups), then maybe this is less of a problem.
Comment by dietr1ch 6 hours ago
Since the correction apparently dates from May 8th, I think that a post from July 24th might want to acknowledge that the popular post asserting this feature doesn't (didn't?) scale was not made in bad faith or was even wrong about their claims at the time.
Comment by KraftyOne 5 hours ago
> As an aside, there’s been some online discussion of a Postgres patch (https://github.com/postgres/postgres/commit/282b1cde9dedf456...) related to this issue. This patch (to be released in Postgres 19) does not remove the global lock or fix the bottleneck we observed. Instead, it optimizes the narrower case where there are many notification channels and each listener is waiting only on a specific channel.
Comment by sandeepkd 5 hours ago
1. What I find interesting is that the experiment seems to be using a DB server with 96 cores, 384 GB RAM (https://github.com/dbos-inc/dbos-postgres-benchmark/blob/mai...). This is very critical part of any such experiment, it should have been called out. The database is vertically scalable and that too has its limits
2. Who is making connection, and from where has its own impact on performance and overall latency
3. 60k may seem big number, however in real world the things which bring the systems down are the bursts of traffic, not the regular traffic.
Personally I would never start with such a big server unless I am a big business. Its > 100K cost for one production DB cluster if I include read replicas and cross region redundancy
Comment by Latty 6 hours ago
Comment by dietr1ch 6 hours ago
- Keeps messages O(1) so I can focus on scaling in the amount of notifications
- Tells whoever runs into this that,
- I didn't planned for arbitrarily large messages as they *might* otherwise grind performance to a halt.
- They *might* be misusing my notification systemComment by newswangerd 3 hours ago
In the end, I decided to just go with the simplest solution possible. In my case it's just a barebones Go gRPC service that uses an in memory channel to send notifications between connected clients.
The reality is that this simple Go server will scale up to about 1000 simultaneously connected customers on about 2gb of RAM. I don't expect to have more than that many paying customers, and if I do I can always just throw a bigger VM at the problem.
Engineers love to over complicate things in the name of infinite scalability, when in reality you can save a lot of time and effort by just understanding the scope of the actual problem you're trying to solve. Fingers crossed that this will become an issue for me some day, but until then most of us just don't need to worry about it!
Comment by anachronox 3 hours ago
Comment by znpy 3 hours ago
The real questions are:
- how many messages per second are you processing on that 2gb machine (and using how many cpus)?
- does your message processing involve transaction handling, including saving data ti disk durably?
No offense but it really seems you’re comparing apples and oranges, with your use case being much much simpler than the one described.
Comment by newswangerd 26 minutes ago
My point here is that it is important to match the tech stack to the challenge you're facing. When I started thinking about how to solve this problem my first reaction was to design an overly complicated distributed message queue using PG Notify, Redis, Kafka or something along those lines. The key takeaway here is that I realized that I probably wouldn't end up with more than 1000 customers, so I just needed to design a system that could comfortably handle that level of traffic without much effort. If, by some miracle, my business goes crazy viral, I know that my cloud provider can probably handle up to 200,000 customers by just updating a slider in my dashboard, which is way more business than I want anyway.
Engineers love to fantasize about Google levels of scale, but that's just not realistic for a lot of services.
Comment by b-man 58 minutes ago
Comment by d9127acbd6fe281 4 hours ago
Comment by qphe95 2 hours ago
Comment by tjadfsaj 5 hours ago
The word "scale" does a lot of load-bearing, maybe it's just not a useful or productive word in practice.
Comment by acaloiar 5 hours ago
Choose database queue technology https://news.ycombinator.com/item?id=37636841
Comment by hxtk 41 minutes ago
Comment by mamcx 5 hours ago
Comment by KraftyOne 5 hours ago
Comment by mamcx 5 hours ago
Comment by gordonhart 1 hour ago
Comment by jrochkind1 5 hours ago
It does seem interesting, and possibly welcome if there were a configuration option or even a way to set individual notifies as serialized or not.
Comment by KraftyOne 5 hours ago
Comment by abratabia 4 hours ago
Comment by David_runai 5 hours ago
Comment by luciana1u 2 hours ago
Comment by zenith605 5 hours ago