PgQue: Zero-Bloat Postgres Queue

cout

I think it's great that projects like this exist where people are building middleware in different ways than others. Still, as someone who routinely uses shared memory queues, the idea of considering a queue built inside a database to be "zero bloat" leaves me scratching my head a bit. I can see why someone would want that, but once person's feature is bloat to someone else.

saberd

I don't understand the latency graph. It says it has 0.25ms consumer latency. Then in the latency tradeof section it says end to end latency is between 1-2 seconds. Is this under heavy load or always? How does this compare to pgmq end to end latency?

odie5533

Postgres durability without having to run Kafka or RabbitMQ clusters seems pretty enticing. May reach for it when I next need an outbox pattern or small fan out.

halfcat

So if I understand this correctly, there are three main approaches: 1. SKIP LOCKED family 2. Partition-based + DROP old partitions (no VACUUM required) 3. TRUNCATE family (PgQue’s approach) And the benefit of PgQue is the failure mode, when a worker gets stuck: - Table grows indefinitely, instead of - VACUUM-starved death spiral And a table growing is easier to reason about operationally?

mind-blight

The vacuum pressure is real. Using a system with the skip locked technique + polling caused massive DB perf issues as the queue depth grew. The query to see the current jobs in the queue ended up being the main performance bottleneck, which cause slower throughput, which caused a larger queue depth, which etc. Scaling the workers sometimes exacerbates the problem because you run into connection limits or polling hammering the DB. I love the idea of pg as a queue, but I'm a more skeptical of it after dealing with it in production

killingtime74

I got Claude to analyze the code and it's not really comparable to SKIP LOCKED queues. It's more like Kafka. There's no job queue semantics with acks, workers taking from same job pool. It's Kafka like one event stream and multiple independent worker cursors. It's more SNS than SQS or Kafka than Rabbitmq/Nats

andrewstuart

Postgres is not the only database that does queues. Any database that supports SKIP LOCKED is fine including MySQL, MSSQL, Oracle etc. Even SQLite makes a fine queue not via skip locked but because writes are atomic.

adhocmobility

Why insist on calling this a queue when it doesn't really have queue semantics? Queues do the job of load balancing between different workers. When workers acknowledge tasks, they get deleted, and there are visibility timeouts. This is a log. It's not really solving the problems you claim it solves. It's not, for instance, a replacement for SKIP LOCKED based queues.

ozgrakkurt

What do you think about trusting something LLM coded with your production data?

wewewedxfgdf

How many message per second does this do I wonder?