pg_durable: Microsoft open sources in-database durable execution

levkk

2026 is the year of the Postgres queue! (DBOS[0], pgQue[1]) It's awesome that the community is contributing this and giving us the option to use it. As an ex-app engineer though, I kind of prefer my queue logic to be in code, in Git, but maybe with the right tooling, you can change my mind. :) [0]: https://www.dbos.dev/ [1]: https://github.com/NikolayS/pgque

faxmeyourcode

This feels like the wrong solution to an age old problem solved by the DAG schedulers like Apache Airflow for a while now. Why would I want to store my control flow in the database and not in code? It feels strange. Not trying to dismiss the project, I'm just not getting it yet I think.

kilobaud

> When not to use it > … > The workflow mostly lives outside Postgres and spans many heterogeneous systems. How is this project at all comparable to something like Temporal? Am I misunderstanding the limitation implied by this particular recommendation?

cpursley

Looks pretty good but I wonder why they didn’t build it on pgmq? If you’re on elixir I maintain a DAG package around this (based on and compatible with pgflow.dev which is TS/Deno). https://github.com/agoodway/pgflow

jraedisch

If understanding correctly, Absurd (by the Pi LLM harness devs) minimizes the pure db approach as much as possible. I only just started getting into the topic myself, though. https://github.com/earendil-works/absurd

oa335

Can anyone explain why I would want to use this over an orchestration tool that lives outside the DB? Read through the Readme and some of the examples, I still don't get it.

rastignack

I hope it could be used in the future to export pg_dump formated exports to s3. One would be able to trigger maintenance jobs via simple lambda functions whose duration is capped.

mikey_p

Is this an open sourcing of something they use internally? My first thought on durable jobs was GHA aka Azure Devops.

redmonduser

Seems like an interesting idea to add durability and resumability to lengthy cron jobs.

TuringNYC

I'm trapped on Azure at work and we're constantly waiting for Azure pg to catch up with modernity. For example, you cant use this: https://www.paradedb.com/blog/hybrid-search-in-postgresql-th... Also for example, you dont get ultra-wide high dimensionality vectors. It is nice they are open sourcing pg_durable, but how about adopting table stakes I'd get with AWS?

joelthelion

Isn't the database already one of the hardest piece of infras to scale? Why would you want to load it with additional long-running jobs?

CharlieDigital

A few things are not clear to me from reading through docs and examples: df.wait_for_schedule() How does this call work? Is it idempotent if I call it from an application? If I run it 2x with the same parameters, does it double tick? Am I invoking this manually from a query console to only do this one time? Am I running this as part of a migration script? For this[0]: -- Wait for human signal (5 minute timeout) ~> (df.wait_for_signal('approval', 300) |=> 'sig') ~> df.if( $$SELECT NOT ($sig::jsonb->>'timed_out')::boolean AND ($sig::jsonb->'data'->>'approved')::boolean$$, Is the `timed_out` a fixed constant that is returned on timeout? Also not immediately clear: how to handle errors/exceptions? [0] https://github.com/microsoft/pg_durable/blob/main/examples/i...

steno132

I would argue that for all but the largest tech companies you only need a single data system which is Postgres. Message brokers, analytical databases all can be built on Postgres. Unfortunately, Postgres as it's built now lacks any semblence of extensibility which makes this impossible in practice. I would propose a rewrite of Postgres in another language like Rust, introducing a pluggable application layer on top. While ambitious in scope I think it would be helpful and even necessary.

linuxhiker

Hopefully they will start sponsoring PGRX now that they are so publicly using it.

7373737373

Feels like perhaps yet another https://en.wikipedia.org/wiki/Inner-platform_effect that would be unnecessary if popular programming languages/virtual machines already supported determinism, metered and controllable stepwise execution and runtime state suspension, (de)serialization and resumption?

ijustlovemath

We made a very functional job queue in Postgres with PostgREST. highly recommend, as the automatic REST API makes building new clients a breeze

junto

This smells like stored procedures. You can’t unit test it. You can’t version it. Business logic in the database, (hidden brain problem), harder to isolate noisy workloads, no observability, scaling pressure lands solely in Postgres, lack of IO, especially API calls. Good for local database only jobs though. Niche use cases.

fragmede

What's the one GitHub uses? Because I may not be GitHub scale, but it seems to be having problems.