Mizu – zero-copy parallel R and Python over shared memory

shikokuchuo

Author here: I wrote mizu (R) and pymizu (Python), packages that offer lock-free async and parallel computing over shared memory, built on the same C core, libmizu. The link above is for the Python package. A task round trip drops to under one microsecond, from around 100 µs over sockets with Python's ProcessPoolExecutor or mirai, my other R framework. These are numbers from my laptop (an M4 Pro), and benchmark scripts are in the repos for you to run. Workers claim tasks off lock-free queues, there is no task dispatcher, and system calls are avoided on the hot path. `PoolExecutor` is a drop-in `concurrent.futures.Executor`, tasks are awaitable in asyncio, NumPy arrays and polars, pyarrow and pandas frames cross as zero-copy views. As mizu and pymizu share the same wire format, data frames and Arrow types also cross zero-copy between worker pools of different languages. Of course, this is single-machine only as it uses shared memory. The API is still experimental. Usability feedback and bug reports are most welcome at the repos. I'd love to hear about places you'd like to see this used. Write-up with full benchmarks and the R/Python code examples: https://opensource.posit.co/blog/2026-11-05_mizu-0-1-0/