Concurrent Shell Command Queues
speckx
42 points
17 comments
November 13, 2026
Related Discussions
Found 5 related stories in 129.0ms across 12,728 title embeddings via pgvector HNSW
- Show HN: A local merge queue for parallel Claude Code agents funador · 26 pts · July 30, 2026 · 49% similar
- Terminal2Shell mritzmann · 51 pts · November 09, 2026 · 46% similar
- Making Postgres queues scale KraftyOne · 110 pts · July 30, 2026 · 45% similar
- Go Concurrency Distilled chmaynard · 139 pts · September 26, 2026 · 44% similar
- I wrote an bash enumerator because I was sick of xargs wallach-game · 103 pts · July 20, 2026 · 44% similar
Discussion Highlights (8 comments)
kernc
> language with poor concurrency controls Few other languages have as many builtins/keywords for concurrency control: &, wait, jobs ...
js2
> How to run shell commands in parallel while limiting their concurrency. Depending upon the task, generate a Makefile and run it with: make -j <concurrency> > My original use case had to do with making a large number of web requests This is in fact the use case that caused me to think about using a Makefile for concurrency probably 20 years ago. A rule for downloading each URL to a file. Then a simple `make -j <jobs>` with whatever concurrency I desired and bonus: automatically skipping already downloaded files. e.g.: BASE := https://example.com/files FILES := a.txt b.txt c.txt d.txt e.txt .DELETE_ON_ERROR: all: $(FILES) $(FILES): curl -sS --fail --max-time 30 -o $@ $(BASE)/$@ Or, to emulate one of the examples in the blog post: SHELL := bash .SHELLFLAGS := -o pipefail -c N := 1 2 3 4 5 REQS := $(N:%=req-%) .PHONY: all $(REQS) all: $(REQS) $(REQS): req-%: @curl -sS --fail-with-body --max-time 10 -X POST -H 'Content-Type: application/json' \ -d '{"request": $*}' https://postman-echo.com/post | jq -c .json Run with: $ make -O -j 3 > You do not need to know all the commands that you want executed up-front, you can submit commands to it over time. Ah, yeah, that does put Make in the same bucket as `xargs -P` / GNU `parallel` and requires something more clever.
tacostakohashi
I guess that works. I've always been fond of using make for things to get free parallelism / ability to interrupt / restart / incremental updates.
TacticalCoder
I shamelessly use concurrency in Bash shells but typically I do it because I already have a Bash shell and it's now outgrowing its purpose and becoming to slow (say because I begin to use on too much data). So instead of rewriting everything to another language, I add concurrency to that already existing shell scripts. xargs / max args / forking to "bash -c" etc. It's tricky. There are gotchas (with piping for example but also how to behave when one subshell fails, how to cleanly interrupt everything if one fails, etc.). But it works and it's a cheap way to greatly speed things up. Also that's the kind of stuff that became not just a bit but much easier now: just give a shell script to a LLM and ask it to add concurrency where needed.
reconquestio
There is also GNU Parallel's sem: https://www.gnu.org/software/parallel/sem.html sem --id jobs -j 2 'sleep 1; echo FIRST' sem --id jobs -j 2 'echo SECOND' sem --id jobs -j 2 'echo THIRD' sem --id jobs --wait SECOND THIRD FIRST
hoppp
What about using GNU Parallel ?
0cf8612b2e1e
I really like pueue for this. Like a light weight slurm or parallel without the nagging citation notice. https://github.com/Nukesor/pueue
partdavid
Built-in to Powershell, of course.