shell - Queue using several processes to launch bash jobs

Question

I need to run many (hundreds) commands in shell, but I only want to have a maximum of 4 processes running (from the queue) at once. Each process will last several hours.

When a process finishes I want the next command to be "popped" from the queue and executed.

I also want to be able to add more process after the beginning, and it will be great if I could remove some jobs from the queue, or at least empty the queue.

I have seen solutions using makefile, but this only work if I have all my list of commands before the beginning. Also tried using mkfifo sjobq, and others, but I never could reach my needs...

Does anyone have code to solve this problem?

Edit: In response to Mark Setchell

The solution with tail -f and parallel is almost perfect, but when I do it, it always keep not launching the last 4 commands until I add more, and so on, I don't know why, and it is quite troublesome...

As for Redis, good solution also, but it takes more time to master all of it.

Thanks !

Answer 1

Use GNU Parallel to make a job queue like this:

# Clear out file containing job queue
> jobqueue        

# Start GNU Parallel processing jobs from queue
#   -k   means "keep" output in order
#   -j 4 means run 4 jobs at a time
tail -f jobqueue | parallel -k -j 4

# From another terminal, submit 40 jobs to the queue
for i in {1..40}; do echo "sleep 5;date +'%H:%M:%S Job $i'"; done >> jobqueue

Another option is to use REDIS - see my answer here Run several jobs parallelly and Efficiently