linux - Run Serial inside Paralell Bash

Question

I have added to my explanation a bit. Conceptually, I am running a script that processes in a loop, calling shells that use the line content as an input parameter.(FYI: a kicks off an execution and b monitors that execution)

1. I am needing 1a and 1b to run first, in paralell for the first two $param
2. Next, 2a and 2b need to run in serial for $params when step 1 is complete
3. 3a and 3b will kick off once 2a and 2b are complete (irrelevant if serial or parallel)
4. Loop continues with next 2 lines from input .txt

I cant get it to process the second in serial, only all in parallel: What I need is the following

cat filename | while readline 
export param=$line
do
./script1a.sh "param" > process.lg && ./script2b.sh > monitor.log &&
##wait for processes to finish, running 2 in parallel in script1.sh
./script2a.sh "param" > process2.log && ./script2b.sh > minitor2.log &&
##run each of the 2 in serial for script2.sh
./script3a.sh && ./script3b.sh

I tried adding in wait, and tried an if statement containing script2a.sh and script2b.sh that would run in serial, but to no avail.

if ((++i % 2 ==0)) then wait fi
done
#only run two lines at a time, then cycle back through loop

How on earth can I get the script2.sh to run in serial as a result of script1 in parallel??

Answer 1

Locking!

If you want to parallelize script1 and script3, but need all invocations of script2 to be serialized, continue to use:

./script1.sh && ./script2.sh && ./script3.sh &

...but modify script2 to grab a lock before it does anything else:

#!/bin/bash
exec 3>.lock2
flock -x 3
# ... continue with script2's business here.

Note that you must not delete the .lock2 file used here, at risk of allowing multiple processes to think they hold the lock concurrently.