1

I have a bunch of jobs to submit to a PBS queue and the output is supposed to be post-processed locally afterwards. As the jobs are similar (and I don't want to manually change it that often), I wrote myself a bash script that locally generates a PBS job script and then submits the job. My question is the following: How can I make my bash script wait until the PBS job has finished and then start post-processing? (Constantly checking the status of the job is not an option.) What possibilities do I have to do this?

Thank you very much to all hints, tips or solutions!

Edit: As indicated in the comment (thanks), some edits:

  • PBS = Portable Batch System
  • Documentation is available here: http://resources.altair.com/pbs/documentation/support/PBSProUserGuide12.1.pdf
  • I thought about possible solutions and came up with the following one that won't work, though: I could my bash script wait for the amount of time that is requested as wall clock time of the job. However, there are two complications: firstly, the job might not need that much time and terminate earlier, hence having the bash script wait too long; secondly, the job might be queued and not immediately executed, hence having the bash script likely wait too little.

Edit 2: As I am not sure if I can always run interactively as suggested below, I thought about the following: I have the PBS job script create an (empty) file just before finishing. My bash script checks every couple minutes (say) whether that file exists (using a while loop). If it exists, I know that the job terminated and can start post-processing by deleting the just created (empty) file. This way, I only have a delay of a couple of minutes maximum and don't really need any resources. It is definitely not ideal, but kind of a workaround. Any ideas that are more practical are highly appreciated.

Any thought is appreciated even if not directly related to PBS.

Cari

Improve this question
1
  • Consider adding a link to documentation for pbs ( in the US that means Public Broadcasting System). Notice that there are only 18 followers for pbs, so this probably not a widely understood technology. That said, S.O. isn't a free coding service, you're expected to have made an attempt at solving your problem and to have posted that code (correctly formatted) in your question. Good luck. Commented Aug 22, 2013 at 15:00

2 Answers 2

Reset to default
0

I don't believe PBSPro supports this, but TORQUE (another PBS derivative) has a -x option that you might be interested in. You can submit a job like this:

qsub -I -x <executable>

This would run your job interactively and run the executable, with all of the output directed to your terminal, and the job will execute as soon as that executable terminates. You could then begin post-processing at that point. PBSPro may have similar functionality, but what I've described here is for TORQUE.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thank you! This sounds to be very useful not only for the above problem. However, I am not sure if I can always run the job interactively. I will definitely have a look into it!
0

This is related to an answer I wrote to a different question. I have access to a few machines at my campus high performance computing center. On at least one of the clusters, it is permitted to call qsub from a parent PBS script. This allows for job chaining as described on this page. So, you may be able to use a solution like the following skeleton:

# ...do your PBS stuff
# launch your code on the cluster that creates, say "JOBID.txt"
parallel -j0 mycode.sh

# you may need to use PBS environment variables to figure out what your output file is called
filename=$${PBS_JOBID%%.*}.txt # e.g., this would be like the JOBID.txt I wrote above

# next, use qsub to launch the post-processing script
echo "${PBS_O_WORKDIR}/postprocess.pbs $filename" | qsub

posprocess.pbs would take the filename to process as an argument. In the answer I linked to, this worked for me to copy PBS standard output file to another file upon finishing. However, I'm not sure about possible race conditions, and you may need to use the -W depend= qsub option, something like this (not tested):

echo "-W depend=afterok:$PBS_JOBID postprocess.pbs $filename" | qsub
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.