Unix shell commands can run in the background. For daemons that run unattended, consult the notes below if the daemon will not disassociate from the invoking terminal. Other options include running the program under a terminal emulator such as dtach or screen(1). Bourne shells such as zsh(1) can disown programs still associated with a session, avoiding the need to kill and relaunch the program under nohup(1).
$ jobs
$ sleep 42 &
[1] 7425
$ jobs
[1] + running sleep 42
$ disown
$ jobs
$ ps au | grep 'slee[p]'
jmates 7425 0.0 0.1 27240 720 p5 SN 6:31PM 0:00.01 sleep 42
Once backgrounded, a tool like retty might restore access to a process.
Waiting on Completion
Use the wait shell builtin to execute an action following the completion of a background job.
$ sleep 30 &
[1] 7425
$ wait && echo fin
[1] + done sleep 30
fin
One use of wait is to alert yourself after a job completes. E-mail can be used to ping or page, depending on the importance of followup work. For pings, setup a mail filter: "on messages from myself, with the word ping in the subject, play a sound and display an alert". Then create a shell script to create a ping e-mail:
#!/bin/sh
# do-ping - Mail client handles ping via filter.
echo "$@" | mail -s ping `id -un`
For pages, either use a subject filter, or send them directly to a pager address.
#!/bin/sh
# do-page - Assumes site standared page address setup.
echo "$@" | mail -s page `id -un`-page
$ a-long-running-process &
[1] 7425
$ wait && do-ping done compressing
Disassociate from Invoking Terminal
Some applications do not disassociate from the invoking terminal. To run these applications as background daemons, be sure to close standard input using <&-. Also redirect standard output and error as appropriate, either to /dev/null or a log file. Another good practice: change the working directory to /, to avoid starting a daemon on a mount point hosted on another system. Look for “Complete Dissociation of Child from Parent” in perlipc for more information on the steps required to daemonize a process.
(
cd / || exit 1
sudo -H -u jboss -- \
/somewhere/jboss/bin/run.sh -c server <&- \
>>/var/log/jboss.console 2>&1 &
)
If a shell is hung exiting because of an existing process waiting on standard input, send that shell process a HUP signal, then check that the poorly behaved process is operating as expected.
Better Output Handling
Be sure to rotate the /var/log/jboss.console log somehow, as otherwise it may exhaust all disk space. If possible, avoid log file rotation via tools like httplog. These utilities capture standard output, and direct this data into filenames created from the current time and other variables, such as the system hostname.
(
cd / || exit 1
sudo -H -u jboss -- \
/somewhere/jboss/bin/run.sh -c server <&- 2>&1 &
) | httplog \
-s /var/log/jboss/console.log \
/var/log/jboss/console/%Y/%m/%Y-%m-%d.log
Another option: replace httplog with logger(1), and handle logs via a syslog(3) implementation such as syslog-ng.
Discovering Flawed Applications
To detect applications that do not disassociate from standard input, use lsof and search for the standard input filehandle, which is typically associated with file descriptor 0.
$ sudo /usr/sbin/lsof | grep 0u | grep -v /dev/null
On a related note, Shell Loop Interaction with SSH details a different problem with standard input handling.
Monitoring
A single process running in the background on a single system is easy enough to check on by hand. Beyond this case, monitoring such as Nagios must be used to ensure the service is up, running, and operating as expected. Monitoring may be as simple as a wrapper script to ensure the process remains running, or a complex suite of remote health checks (service ping, or even a service ping that touches the database) from one or many different test systems.
