How to figure out why ssh session does not exit sometimes?

I have a C++ application that uses ssh to summon a connection to the server. I find that sometimes the ssh session is left lying around long after the command to summon the ser开发者_StackOverflow中文版ver has exited. Looking at the Centos4 man page for ssh I see the following:

 The session terminates when the command or shell on the remote machine
 exits and all X11 and TCP/IP connections have been closed.  The exit
 status of the remote program is returned as the exit status of ssh.

I see that the command has exited, so I imagine not all the X11 and TCP/IP connections have been closed. How can I figure out which of these ssh is waiting for so that I can fix my summon command's C++ application to clean up whatever is being left behind that keeps the ssh open.

I wonder why this failure only occurs some of the time and not on every invocation? It seems to occur approximately 50% of the time. What could my C++ application be leaving around to trigger this?

More background: The server is a daemon, when launched, it forks and the parent exits, leaving the child running. The client summons by using:

popen("ssh -n -o ConnectTimeout=300 user@host \"sererApp argsHere\""
      " 2>&1 < /dev/null", "r")

Use libssh or libssh2, rather than calling popen(3) from C only to invoke ssh(1) which itself is another C program. If you want my personal experience, I'd say try libssh2 - I've used it in a C++ program and it works.

I find some hints here:

http://www.snailbook.com/faq/background-jobs.auto.html

This problem is usually due to a feature of the OpenSSH server. When writing an SSH server, you have to answer the question, "When should the server close the SSH connection?" The obvious answer might seem to be: close it when the server-side user program started by client request (shell or remote command) exits. However, it's actually a bit more complicated; this simple strategy allows a race condition which can cause data loss (see the explanation below). To avoid this problem, sshd instead waits until it encounters end-of-file (eof) on the pipes connecting to the stdout and stderr of the user program.

@sienkiew: If you really want to execute a command or script via ssh and exit, have a look at the daemontool of the libslack package. (Similar tools that can detach a command from its standard streams would be screen, tmux or detach.)

To inspect stdin, stdout & stderr of the command executed via ssh on the command line, you can, for example, use lsof.

# sample code to figure out why ssh session does not exit
# sleep keeps its stdout open, so sshd only sees EOF after command completion
ssh localhost 'sleep 10 &'        # blocks
ssh localhost 'sleep 10 1>&- &'   # does not block

ssh localhost 'sleep 10 & lsof -p ${!}'
ssh localhost 'sleep 10 1>&- & lsof -p ${!}'
ssh localhost 'sleep 10 1>/dev/null & lsof -p ${!}'
ssh localhost 'sleep 10 1>/dev/null 2>&1 & lsof -p ${!}'