UNIX named PIPE end of file

I'm trying to use a unix named pipe to output statistics of a running service. 开发者_StackOverflowI intend to provide a similar interface as /proc where one can see live stats by catting a file.

I'm using a code similar to this in my python code:

while True:
  f = open('/tmp/readstatshere', 'w')
  f.write('some interesting stats\n')
  f.close()

/tmp/readstatshere is a named pipe created by mknod.

I then cat it to see the stats:

$ cat /tmp/readstatshere
some interesting stats

It works fine most of the time. However, if I cat the entry several times in quick successions, sometimes I get multiple lines of some interesting stats instead of one. Once or twice, it has even gone into an infinite loop printing that line forever until I killed it. The only fix that I've got so far is to put a delay of let's say 500ms after f.close() to prevent this issue.

I'd like to know why exactly this happens and if there is a better way of dealing with it.

Thanks in advance

A pipe is simply the wrong solution here. If you want to present a consistent snapshot of the internal state of your process, write that to a temporary file and then rename it to the "public" name. This will prevent all issues that can arise from other processes reading the state while you're updating it. Also, do NOT do that in a busy loop, but ideally in a thread that sleeps for at least one second between updates.

What about a UNIX socket instead of a pipe?

In this case, you can react on each connect by providing fresh data just in time.

The only downside is that you cannot cat the data; you'll have to create a new socket handle and connect() to the socket file.

MYSOCKETFILE = '/tmp/mysocket'
import socket
import os
try:
    os.unlink(MYSOCKETFILE)
except OSError: pass
s = socket.socket(socket.AF_UNIX)
s.bind(MYSOCKETFILE)
s.listen(10)
while True:
    s2, peeraddr = s.accept()
    s2.send('These are my actual data')
    s2.close()

Program querying this socket:

MYSOCKETFILE = '/tmp/mysocket'
import socket
import os
s = socket.socket(socket.AF_UNIX)
s.connect(MYSOCKETFILE)
while True:
    d = s.recv(100)
    if not d: break
    print d
s.close()

I think you should use fuse. it has python bindings, see http://pypi.python.org/pypi/fuse-python/ this allows you to compose answers to questions formulated as posix filesystem system calls

Don't write to an actual file. That's not what /proc does. Procfs presents a virtual (non-disk-backed) filesystem which produces the information you want on demand. You can do the same thing, but it'll be easier if it's not tied to the filesystem. Instead, just run a web service inside your Python program, and keep your statistics in memory. When a request comes in for the stats, formulate them into a nice string and return them. Most of the time you won't need to waste cycles updating a file which may not even be read before the next update.

You need to unlink the pipe after you issue the close. I think this is because there is a race condition where the pipe can be opened for reading again before cat finishes and it thus sees more data and reads it out, leading to multiples of "some interesting stats."

Basically you want something like:

while True:
    os.mkfifo(the_pipe)
    f = open(the_pipe, 'w')
    f.write('some interesting stats')
    f.close()
    os.unlink(the_pipe)

Update 1: call to mkfifo

Update 2: as noted in the comments, there is a race condition in this code as well with multiple consumers.