Multi-threading and proxy adds 30 seconds delay. What can I do to avoid

I am usign PyQt, QThreads and couchdb-python to talk to a couchDB instance on the local LAN.

To test that the multithreading will work if the networking is slow, I have placed a proxy between the GUI and couch.

The networking is logged. The log starts off with

13:52:46.303 (1) send: HEAD /cubic HTTP/1.1CRLFHost: localhost:5981CRLFContent-Length: 0CRLFAccept: application/jsonCRLFUser-Agent: CouchDB-Python/0.8CRLFCRLF

13:52:46.308 (1) recv: HTTP/1.1 200 OKCRLFServer: CouchDB/1.0.1 (Erlang OTP/R13B)CRLFDate: Mon, 21 Feb 2011 13:47:18 GMTCRLFContent-Type: application/jsonCRLFContent-Length: 219CRLFCache-Control: must-revalidateCRLFCRLF
13:53:16.312 (1) recv: link closed
13:53:16.319 (2) send: GET /cubic/_design/Company/_view/by_name HTTP/1.1CRLFHost: localhost:5981CRLFContent-Length: 0CRLFAccept: application/jsonCRLFUser-Agent: CouchDB-Python/0.8CRLFCRLF
13:53:16.330 (2) recv: HTTP/1.1 200 OKCRLFTransfer-Encoding: chunkedCRLFServer: CouchDB/1.0.1 (Erlang OTP/R13B)CRLFEtag: "243QGZGN1ETGA4VN9H1OMB86Z"CRLFDate: Mon, 21 Feb 2011 13:47:48 GMTCRLFContent-Type: application/jsonCRLFCache-Control: must-revalidateCRLFCRLF
13:53:16.331 (2) recv: a0CRLF{"total_rows":9,"offset":0,"rows":[CRLF{"id":"c182c1a2f71c3547ccee45556300770e","key":["A first Company","231","345 East of Eden" ,"Manchester",null],"value":null}CRLF
13:53:16.332 (2) recv: 77CRLF,CRLF{"id":"c182c1a2f71c3547ccee455563001254","key":["C-U-B","1","9-11 March Business Centre","March",null],"value":null}CRLF72CRLF,CRLF{"id":"421261e8a3311c356c4900185800d976","key":"Four","104","4 Fourth street","Four Score",null],"value":null}CRLF74CRLF,CRLF{"id":"c182c1a2f71c3547ccee455563003de8","key":["Fred Smith","431","30 High street","Somehow",null],"value":null}CRLF
13:53:16.334 (2) recv: 83CRLF,CRLF{"id":"7bc7f2014593d108b5b681fe03002ad6","key":["Ian Hobson","1002","31 S开发者_运维百科heerwater Dr","Northampton, Nhants",null],"value":null}CRLF76CRLF,CRLF{"id":"c182c1a2f71c3547ccee455563003139","key":["Julie Bloggs","302","30 Back Lane","Somewhere",null],"value":null}CRLF7cCRLF,CRLF{"id":"c182c1a2f71c3547ccee455563002e87","key": "Kingfisher Group","323","33 Westmister Rd","Londonx",null],"value":null}CRLF
13:53:16.335 (2) recv: 7aCRLF,CRLF{"id":"c182c1a2f71c3547ccee455563005894","key":["New Company","212","34 Back Street","Another Town",null],"value":null}CRLF72CRLF,CRLF{"id":"421261e8a3311c356c4900185800d7e2","key":["Three","103","3 High Street","Liverpool 8",null],"value":null}CRLF4CRLFCRLF]}CRLF1CRLFLFCRLF0CRLFCRLF
13:53:46.337 (2) recv: link closed

That is exactly as expected except that there is a 30 second delay after the database has been opened at 13:52:46.308, while the recv link times out. Not until this happens can the code open the view which happens at 13:53:16.319

However the proxy is using a different thread {(2) not (1)}. The application part works just fine without the proxy, so I suspect the proxy.

The proxy code is based on microproxy and is below

import re, sys, datetime
import socket 
import threading 
PORT = 5981
class ConnectionThread(threading.Thread): 
    def __init__(self, (conn,addr),id): 
        self.conn = conn 
        self.addr = addr
        self.id = id
        threading.Thread.__init__(self)

    def report(self,data,dir):
        txt = data.replace("\n",'LF')
        txt = txt.replace("\r",'CR')
        now = datetime.datetime.now()
        print "%s (%s) %s: %s" % (now.isoformat()[11:23],self.id,dir,txt)      

    def run(self): 
        data = self.conn.recv(1024*1024) 
        request = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
        request.connect(('192.168.0.1',5984)) 
        self.report(data,'send')
        request.send(data) 
        while True: 
            temp = request.recv(1024) 
            if ('' == temp):
                self.report('link closed','recv')
                break 
            self.report(temp,'recv')
            self.conn.send(temp)
        self.conn.close()

class ProxyThread(threading.Thread): 
    def __init__(self, port): 
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
        self.sock.bind(('localhost', port))
        self.count = 1
        threading.Thread.__init__(self) 
    def run(self): 
        self.sock.listen(3) 
        while True:
            temp = ConnectionThread(self.sock.accept(),self.count)
            temp.start()   # each message handled in own thread
            self.count += 1
if __name__ == "__main__": 
    print "Starting a proxy on port", PORT
    proxy = ProxyThread(PORT) 
    proxy.run() 

Both the request to open the database and the request for the view are made in the same (non GUI) thread. Why should the delay occur?

And how can I change things so it doesn't?

Your loop reads only from one end of the connection being proxied. So that when the reply is forwarded from the server to the client and client disconnects, the code does not notice client disconnection and keeps waiting for server reply or disconnect.

To fix that it needs to read from both server (request) and client (self.conn) sockets. It is easiest to do that by using non-blocking sockets.

The HTTP spec states (section 9.4)

The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response.

This means that there is nothing following the CRLFCRLF but your proxy is waiting for data rather than reading from the incoming connection. I suspect that the couchdb library is trying to use a keep-alive connection, but you're watching the wrong socket.

Your basic problem is that you claim to speak to HTTP 1.1 but you don't. You should watch for a CRLFCRLF sequence a content-size header. If you find one read that much data then break out of the loop, if there isn't one then break immediately.