Twisted Web Proxy

I have been running this code (from: http://blog.somethingaboutcode.com/?p=155 ):

from twisted.internet import reactor
from twisted.web import http
from twisted.web.proxy import Proxy, ProxyRequest, ProxyClientFactory, ProxyClient
from ImageFile import Parser
from StringIO import StringIO

class InterceptingProxyClient(ProxyClient):
    def __init__(self, *args, **kwargs):
        ProxyClient.__init__(self, *args, **kwargs)
        self.image_parser = None

    def handleHeader(self, key, value):
        if key == "Content-Type" and value in ["image/jpeg", "image/gif", "image/png"]:
            self.image_parser = Parser()
        if key == "Content-Length" and self.image_parser:
            pass
        else:
            ProxyClient.handleHeader(self, key, value)

    def handleEndHeaders(self):
        if self.image_parser:
            pass #Need to calculate and send Content-Length first
        else:
            ProxyClient.handleEndHeaders(self)

    def handleResponsePart(self, buffer):
        print buffer
        if self.image_parser:
            self.image_parser.feed(buffer)
        else:
            ProxyClient.handleResponsePart(self, buffer)

    def handleResponseEnd(self):
        if self.image_parser:
            image = self.image_parser.close()
            try:
                format = image.format
                image = image.rotate(180)
                s = StringIO()
                image.save(s, format)
                buffer = s.getvalue()
            except:
                buffer = ""
            ProxyClient.handleHeader(self, "Content-Length", len(buffer))
            ProxyClient.handleEndHeaders(self)
            ProxyClient.handleResponsePart(self, buffer)
        ProxyClient.handleResponseEnd(self)

class InterceptingProxyClientFactory(ProxyClientFactory):
    protocol = InterceptingProxyClient

class InterceptingProxyRequest(ProxyRequest):
    protocols = {'http': InterceptingProxyClientFactory}
    ports = {"http" : 80}

class InterceptingProxy(Proxy):
    requestFactory = InterceptingProxyRequest

factory = http.HTTPFactory()
factory.protocol = InterceptingProxy

reactor.listenTCP(8000, factory)
reactor.run()

Whenever I get this and go to 127.0.0.1:8000 I get this:

Traceback (most recent call last):
  File "C:\Program Files\Python 2.6.2\lib\site-packages\twisted\python\log.py",
line 84, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "C:\Program Files\Python 2.6.2\lib\site-packages\twisted\python\log.py",
line 69, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "C:\Program Files\Python 2.6.2\lib\site-packages\twisted\python\context.p
y", line 59, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "C:\Program Files\Python 2.6.2\lib\site-packages\twisted\python\context.p
y", line 37, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "C:\Program Files\Python 2.6.2\lib\site-packages\twisted\internet\selectr
eactor.py", line 146, in _doReadOrWrite
    why = getattr(selectable, method)()
  File "C:\Program Files\Python 2.6.2\lib\site-packages\twisted\internet\tcp.py"
, line 460, in doRead
    return self.protocol.dataReceived(data)
  File "C:\Program Files\Python 2.6.2\lib\site-packages\twisted\protocols\basic.
py", line 251, in dataReceived
    why = self.lineReceived(line)
  File "C:\Program Files\Python 2.6.2\lib\site-packages\twisted\web\http.py", li
ne 1573, in lineReceived
    self.allContentReceived()
  File "C:\Program Files\Python 2.6.2\lib\site-packages\twisted\web\http.py", li
ne 1641, in allContentReceived
    req.requestReceived(command, path, version)
  File "C:\Program Files\Python 2.6.2\lib\site-packages\twis开发者_高级运维ted\web\http.py", li
ne 807, in requestReceived
    self.process()
  File "C:\Program Files\Python 2.6.2\lib\site-packages\twisted\web\proxy.py", l
ine 147, in process
    port = self.ports[protocol]
exceptions.KeyError: ''

Whenever I setup firefox or chrome or opera to use the proxy on localhost:8000 there are no connections made to the proxy (and I can no longer connect to any page, though that is probably because it isn't connection to the proxy).

Ok it still fails and with logging I get this output when I set firefox to use the proxy at localhost:8000 and don't visit the proxy directly from the web browser (such as by typing localhost:8000 in firefox's address bar)

2010-08-04 12:31:18-0400 [-] Log opened.
2010-08-04 12:31:29-0400 [-] twisted.web.http.HTTPFactory starting on 8000
2010-08-04 12:31:29-0400 [-] Starting factory <twisted.web.http.HTTPFactory inst
ance at 0x010B3EE0>
2010-08-04 12:33:55-0400 [-] Received SIGINT, shutting down.
2010-08-04 12:33:55-0400 [twisted.web.http.HTTPFactory] (Port 8000 Closed)
2010-08-04 12:33:55-0400 [twisted.web.http.HTTPFactory] Stopping factory <twiste
d.web.http.HTTPFactory instance at 0x010B3EE0>
2010-08-04 12:33:55-0400 [-] Main loop terminated.

However when I do visit the proxy directly I get the key error.

Also for sniffing I can't; Wireshark doesn't seem to sniff the localhost traffic and if I use fiddler 2 it sets itself as the proxy (and so I am no longer using my proxy server) and then works (because it uses fiddler 2's proxy).

The KeyError exception you see when you connect directly is caused by the fact that requests to a proxy must include an absolute URL, not a relative one. If your browser doesn't know it's talking to a proxy, it will request a URL like /foo/bar. If it does know it is talking to a proxy, it will instead request something like http://example.com/foo/bar. The http://example.com/ part is important because it's the only way the proxy knows what it's supposed to go off and retrieve.

As for why none of Firefox, Chrome, nor Opera will connect to the proxy once you configure them to do so, that's a little bit harder to explain. Make sure you're configuring an "HTTP Proxy", not any of the other kinds of proxy supported. Once you've double checked that, you might want to use a tool like Wireshark to get a better look at what's happening on the network layer.

It's possible that the connections are really being made to the proxy but something else is going wrong that prevents them from completing. In this case, without logging enabled, you might not be able to tell that the proxy is receiving connections simply by looking at its output. To enable logging, try:

from sys import stdout
from twisted.python.log import startLogging
startLogging(stdout)