Let's say I have a server listening on port 1234. I have some client software that needs to be able to connect to this port. But I want to prevent malicious users from bypassing the client software and connecting to the port by some other means (such as in a console or another piece of software).

The trusted client software and the server may share a secret key, if necessary. I'd prefer not to send such a key in plaintext, but data after the authentication can be in plaintext. Specifically, I'm trying to figure out how to defeat a man-in-the-middle attack where the malicious user is using the trusted client software to calculate correct responses to server challenges.

Can I get there from here?

I could have the server's listening port bind only to localhost and require that clients first gain access to the machine via ssh. Then the client software could use an ssh library to run a command on the server that connects to the local port (in this scenario, the malicious user would be unable to use ssh to access the machine because he would not have the password). But then all my traffic is encrypted, which is additional overhead. Perhaps there is a program similar to ssh that only does the authentication but then leaves the channel in plaintext after that?

Update:

I ran a test to determine the overhead associated with encrypting all traffic.

spew.rb outputs 10 million 100-character lines.

CONTROL:开发者_运维百科

fantius@machine> time /home/fantius/spew.rb > /dev/null

real    0m35.015s
user    0m34.934s
sys     0m0.084s

top shows 25% cpu usage (one full core, of four cores)

TEST:

fantius@machine> time ssh localhost /home/fantius/spew.rb > /dev/null

real    0m40.704s
user    0m19.981s
sys     0m1.400s

top shows 45% cpu usage (almost two full cores)

So I'm having trouble buying the argument that the cost of encrypting everything is minimal. Granted, this is a very data-heavy test.

Anything you do to secure the link can/will be subverted. You can go with SSL to prevent the link being sniffed, you can embed keys in the app. But then those keys are in the hands of the user and can be extracted (no matter how hard you work to hide them). At most you can make it "hard" for a malicious user to emulate/replace/bypass your remote app. You can never make it impossible.

SSL overhead is minimal - there's generally higher overhead in the initial handshake than there is to actually transmit the data later on. If you keep the link open, the handshake overhead becomes a one-time thing and on a long session, irrelevant. There's occasional bursts of overhead if keys get renegotiated, but again, on a long-term link, it's minimal.

As for man-in-the-middle, there's very little you can do. Banks already have enough trouble as is with trojans draining peoples' accounts dry. If someone's in the middle, they can do pretty much whatever they want. With enough monitoring, they could replicate the entirety of your communications protocol and replicate your app.