Finding the Encryption or Hashing method used

I am trying to find out what algorithm the client application is using to return the session key.

When I initiate a connection, the server first sends a unique session key. The Client has then to respond with an encrypted or hashed password and send together with 开发者_StackOverflow社区the Username to the server.

Sample network trace between client and server: (username: serv1ce / password: test12)

App received from Server << 52 d7 1c 3f 9f 2c 05 c9 (one time session key)

App sent to Server >> 11 83 2d 7d ff 0c 51 8c 53 45 52 56 31 43 45 20

The "53 45 52 56 31 43 45 20" part is the username in clear text as bytes values (serv1ce).

Does anyone know how the bytes "11 83 2d 7d ff 0c 51 8c" have been created with the password 'test12' and the 64bit (8bytes) session key "52 d7 1c 3f 9f 2c 05 c9" ?

If they are using a cryptographically secure hash, then in principle from input and output you should not be able to discover this.

In practice they are returning 8 bytes, which is 64 bits, which suggests that they are using some variant of MD5. If they follow typical practice they are likely to have created a string somehow which includes some combination of the username, password, session key, and a secret hash, then hashed it. (Note that I said typical practice, not best practice. The best practice is to use something slow to calculate for this purpose, such a bcrypt.) If you figure out the magic combination, you have the answer.

You have two decent approaches. The simplest is brute force search. If you search for md5 gpu cracking you can find plenty of tools that let you offload MD5 calculations to your video card. These are ideal for brute force search, and can let you try an astonishing number of variations on the above theme quite quickly. (The feasibility of this attack is why people should use bcrypt for this sort of stuff.)

The other is that you have the application. There are various ways to trace what actually happens inside of the application as it is doing that computation. Succeed in figuring out that, and you'll have the answer.