Search code examples
javaerlangx509diffie-hellman

Java / Erlang: Diffie Hellman Key Exchange not Working

I wanted to implement my own encryption for my application. I did a major overhaul of this post. I really didn't have much time until today to address it. Hopefully this being the more useful than my original. Spent a ridiculous amount of time on this issue. Hopefully can save others that time.

I encountered several issues while doing this. I did not realize what was happening until the very end. I was getting different shared secrets and later some exceptions.

This is what I tried:

  • Used the built in facilities provided by both languages. Couldn't figure out how to get the raw public key into a form Java could use.
  • Scratched that and went with the simple formulas to calculate the public and private keys for each party. (This statistically could've worked ~25% of the time... luckily for me it didn't.)
  • Dived into ASN.1 documentation from the ITU and sent the Erlang public key encoded in a similar manner to Java's keys. Determined this by saving the Java key to a file and using a hex editor. I didn't go back and test at great length. It did get rid of java.security.spec.InvalidKeySpecException: Inappropriate key specification. Think statistics didn't work in my favor here either. The secrets still did not match.
  • Sent all numbers from Java to the Erlang-side to compute keys, shared secret using Java numbers... Same numbers. There is hope!!!
  • Started carefully examining the data they were communicating. This was a bit time consuming as Erlang has the data organized in unsigned bytes. The Eclipse IDE (maybe there's a setting somewhere to change) uses signed bytes in byte arrays and a signed integer array within BigInteger.

This is where I began to see things. This all was manually entered over many iterations to make sure I found the correct pattern of events. In Erlang I see my public key beginning with <<215, 101, 208, 153,. The first element of the BigInteger on the Java-side is 681193318. The buffer the byte data was read into reads: [-41, 101, -48, -103. (Same as Erlang's). However taking the time convert the 1st four elements of the binary string to an integer...

<<I:32/signed-integer>> = <<215,101,208,153>>.

That yields -681193319 versus the big integer's 681193318

The code I was using was some what simple:

Erlang "Server":

-module(echo).
-export([start/0]).

start() ->
    crypto:start(),
    spawn(fun () -> {ok, Sock} = gen_tcp:listen(12321, [binary, {packet, raw}]),
    echo_loop(Sock)
    end).

echo_loop(Sock) ->
    {ok, Conn} = gen_tcp:accept(Sock),
    io:format("Got connection: ~p~n", [Conn]),
    Handler = spawn(fun () -> handle(Conn) end),
    gen_tcp:controlling_process(Conn, Handler),
    echo_loop(Sock).

p() ->
    16#ffffffffffffffffc90fdaa22168c234c4c6628b80dc1cd129024e088a67cc74020bbea63b139b22514a08798e3404ddef9519b3cd3a431b302b0a6df25f14374fe1356d6d51c245e485b576625e7ec6f44c42e9a637ed6b0bff5cb6f406b7edee386bfb5a899fa5ae9f24117c4b1fe649286651ece65381ffffffffffffffff.

g() ->
    2.

handle(Conn) ->
    receive
        {tcp, Conn, Yc} ->
            Xs = crypto:strong_rand_bytes(64),
            Ys = crypto:mod_pow(g(),Xs,p()),
            S = crypto:mod_pow(Yc, Xs, p()),

            AESKey = crypto:hash(sha256, S),

            gen_tcp:send(Conn, Ys),%KeyCert),
            handle(Conn);
        {tcp_closed, Conn} ->
            io:format("Connection closed: ~p~n", [Conn])
    end.

Java "Client":

public class MyProgram {
    private static Socket s;
    private static OutputStream out;
    private static InputStream in;
    /**
     * @param args
     */
    public static void main(String[] args) {
        // TODO Auto-generated method stub
        MessageDigest hash;
        byte buffer[] = new byte[1024];
        byte buf2[];
        int len = 0;
        byte[] aeskey;

        try {
            hash = MessageDigest.getInstance("SHA-256");
            byte    keybuffer[] = new byte[64];
            SecureRandom srnd = SecureRandom.getInstance("SHA1PRNG");
            BigInteger Xc, Yc, Sc, Ys;

            srnd.nextBytes(keybuffer);
            Xc = new BigInteger(keybuffer);
            Yc = new BigInteger("2").modPow(Xc, DiffieHellman.Group2.P);

            s = new Socket("localhost",12321);
            out = s.getOutputStream();
            in = s.getInputStream();

            out.write(Yc.toByteArray());
            out.flush();

            len = in.read(buffer);
            buf2 = new byte[len];
            System.arraycopy(buffer, 0, buf2, 0, len);

            Ys = new BigInteger(buf2);          
            Sc = Ys.modPow(Xc, DiffieHellman.Group2.P);
            aeskey = hash.digest(Sc.toByteArray());

            out.close();
            in.close();
            s.close();
        } catch (Exception e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }       
    }
}

What was wrong?

Solution

  • The issue is reading but not understanding the documentation. I spend a lot of time in reference pages because I don't code very often. I did not think much of this particular detail in the documentation for BigInteger:

    All operations behave as if Bigintegers were represented in two's-complement notation...

    There are two spots in my original code where this presents an issue:

           Ys = new BigInteger(buf2);          
       Sc = Ys.modPow(Xc, DiffieHellman.Group2.P);

    The issue with the first line is that if bit 8 is set in the first byte the entire buf2 array needs to be prepended with a 0x00 byte. There is an issue with the second line as well... it does not become evident until the following line is executed: aeskey = hash.digest(Sc.toByteArray());

    The issue here is if bit 8 is set in the first byte of the result... 0x00 is prepended to it. This is forwarded to the digest() function but needs to be omitted.

    My code changed to what is below and works: :)

        len = in.read(buffer);
    buf2 = new byte[len+1];
    System.arraycopy(buffer, 0, buf2, 1, len);
    buf2[0] = 0;

    if(buf2[1] < 0)
        Ys = new BigInteger(buf2);
    else
        Ys = new BigInteger(Arrays.copyOfRange(buf2, 1, buf2.length));

    Sc = Ys.modPow(Xc, DiffieHellman.Group2.P);
    buffer = Sc.toByteArray();
    if(buffer[0] == 0)
        aeskey = hash.digest(Arrays.copyOfRange(buffer, 1, buffer.length));
    else
        aeskey = hash.digest(buffer);

    These two lines were left as is:

        Xc = new BigInteger(keybuffer);
    Yc = new BigInteger("2").modPow(Xc, DiffieHellman.Group2.P);

    This is because the private key can be "any random number." The 0x00 byte is prepended if necessary to the public key of the client in the second line. However Erlang interprets integers as big-endian and any leading 0x00 bytes end up being irrelevant as it does not affect the numerical value and hence the result when conducting a crypto:mod_pow().

    Comments on how to improve code very welcome.