Is there a specific difference in the one-time-password algorithm when using sha512 to using sha1

Question

I am writing a pam module, that does authentication with one time passwords. I started writing it with sha512 as algorithm for the hmac and ran into a problem. I also did it with sha1, which works. The code is at: https://github.com/Ongy/pam_totp

I tested the hmac with sha512, as far as I can tell that is correct, so the error should be in the get_truncate function, which does work with sha1, but not with sha512.

static int get_truncate(const uint8_t * hash, size_t len, char * buffer,
        size_t maxlen)
{
    uint32_t value;
    uint8_t offset;
    offset = hash[len-1] & 0x0F;
    value = *((uint32_t *) (hash+offset));
    value = be32toh(value) & 0x7FFFFFFF;
    value %= 100000000;

    return snprintf(buffer, maxlen, "%08d", value);
}

which is called in

static int get_totp_sha512(const uint8_t * hashdata, size_t len, uint64_t time,
            char * dst, size_t maxlen)
{
    uint8_t buffer[64];
    uint64_t counter = htobe64(time);

    memset(buffer, 0, sizeof(buffer));

    calculate_hmac_sha512(hashdata, len, (uint8_t *) &counter,
                     sizeof(counter), buffer, sizeof(buffer));

    return get_truncate(buffer, sizeof(buffer), dst, maxlen);
}

Most of the relevant code should be in src/main.c.

The rfcs relevent are: 6238 and 4226. (The second one describes the general algorithm, the first one also allows other hashes than sha1)

Answer 1

Ok so the thing I had to fix was my testcase, I used the test vectors given in https://www.rfc-editor.org/rfc/rfc6238#appendix-B. What I had to change:

The key is not actually "12345678901234567890" for sha512 but "1234567890123456789012345678901234567890123456789012345678901234".