Testing a custom openSSL engine from the command line

Question

Quick question: How can I verify the functionality of a custom openSSL engine I'm writing from the command line?

Right now I am following along with this great tutorial, and am successfully able to exercise the engine (which returns a digest value of all 2's) using my test program (source code here, test program located at test/wssha256engine_test.c).

brett@wintermute:~/openssl_ws/wssha256engine$ make test
make[1]: Entering directory 
/home/brett/openssl_ws/wssha256engine/test
make[1]: '../bin/test' is up to date.
make[1]: Leaving directory 
/home/brett/openssl_ws/wssha256engine/test
brett@wintermute:~/openssl_ws/wssha256engine$ bin/test
Engine Loaded...
Initializing wssha256 engine...
*TEST: Successfuly initialized engine [A test engine for the ws sha256 hardware encryption module, on the Xilinx ZYNQ7000]
    init result = 1
Digest NID=672 requested
SHA256 algorithm context initialized
*TEST: Digest INIT 1
SHA256 update
*TEST: Digest Update 1
SHA256 final: sizeof(EVP_MD)= 120
SHA256 cleanup
*TEST: Digest Final 1 Digest size:32
22222222222222222222222222222222

However, because reasons, I would like to also use the openssl command line interface to exercise the engine I just wrote, and have it compute the digest of a random string like in this other tutorial, just using sha256 and not md5.

But when I try and do this, the engine does not load and results in an error telling me that the NID of my digest doesn't exist, and just hashes the string with the standard algorithm instead:

brett@wintermute:~/openssl_ws/wssha256engine$ echo "Hello" | openssl dgst -engine /home/brett/Thesis/openssl_ws/wssha256engine/bin/libwssha256engine.so -sha256
ERROR: Digest is empty! (NID = 0)
engine "wssha256" set.
(stdin)= 66a045b452102c59d840ec097d59d9467e13a3f34f6494e539ffd32c1bb35f18

Why can't I use my engine on the command line, but I can create a C program to load it? And how can I use my engine to compute a digest from the command line?

Note: I am able to load the engine from the command line, just not use it.

Answer 1

After some digging, I was able to figure out why the CLI invocation of my engine did not work, and fix the issue.

The problem was in my digest selector function (original broken code is commented out below). Originally, I returned a failure status from the digest selector function, rather than returning the number of digest IDs that the engine supports.

static int wssha256engine_digest_selector(ENGINE *e, const EVP_MD **digest, const int **nids, int nid)
{
    if (!digest)
  {
    *nids = wssha256_digest_ids;

    // THIS ORIGINAL CODE CAUSES THE ENGINE TO NEVER INITIALIZE FROM A CLI INVOCATION
    //printf("ERROR: Digest is empty! (NID = %d)\n",nid);
    //return FAIL;

    // THIS FIXES IT
    int retnids = sizeof(wssha256_digest_ids - 1) / sizeof(wssha256_digest_ids[0]);
    printf("wssha256engine: digest nids requested...returning [%d]\n",retnids);
    return retnids;
}

Reading more closely, we can find that OpenSSL calls the digest selector function in following (separate) ways:

1. with digest being NULL. In this case, *nids is expected to be assigned a zero-terminated array of NIDs and the call returns with the number of available NIDs. OpenSSL uses this to determine what digests are supported by this engine.
2. with digest being non-NULL. In this case, *digest is expected to be assigned the pointer to the EVP_MD structure corresponding to the NID given by nid. The call returns with 1 if the request NID was one supported by this engine, otherwise 0.

So even though my engine supported sha256, the CLI invocation didn't work because it must have not been initialized with a digest before the digest selector function is invoked (no clue as to the order of operations when using the CLI). However when I manually initialized it in my test program, all worked fine. Changing to a non-failure return value to support invocation 1. above fixed the problem.