how to use ECDSA to create a self signed digital certificate programatically

I am trying to create a digital certificate which to be self signed with EC keys instead of those from RSA and followed these SO link1 and link2. I replaced the signature algorithm from RSA given in link1 to EC as

EC_KEY *ecc = NULL;
int eccgrp = OBJ_txt2nid("secp256k1");
ecc = EC_KEY_new_by_curve_name(eccgrp);
EC_KEY_set_asn1_flag(ecc, OPENSSL_EC_NAMED_CURVE);

if(!(EC_KEY_generate_key(ecc))) {
    BIO_printf(out, "Error in generating key");
    printf("Error 1\n");
}

if(!EVP_PKEY_assign_EC_KEY(pk, ecc)) {
    BIO_printf(out, "Error assigning EC_KEY to EVP_PKEY");
    printf("Error 2\n");
}
.
.
X509_set_pubkey(x,pk);
.
.
if (!X509_sign(x,pk,EVP_md5()))
    goto err;
.
.

Rest of the code is same as given in link1. No error is printed but i get a seg-fault when i try to print the x509 cert X509_print_fp(stdout,x509);. What is correct way of doing this?

Solution

You changed that code more than you said because it doesn't have variables 'pk' or 'x'.

Anytime you get an error/failure return from a libcrypto routine you should always look at the error queue; see https://www.openssl.org/docs/faq.html#PROG6 and https://www.openssl.org/docs/faq.html#PROG7 . (For libssl routines you should do so depending on the return from SSL_get_error.) If you do so as in my code below for the 'bad' case, you see X509_sign fails with

140018941793960:error:100C508A:elliptic curve routines:PKEY_EC_CTRL:invalid digest type:ec_pmeth.c:388:

because standard ECDSA signature schemes don't include MD5. You must use SHA1 or SHA2, and you should use a hash matched in strength to the ECC key, which in this case is SHA256. Since the signature failed, the x509 structure does not contain valid data and can't be successfully printed.

Also note: since 1.0.0 (in 2010) PEM_write_PrivateKey with enc nonnull uses 'new' (ca. 2000!) PKCS#8/PBES2 format, which needs a suitable variant of OpenSSL_add_all_algorithms.

Complete working demo-quality code:

/* SO #35899969 */
#include <stdio.h>
#include <stdlib.h>
#include <openssl/err.h>
#include <openssl/evp.h>
#include <openssl/pem.h>
#include <openssl/x509.h>
#ifdef _WIN32
#include <openssl/applink.c>
#endif

/* minimal error handling for demo; real code do better */
void err (const char *label)
{
  fprintf (stderr, "Error in %s:\n", label);
  ERR_print_errors_fp (stderr);
  exit (1);
}

int main (int argc, char**argv)
{
  int bad = argc>1;
  ERR_load_crypto_strings(); /* or SSL_load_error_strings */
  OPENSSL_add_all_algorithms_noconf(); /* for PKCS8 w PBES2 */

  EVP_PKEY * pkey = EVP_PKEY_new();
  EC_KEY *ecc = EC_KEY_new_by_curve_name(NID_secp256k1);
  /* simpler than going through OBJ_txt2nid */
  if(!ecc) err("ECCnewbyname");
  EC_KEY_set_asn1_flag(ecc, OPENSSL_EC_NAMED_CURVE);
  if(!(EC_KEY_generate_key(ecc))) err("ECCgen");
  if(!EVP_PKEY_assign_EC_KEY(pkey, ecc)) err("PKEYassign");

  X509 * x509 = X509_new();
  /* REALLY shouldn't use fixed serial if DN isn't unique */
  ASN1_INTEGER_set(X509_get_serialNumber(x509), 1);
  X509_gmtime_adj(X509_get_notBefore(x509), 0);
  X509_gmtime_adj(X509_get_notAfter(x509), 365L*86400);
  X509_set_pubkey(x509, pkey);
  X509_NAME * name = X509_get_subject_name(x509);
  X509_NAME_add_entry_by_txt(name, "C",  MBSTRING_ASC,
                           (unsigned char *)"CA", -1, -1, 0);
  X509_NAME_add_entry_by_txt(name, "O",  MBSTRING_ASC,
                           (unsigned char *)"MyCompany Inc.", -1, -1, 0);
  X509_NAME_add_entry_by_txt(name, "CN", MBSTRING_ASC,
                           (unsigned char *)"localhost", -1, -1, 0);
  X509_set_issuer_name(x509, name);
  if(!X509_sign(x509, pkey, bad? EVP_md5(): EVP_sha256())) err("X509sign");

  /* simplified */
  if(!PEM_write_PrivateKey(stdout, pkey, EVP_des_ede3_cbc(),
                          NULL,0,NULL,"passphrase")) 
    err("writeKey");
  if(!PEM_write_X509(stdout, x509))
    err("writeCert");
  /* added */
  X509_print_fp (stdout, x509); 
  return 0;
}