RSA Encryption. EVP_PKEY_CTX_new Failed

- 1 answer

I am implementing RSA encryption with OAEP SHA256. Following samples on stackoverflow and openssl docs, I implemented below code.But I failed to find why EVP_PKEY_size and EVP_PKEY_CTX_new() are failing.

Can anyone point the mistake? or can anyone provide a code which can do SHA256 rsa public key encryption?

   keybio = BIO_new(BIO_s_mem());
    if (keybio == NULL)
        printf("Failed to create key BIO");
        return 0;
    int len = BIO_write(keybio, Key, strlen(Key));
    EVP_PKEY*  evppubKey = PEM_read_bio_PUBKEY(keybio, NULL, NULL, NULL);
//  EVP_PKEY* evp_key = PEM_read_bio_RSAPublicKey(keybio, NULL, NULL, NULL);
    int a = EVP_PKEY_size(evp_key);
    EVP_PKEY_CTX* ctx = EVP_PKEY_CTX_new(evp_key, eng);

    if (ctx)
        if (EVP_PKEY_encrypt_init(ctx) > 0)
            EVP_PKEY_CTX_ctrl_str(ctx, "rsa_padding_mode", "oaep");
            EVP_PKEY_CTX_ctrl_str(ctx, "rsa_oaep_md", "sha256");
            EVP_PKEY_CTX_ctrl_str(ctx, "rsa_mgf1_md", "sha256");

            if (EVP_PKEY_encrypt(ctx, NULL, &outlen, data, data_len) <= 0)
                return -1;

            outdata = OPENSSL_malloc(outlen);

            if (!outdata) {
                return -1;
            if (EVP_PKEY_encrypt(ctx, outdata, &outlen, data, data_len) <= 0) {
                return -1;


Update: I made changes to code. PEM_read_bio_PUBKEY() returning NULL. My public starts with -----BEGIN RSA PUBLIC KEY----- I am not sure if PEM_read_bio_PUBKEY() is right method to read.



It's the type.

PEM_read_bio_RSA_PUBKEY takes by reference (if nonnull) AND returns RSA * not EVP_PKEY *; see the man page on your system or on the web. Using the wrong pointed-to type violates a constraint in C, and if you use a decent compiler in conforming mode (which is almost always a good idea) it should have given you diagnostics, which you should have paid attention to.

Similarly, RSA_size requires RSA * which you actually have not EVP_PKEY * as you falsely declared and thus works, but EVP_PKEY_CTX_new requires EVP_PKEY * which you don't have and thus doesn't work. If you had been luckier it would have at least crashed, or given you nasal demons or reformatted your disk.

The simplest solution is to use PEM_read_bio_PUBKEY (note NO RSA), also on that same man page, which takes&returns EVP_PKEY * -- and don't call RSA_size on it. If you actually need the output size or key size, which it doesn't appear you do, use EVP_PKEY_size or EVP_PKEY_bits as appropriate.

Alternatively you could use an RSA * for the result of PEM_read_bio_RSA_PUBKEY and then set it into a separately allocated EVP_PKEY * to use for the encryption. This will work but is more fiddly.

Added: Also you need to use EVP_PKEY_encrypt_init NOT _decrypt_init for EVP_PKEY_encrypt,

I have access only to pub key, not to .pem file

The extension on a file doesn't matter in itself, only in that it is commonly used to indicate the content of the file. PEM format (in a file) can be used for many kinds of data, including a public key, in fact PEM_read_[bio]_[type]_PUBKEY reads exactly that: a PEM file (or other data source) containing a public key. If that's not what you have, your code is much more wrong than I diagnosed above, but since since you don't say what you do have it's impossible to give any useful advice. If you mean you have the private key in a PEM file, and want the public key, use openssl rsa -in private -pubout -out public or openssl pkey -in private -pubout -out public. OpenSSL actually can encrypt with the private key, since the private key value fully determines the public key, but any system that uses keys that way is badly designed and probably insecure.

UPDATE: you now disclose your file contains -----BEGIN/END RSA PUBLIC KEY-----. This is a very rarely used format that is NOT the format that OpenSSL calls 'PUBKEY', which is the SubjectPublicKeyInfo structure from X.509/PKIX and is labelled -----BEGIN/END PUBLIC KEY----- (no 'RSA' even though it contains an RSA publickey). To read your file you must use (as you had in a comment) PEM_read_[bio]_RSAPublicKey -- which takes&returns RSA * not EVP_PKEY *. There is no capability to read this type of file as EVP_PKEY *, but since you need EVP_PKEY * for the encrypt routines you want, you must use the 'alternative' I mentioned previously: first read the file into an RSA *, then build an EVP_PKEY * from that RSA *:

RSA * rsa = PEM_read_bio_RSAPublicKey (bio, NULL, NULL, NULL);
if( rsa == NULL ) error;
EVP_PKEY * evp_key = EVP_PKEY_new();
if( evp_key == NULL ) error; // very unlikely
if( ! EVP_PKEY_set1_RSA (evp_key, rsa) ) error;
RSA_free (rsa);
// and then continue as you already have:
EVP_PKEY_CTX* ctx = EVP_PKEY_CTX_new(evp_key, eng/*normally NULL*/);
if nonnull and EVP_PKEY_encrypt_init(ctx) > 0:
  EVP_PKEY_CTX_ctrl ... and EVP_PKEY_encrypt