c++ - OpenSSL's rsautl cannot load public key created with PEM_write_RSAPublicKey

Question

I've generated a public key using openssl

BIGNUM* e = BN_new();
BN_set_word(e, 17);
RSA* rsa = RSA_new();
if(!RSA_generate_key_ex(rsa, 2048, e, NULL)) {
    LOG(security, debug) << "failed to generate private key";
}

And these are written to files:

FILE* pubwriter = fopen("key.pub", "wb");
int err = PEM_write_RSAPublicKey(pubwriter, key);
if(!err) {
    throw new std::runtime_error("Failed to store public key");
}
FILE* privwriter = fopen("key.priv", "wb");
std::string password = "password";
err = PEM_write_RSAPrivateKey(privwriter, key, EVP_des_ede3_cbc(),
                                (unsigned char*)password.c_str(),
                                password.size(), NULL, NULL);

And they seem to be stored correctly, key.pub then contains something like

-----BEGIN RSA PUBLIC KEY-----
MIIBCAKCAQEA0rG1b0g3NIsDT8HkzgTx8BUI9LhueWbH1NuAvIh9qTF57GzEXuTu
jxBcuSzWXlR83ci4oITp7VqV6KLVoJryf8orGXBVi9A73JYOQVB6FEzARKym/g8E
fSEwSmdQ4NfiTESwOCtIextdu3x8pANSYDyyqdiWsSHy0SiZmkbvdVYomIBJZOV9
jhb3mkmD0WUYie9AXziTbli97YqDiN168kMI+7ePpbNJFSVSIUkPPocSgvgcAux/
HuDqftzBgyAF3NGb3AAra1A8T7yPOqLyYyXdIJmF+/Svk5PdMbZVE/U76cjBThch
Q9AiLo25hOjkmtuEQubCuwrUDleblr93aQIBEQ==
-----END RSA PUBLIC KEY-----

Now I want to try these to make sure I haven't got anything wrong but it wont' load:

$ openssl rsautl -encrypt -inkey key.pub -pubin -in data.txt -out enc.txt
unable to load Public Key

What am I missing?

Answer 1

int err = PEM_write_RSAPublicKey(pubwriter, key);

PEM_write_RSAPublicKey writes just the public key. You can make the command work using PEM_write_PUBKEY. The various *_PUBKEY routines write the SubjectPublicKeyInfo, which includes the algorithm OID and public key.

When you write the SubjectPublicKeyInfo, OpenSSL calls it "traditional" format. It will also have the header -----BEGIN PUBLIC KEY----- (and not -----BEGIN RSA PUBLIC KEY-----).

Below, I used PEM_write_PUBKEY to save the public key rsa-public.pem:

$ openssl rsautl -encrypt -inkey rsa-public.pem -pubin -in data.txt -out enc.bin
$ hexdump enc.bin 
0000000 45 53 31 ad 9d 6a c4 37 1e 22 4b 83 c6 27 c8 3c
0000010 df cb 87 a4 60 d8 63 9a 83 9f ee ca e5 8f 8e dd
0000020 d4 d0 98 97 1c b3 36 55 f1 84 ea 7f fe bf 22 b6
0000030 93 20 a2 d5 b2 bd 20 cc 52 8e c7 1b 33 e6 40 40
0000040 cb 7d 6f 17 f1 eb f1 d4 9d 66 fb 67 eb 67 ba 2a
0000050 44 c2 52 15 54 8d 79 76 ad 26 61 35 27 9c bb 6c
0000060 5b 0e 79 b3 d3 27 0b a9 72 17 0d 2d 19 d7 60 19
0000070 16 46 80 4b c0 ae 75 53 9e 6f f5 24 d9 1a a3 6a
0000080 2f 38 13 f6 72 19 20 94 de 40 75 20 51 f4 08 f4
0000090 74 b8 ac 49 01 d6 f8 f4 e5 79 38 88 2d 02 b7 bd
00000a0 f7 63 c1 e1 e5 ec 39 a1 fa 7c ce 0f 83 16 70 7e
00000b0 cd 7e f5 6b 51 c2 db d7 f6 c4 46 5d e5 93 d3 3d
00000c0 ab e6 3b 1a 97 d4 c9 54 e7 aa 90 2d 0a b9 c2 4b
00000d0 3c 58 fd 26 58 5a 63 c0 8c ae b9 72 24 a1 68 5d
00000e0 83 d7 5b ae 56 2a 78 46 8c f4 21 96 bd d3 0c 93
00000f0 8e 35 61 9c b8 56 2e 3a 4e 05 d9 1e 0b 59 14 11
0000100

---

PEM_write_PUBKEY requires a EVP_PKEY. Just use something like:

EVP_PKEY* pkey = EVP_PKEY_new();
ASSERT(pkey != NULL);

int rc = EVP_PKEY_set1_RSA(pkey, rsa);
ASSERT(rc == 1);
...

EVP_PKEY_free(pkey);

The set1 bumps the reference count on the RSA key, so you have to free it through EVP_PKEY_free.

---

The difference between PEM_write_RSAPublicKey and PEM_write_PUBKEY is very obvious when you save in ASN.1/DER. But it gets lost in the PEM encoding.

Here's the non-traditional key in ASN.1/DER and dumped. Its the ASN.1 equivalent of PEM_write_RSAPublicKey. Its just {n,e}:

$ dumpasn1 rsa-public-1.der 
  0 266: SEQUENCE {
  4 257:   INTEGER
       :     00 D1 C8 05 BF AC 04 72 AA 0E 84 FB 47 75 59 97
       :     E1 81 65 0B 0A 1D 9D 2A A8 A1 E0 B1 14 5D 57 69
       :     D4 D2 E2 C6 64 54 94 C2 92 CC C7 99 1A 97 89 72
       :     F6 36 6A A7 B8 34 2C AB A9 CB 77 EB 0D A1 4E 72
       :     24 9F 96 D6 1C 28 ED 44 E8 0B 22 7F F3 5B 52 E2
       :     7E A6 5E F1 7C A2 29 4F F1 8B 9D 0F 94 27 05 D5
       :     BD 2E 1A AD B4 12 0D E0 69 3E 0B 1B A7 F8 71 B5
       :     AD 22 4B 18 FF 72 88 F3 C5 77 B0 CF 88 5C F4 19
       :             [ Another 129 bytes skipped ]
265   3:   INTEGER 65537
       :   }

0 warnings, 0 errors.

Here's the traditional public key in ASN.1/DER and dumped. Its the ASN.1 equivalent of PEM_write_PUBKEY. Its the one that writes the SubjectPublicKeyInfo, and it includes an algorithm OID and public key:

$ dumpasn1 rsa-public-2.der 
  0 290: SEQUENCE {
  4  13:   SEQUENCE {
  6   9:     OBJECT IDENTIFIER rsaEncryption (1 2 840 113549 1 1 1)
 17   0:     NULL
       :     }
 19 271:   BIT STRING, encapsulates {
 24 266:     SEQUENCE {
 28 257:       INTEGER
       :         00 D1 C8 05 BF AC 04 72 AA 0E 84 FB 47 75 59 97
       :         E1 81 65 0B 0A 1D 9D 2A A8 A1 E0 B1 14 5D 57 69
       :         D4 D2 E2 C6 64 54 94 C2 92 CC C7 99 1A 97 89 72
       :         F6 36 6A A7 B8 34 2C AB A9 CB 77 EB 0D A1 4E 72
       :         24 9F 96 D6 1C 28 ED 44 E8 0B 22 7F F3 5B 52 E2
       :         7E A6 5E F1 7C A2 29 4F F1 8B 9D 0F 94 27 05 D5
       :         BD 2E 1A AD B4 12 0D E0 69 3E 0B 1B A7 F8 71 B5
       :         AD 22 4B 18 FF 72 88 F3 C5 77 B0 CF 88 5C F4 19
       :                 [ Another 129 bytes skipped ]
289   3:       INTEGER 65537
       :       }
       :     }
       :   }

0 warnings, 0 errors.

---

 err = PEM_write_RSAPrivateKey(privwriter, key, EVP_des_ede3_cbc(),
                            (unsigned char*)password.c_str(),
                            password.size(), NULL, NULL);

I believe the OpenSSL folks recommend you use PEM_write_PKCS8PrivateKey. See PEM(3) and pkcs8(1).

---

Now I want to try these to make sure I haven't got anything wrong but it wont' load:

 $ openssl rsautl -encrypt -inkey key.pub -pubin -in data.txt -out enc.txt

You can understand the behavior by looking at <openssl src>/apps/rsautl.c. Here are the relevant lines:

else if (!strcmp(*argv, "-pubin")) {
    key_type = KEY_PUBKEY;
}
...

case KEY_PUBKEY:
    pkey = load_pubkey(bio_err, keyfile, keyform, 0, NULL, e, "Public Key");
    break;
...

Then, in apps.c:

if (format == FORMAT_ASN1) {
    pkey = d2i_PUBKEY_bio(key, NULL);
}
...

else if (format == FORMAT_PEM) {
    pkey = PEM_read_bio_PUBKEY(key, NULL, ...);
}
...

The observation above is the routines are using *_PUBKEY.

There's also a code path based on format == FORMAT_PEMRSA that calls PEM_read_bio_RSAPublicKey, but I don't know how to trigger it. Looking at rsautl(1), I don't think you can because there's no switch that exposes it.

If it was going to trigger, it would be based on a combination of the -keyform option combined with format == FORMAT_PEMRSA. But apps.c's str2fmt does not return a FORMAT_PEMRSA.

I think that means your only option is to use a SubjectPublicKeyInfo. And that means using PEM_write_PUBKEY (or convert the key after the fact).

---

Now related: OpenSSL Bug Report, Issue 3887: rsautl and intelligent retry for Public Key parse after Traditional/Subject Public Key Info parse fails.