int RSA_private_encrypt(int flen, unsigned char *from, unsigned char *to, RSA *rsa, int padding);
int RSA_public_decrypt(int flen, unsigned char *from, unsigned char *to, RSA *rsa, int padding);
These functions handle RSA signatures at a low level.
RSA_private_encrypt() signs the flen bytes at from (usually a message digest with an algorithm identifier) using the private key rsa and stores the signature in to. to must point to RSA_size(rsa) bytes of memory.
padding denotes one of the following modes:
Raw RSA signature. This mode should only be used to implement cryptographically sound padding modes in the application code. Signing user data directly with RSA is insecure.
RSA_public_decrypt() recovers the message digest from the flen bytes long signature at from using the signer's public key rsa. to must point to a memory section large enough to hold the message digest (which is smaller than RSA_size(rsa) - 11). padding is the padding mode that was used to sign the data.
RSA_private_encrypt() returns the size of the signature (i.e., RSA_size(rsa)). RSA_public_decrypt() returns the size of the recovered message digest.
On error, -1 is returned; the error codes can be obtained by ERR_get_error(3).
The padding argument was added in SSLeay 0.8. RSA_NO_PADDING is available since SSLeay 0.9.0.