Tried every permutation of this that I can think of any hints on what I'm doing wrong:

echo -n -e "\x36\x34" | openssl rsautl -inkey rsakey -raw | od -H
echo -n -e "\x64" | openssl rsautl -inkey rsakey -raw | od -H
echo -n -e "\x64\x00\x00\x00\x00\x00\x00" | openssl rsautl -inkey rsakey -raw | od -H

@deadzero, looks like you're verifying the test vectors by encrypting, which unfortunately is going to result in padding errors. I recommend going the other way, try verifying the test vectors by decrypting (which is what you'll need to do to get the final result also). Input the encrypted result and add -decrypt to the openssl parameters.

Also another solution is to extract the RSA parameters from the key and do the computation manually instead of having rsautl do it. It's fast and easy once you understand RSA. Checkout the Intro to RSA problem and the RSA key reference on the info page if you'd like to tackle it that way.

As already stated, remember that we have the private key here. This means we are able to decrypt ciphertexts, so encryption should not be needed to solve this one.

Hey there,

I was trying to decrypt one of the test ciphers, with

echo "71b71b4f55ba6c921c33fbe91f51bf888bba9761134c2ac6456e02f3802b1b69"| openssl rsautl -decrypt -inkey private.pem

I'm getting this error

RSA operation error
139815980078752:error:0406506C:rsa routines:RSA_EAY_PRIVATE_DECRYPT:data greater than mod len:rsa_eay.c:523:

Don't know why ?

@vigo85, what's happening is that the data being input to rsautl is ascii text instead of binary. Check @deadzero's comment above to see how to use the echo command to output binary.