When renewing my SSL/TLS certificate for my DreamHost shared hosting account, I generated a new 4096-bit RSA Private Key using OpenSSL 1.0.1e. I was surprised and confused when DreamHost reported "Invalid private key". I initially thought it was a problem with the 4096-bit key but found documentation indicating 4096 is a supported option.

I checked the that my key was PEM formatted as expected, and finally realized it was an incompatibility between the "-----BEGIN RSA PRIVATE KEY-----" and the "-----BEGIN PRIVATE KEY-----" variants of the PEM format when I couldn’t create a self-signed cert using OpenSSL 0.9.8za and my brand new key. Once I realized this, it was a simple conversion using the command

openssl rsa -in private.key -inform PEM -out outfile.key -outform PEM

with OpenSSL 1.0.1. The in/out forms aren’t strictly necessary, but make the command a little clearer to read.