There are a quite a few good guides for configuring your own CA signed certificate for RDP; however, details it’s easy to gloss over and most of the troubleshooting is buried in the comments:
- Create an RDP certificate with an RSA key. The signing key of the CA does not matter.
- Create an RDP certificate with the TLS (web) Server EKU, not the Remote Desktop EKU.
- Add the certificate to the Personal certificate store, not the Remote Desktop certificate store.
Example errors:
Error log when using an RDP certificate with an ECDSA key.

Windows 10 and Remote Desktop 10 on macOS report an Unknown/Invalid EKU.

Set-WmiInstance error trying to use a certificate in the Remote Desktop certificate store.

WMIC error trying to use a certificate in the Remote Desktop certificate store.
