There are a quite a few good guides for configuring your own CA signed certificate for RDP; however, details it’s easy to gloss over and most of the troubleshooting is buried in the comments:
Create an RDP certificate with an RSA key. The signing key of the CA does not matter.
Create an RDP certificate with the TLS (web) Server EKU, not the Remote Desktop EKU.
Add the certificate to the Personal certificate store, not the Remote Desktop certificate store.
Example errors:
Error log when using an RDP certificate with an ECDSA key.
Windows 10 and Remote Desktop 10 on macOS report an Unknown/Invalid EKU.
Set-WmiInstance error trying to use a certificate in the Remote Desktop certificate store.
WMIC error trying to use a certificate in the Remote Desktop certificate store.
First, ensure you download XCA v2.x from the official download page https://hohnstaedt.de/xca/index.php/download. The instructions didn’t make sense at first since I was running xca v1.4.1.
Create a file named “oids.txt” in the user’s XCA directory:
Copy the eku.txt file from the XCA installation location to the user’s XCA directory:
Windows: C:\Program Files\xca
macOS: /Applications/xca.app/Contents/Resources This can be accessed through the command line or right clicking on the xca application and selecting “Show Package Contents”
Linux: /usr/share/xca or /usr/local/share/xca
Note: The whole file eku.txt file must be copied, because xca only parses the first eku.txt it encounters.
Add a line to the user’s eku.txt referencing your new EKU:
Close and re-open XCA and your new EKU will be available:
After adding the Remote Desktop Authentication EKU, I found out it is no longer supported/recognized. The Microsoft Remote Desktop 10 app on macOS and Windows 10 both report the EKU as invalid/unknown.