RDP Encryption

RDP features data stream encryption, which is based on the RC4 symmetric cipher, with keys up to 128-bit. The RC4 keys are replaced at regular intervals, every 4096 packets.

RDP provides the following different server authentication methods:

RDP 4 authentication was used historically. With RDP 4, the RDP client does not perform any checks in order to verify the identity of the server it connects to. Since user credentials can be obtained using a man in the middle (MITM) attack, RDP4 authentication is insecure and should not be used.
RDP 5.1 authentication employs a server certificate for which the client possesses the public key. This way it is guaranteed that the server possess the corresponding private key. However, as this hard-coded private key became public some years ago, RDP 5.1 authentication is also insecure.
RDP 5.2 or later authentication uses Enhanced RDP Security, which means that an external security protocol is used to secure the connection. RDP 4 and RDP 5.1 use Standard RDP Security. The VRDP server supports Enhanced RDP Security with TLS protocol and, as a part of the TLS handshake, sends the server certificate to the client.

The Security/Method VRDE property sets the required security method that is used for a connection. You can also change this in the VM Settings, Remote Display tab. Valid values are as follows:
- Negotiate. Both Enhanced (TLS) and Standard RDP Security connections are allowed. The security method is negotiated with the client.
- RDP. Only Standard RDP Security is accepted.
- TLS. Only Enhanced RDP Security is accepted. The client must support TLS. This is the default setting.
  
  The version of OpenSSL used by supports TLS versions 1.0, 1.1, 1.2, and 1.3.
For example, the following command configures a client to use either Standard or Enhanced RDP Security connection:
```
VBoxManage modifyvm VM-name --vrde-property "Security/Method=negotiate"
```
If the Security/Method property is set to either Negotiate or TLS, the server uses TLS if the client supports it. However, to use TLS the server must have the Server Certificate and the Server Private Key. A Certificate Authority (CA) Certificate is optional.

If you choose TLS as the security method, generates a server key and certificate pair called VRDEAutoGeneratedPrivateKey.pem and VRDEAutoGeneratedCert.pem) for the VM. also recreates the auto-generated certificate and key if they are about to expire, or if one or both are deleted or corrupted.

To generate a custom server key and certificate pair, with a CA certificate, follow these steps.
1. Create a CA self signed certificate.
```
openssl req -new -x509 -days 365 -extensions v3_ca \
  -keyout ca_key_private.pem -out ca_cert.pem
```
2. Generate a server private key and a request for signing.
```
openssl genrsa -out server_key_private.pem
openssl req -new -key server_key_private.pem -out server_req.pem
```
3. Generate the server certificate.
```
openssl x509 -req -days 365 -in server_req.pem \
  -CA ca_cert.pem -CAkey ca_key_private.pem -set_serial 01 -out server_cert.pem
```
Configure the server to access the required files. For example:
```
VBoxManage modifyvm VM-name \
  --vrde-property "Security/CACertificate=path/ca_cert.pem"
```
```
VBoxManage modifyvm VM-name \
  --vrde-property "Security/ServerCertificate=path/server_cert.pem"
```
```
VBoxManage modifyvm VM-name \
  --vrde-property "Security/ServerPrivateKey=path/server_key_private.pem"
```
Note that does not maintain custom certificates. You are responsible for keeping these updated.

As the client that connects to the server determines what type of encryption will be used, with rdesktop, the Linux RDP viewer, use the -4 or -5 options.