The client generates and sends the server 2048-bit RSA public key, with a hash of the public key + the secret key. The server generates a AES key and sends it to the client, encrypted with RSA, using the public key.
All connections after the AES key is sent will be encrypted.
The jProxy Protocol can be found in the PROTOCOL.md file. We are currently on version 3.0.