Adding DNS entries in a sub-domain

I was setting up a training environment for one of our customers where they would regularly build and destroy VMs on AWS. 

Our AWS infrastructure sits under the example.com (name changed for confidentiality) domain. So all servers that are created have a name resembling server1.example.com, server2.example.com etc. For the training environment though, I wanted them to be under a different sub-domain, e.g. training.example.com. 

This would allow to have more meaningful names for the training environments and prevent accidental modification of servers in the main domain, e.g. server1.training.example.com and server2.training.example.com.

At first, I thought just creating another hosted zone (training.example.com) under route53 will solve the problem but it did not. I would get 

** server can't find server1.training.example.com: NXDOMAIN

To get around this problem, I had to do the following:

1. Find the Name Server (NS) of training.example.com. This information is available when you click on the hosted zone.
2. Add a new entry of type (NS) under example.com, the entry should resemble:
   

training.example.com NS <value of NS from #1>

Now I could play around with the training environments without having to worry about accidentally destroying any entry in the main domain (example.com)

certificate_unknown - javax.net.ssl.SSLHandshakeException: General SSLEngine problem

I was recently debugging a SSL configuration issue with a colleague of mine. He was implementing a 2 way SSL handshake (mutual-auth) for Confluent Kafka (which is built on top of Apache Kakfa). He was trying to get the following communication happening using mutual SSL authentication:

Confluent Control Center (client) -> Broker (server)

Whilst one-way was working properly the system dumped the following generic message as soon as mutual authentication was turned on (on the broker end):

fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem

I verified the configurations and couldn't find anything, time to enable debug logs. The thing with debugging SSL related problems is always enable debug. Searching google with just

fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem

is going to lead you down a rabbit hole. So, after enabling debug logs by adding -Djavax.net.debug=all to the startup arguments, I could see the following information:

qtp733461760-38, fatal error: 46: General SSLEngine problem
sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication
%% Invalidated:  [Session-716, SSL_NULL_WITH_NULL_NULL]
%% Invalidated:  [Session-717, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]
qtp733461760-38, SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown
qtp733461760-38, WRITE: TLSv1.2 Alert, length = 2
qtp733461760-38, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem
qtp733461760-38, called closeOutbound()
qtp733461760-38, closeOutboundInternal()
[Raw write]: length = 7
0000: 15 03 03 00 02 02 2E                               .......
Using SSLEngineImpl.
%% Initialized:  [Session-720, SSL_NULL_WITH_NULL_NULL]
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false

Of all the errors above, line 3 is what was causing the error. It turns out that if you do not specify Extended Key Usage (EKU) in your certificate and enable client-auth specifically, that certificate cannot be used for client authentication. Generally, all CAs when signing the request only put server-auth in the EKU. In our case, getting the CA to return a certificate with both client-auth and server-auth fixed the problem. A quick summary of what was happening (note that this is not the SSL handshake sequence, I have cut it down the problem areas):

1. Client makes a connection over SSL channel to the server
2. Server looks up its certificate store, finds its certificate and sends it to the client
3. Client checks if that certificate can be trusted (by validating it against its own trust store)
4. Server asks the client to send its certificate which the client does
5. Server looks at the client certificate and finds the EKU section and notices that this certificate can only be used for server-auth, whereas this handshake is client-auth
6. Server denies the certificate with the message - Extended key usage does not permit use for TLS client authentication and then terminates the SSL connection

Just because you have a certificate is not enough, you must have the right flags enabled on the certificate.