certificate_unknown - javax.net.ssl.SSLHandshakeException: General SSLEngine problem

I was recently debugging a SSL configuration issue with a colleague of mine. He was implementing a 2 way SSL handshake (mutual-auth) for Confluent Kafka (which is built on top of Apache Kakfa). He was trying to get the following communication happening using mutual SSL authentication:

Confluent Control Center (client) -> Broker (server)

Whilst one-way was working properly the system dumped the following generic message as soon as mutual authentication was turned on (on the broker end):

fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem

I verified the configurations and couldn't find anything, time to enable debug logs. The thing with debugging SSL related problems is always enable debug. Searching google with just

fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem

is going to lead you down a rabbit hole. So, after enabling debug logs by adding -Djavax.net.debug=all to the startup arguments, I could see the following information:

qtp733461760-38, fatal error: 46: General SSLEngine problem
sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication
%% Invalidated:  [Session-716, SSL_NULL_WITH_NULL_NULL]
%% Invalidated:  [Session-717, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]
qtp733461760-38, SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown
qtp733461760-38, WRITE: TLSv1.2 Alert, length = 2
qtp733461760-38, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem
qtp733461760-38, called closeOutbound()
qtp733461760-38, closeOutboundInternal()
[Raw write]: length = 7
0000: 15 03 03 00 02 02 2E                               .......
Using SSLEngineImpl.
%% Initialized:  [Session-720, SSL_NULL_WITH_NULL_NULL]
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false

Of all the errors above, line 3 is what was causing the error. It turns out that if you do not specify Extended Key Usage (EKU) in your certificate and enable client-auth specifically, that certificate cannot be used for client authentication. Generally, all CAs when signing the request only put server-auth in the EKU. In our case, getting the CA to return a certificate with both client-auth and server-auth fixed the problem. A quick summary of what was happening (note that this is not the SSL handshake sequence, I have cut it down the problem areas):

1. Client makes a connection over SSL channel to the server
2. Server looks up its certificate store, finds its certificate and sends it to the client
3. Client checks if that certificate can be trusted (by validating it against its own trust store)
4. Server asks the client to send its certificate which the client does
5. Server looks at the client certificate and finds the EKU section and notices that this certificate can only be used for server-auth, whereas this handshake is client-auth
6. Server denies the certificate with the message - Extended key usage does not permit use for TLS client authentication and then terminates the SSL connection

Just because you have a certificate is not enough, you must have the right flags enabled on the certificate.