Our product has a server that uses embedded Jetty (version 11.0.11) providing functionality with clients connecting through HTTP calls. It can be configured for HTTPS and a lot of our customers use it with this configuration. Recently we came across an issue from one of our customers where they see a strange issue with the https configuration. The request sometimes works correctly and sometimes throws an Invalid SNI error from Jetty. We are aware that this type of issue usually happens when the SNI sent in the request does not match what is in the certificate. So in this case, as part of the troubleshooting exercises, we collected wireshark captures to look at what is being sent and compare with what is on the certificate.
The certificate itself has the server in the subject alternate name as something like "server.xyz.abc.com". We are seeing from wireshark logs that the TLS layer is sometimes sending the SNI as "server.xyz.abc.com" while some other times it sends it as "SERVER.xyz.abc.com". Notice the server name portion in uppercase.
The request is succeeding when the TLS layer is sending the SNI as "server.xyz.abc.com", and the request seems to fail when the SNI comes in "SERVER.xyz.abc.com".
We get the message "org.eclipse.jetty.http.BadMessageException: 400: Invalid SNI".
To fix this, we then had the customer fix the certificate to add "SERVER.xyz.abc.com" in the subject alternate name as well, and deployed this certificate on the server. So the certificate now contains:
server.xyz.abc.com
SERVER.xyz.abc.com
However this still hasn't fixed the issue and the Invalid SNI message still keeps coming intermittently even though wireshark shows the Client Hello SNI coming in exactly as it is in the certificate. One thing we noticed is that the failure still happens when the uppercase name comes in as the SNI in the Client Hello request.
I have the following questions:
- Is there any known issue with Jetty with the case sensitivity of SNI, and is it the right thing to add the uppercase server name to the certificate?
- Why does the TLS layer alternate between sending an uppercase and lowercase name, and is there any type of setting somewhere to force it to send it in say, lowercase?
- As a worst case, we may need to disable the SNI check to fix this. Is that the only way to do this, as this would require us to supply code changes to disable this check. From looking at other posts, I think there is no configurable parameter to disable this check in Jetty.
Please help with any ideas you may have, as we are not sure how to proceed with helping our customer.
