.NET 6 SslStream.AuthenticateAsServer is slow with RSA 4096 key size

I have a HTTPs Server .NET 6 application which uses TCPclient network and SslStream.AuthenticateAsServer. What I have notice using wireshark is that it takes about 4 seconds delay for my server to send "server hello, Certificate, Server key Exchange, Server hello Done" after receiving "Client hello" under load. The load is Jmeter 50 threads sending ~10KB payload https request to my server.

I have noticed that using different key (2048 vs 4096) side will affect the delay of server hello.

The server key is generated using openssl:

openssl genrsa -out priKey.pem 4096

I have narrow and identify down to the single line of code which is:

sslstream.AuthenticateAsServer(serverCert, false, System.Security.Authentication.SslProtocols.Tls | System.Security.Authentication.SslProtocols.Tls11 | System.Security.Authentication.SslProtocols.Tls12 | System.Security.Authentication.SslProtocols.Tls13, false);

This is the line that does the TLS handshake which is using .NET 6 API to execute the TLS connection which is out of my control. I have search high and low for options but would like to check if is there anything alternative can be done or setting to improve the delay?

The setup:

• CPU: Intel Atom® Processor C3758
• RAM: 8GB DDR4