I have a Spring Boot reactive application running on both ECS and EKS, with similar hardware configuration (cpu:6 and memory 16G).This applications has a REST API endpoint that updates multiple DynamoDB tables via AWS SDK 2.x Async client (DynamoDbAsyncClient).
We find that the AWS SDK latency is higher (nearly 6x) in the pod in EKS compared to that of ECS. There are no errors/exceptions, just slower latency. This happens even though the traffic on EKS pod is very less (1/100th) of the ECS task.
I enabled AWS SDK metrics on both deployments, and found that the ApiCallDuration is higher in EKS.
Datadog profiles between the two environments show that the SslHandler.flush(ChannelHandlerContext) method call in NettyRequestExecutor.writeRequest(HttpRequest) is taking more that 5x time in EKS compared to ECS.
Screenshot 1 - ECS :
Screenshot 2 - EKS :
- Is this reasoning correct?
- Can outgoing HTTPS calls be slower in EKS?
- How can I troubleshoot this further?
- How can I increase speed of Https calls from Netty client in EKS?