Repository navigation
TLS performance degradation when using Java 11 #856
Description
Significant TLS performance degradation after switching to AdoptOpenJDK 11 or above. Checked on Debian and Mac. Reproducible both locally and using k8s cluster with official AdoptOpenJDK images.
Platform: Debian, Mac
Sbt: 1.3.12
Scala: 2.13.2, 2.12.8, 2.11.12
Finagle: 20.5.0
Expected behavior
Similar CPU utilization in Java 8 and Java 11
Actual behavior
CPU utilization was about 3 times higher which also increase request processing time.
A profiler shows the issue directly related to TLS handling.
Steps to reproduce the behavior
A service code processing TLS conection
import java.io.File
import com.twitter.finagle.ssl.KeyCredentials
import com.twitter.finagle.ssl.server.SslServerConfiguration
import com.twitter.finagle.{Http, Service, http}
import com.twitter.util.{Await, Future}
object Main extends App {
val keyCredentials =
KeyCredentials.CertKeyAndChain(
new File("src/main/resources/localhost.crt"),
new File("src/main/resources/localhost.key"),
new File("src/main/resources/RootCA.pem"))
val sslConfiguration = SslServerConfiguration(keyCredentials)
val service = new Service[http.Request, http.Response] {
def apply(req: http.Request): Future[http.Response] = {
Future.value(http.Response(req.version, http.Status.Ok))
}
}
val server = Http.server.withTransport.tls(sslConfiguration).serve(":8443", service)
Await.ready(server)
}
Please checkout complete project from https://github.com/andrievsky/finagle-tls
Java 8
sdk use java 8.0.222.hs-adpt
sbt run
ab -n 10000 -c 10 https://localhost:8443/
Percentage of the requests served within a certain time (ms)
50% 43
66% 44
75% 45
80% 46
90% 49
95% 52
98% 56
99% 60
100% 81 (longest request)
Java 11
sdk use java 11.0.7.hs-adpt
sbt run
ab -n 10000 -c 10 https://localhost:8443/
Percentage of the requests served within a certain time (ms)
50% 194
66% 200
75% 203
80% 205
90% 211
95% 216
98% 222
99% 227
100% 539 (longest request)
Activity
hamdiallam commentedon Jun 17, 2020
Can you paste outputs from something like async profiler? (https://github.com/jvm-profiling-tools/async-profiler) This seems to be more of a problem with AdoptOpenJDK perhaps? Have you tried to see if this regression is present using OpenJDK 11?
andrievsky commentedon Jun 23, 2020
Yes, basically the results are the same. This regression is present in OpenJDK 11.
I tested both Oracle and Adopt images using Docker on Mac. I also got similar results on native Linux and Mac Java runtime earlier.
Native OpenSSL has been used in all cases (using static lib io.netty.tcnative.boringssl).
Async-profiler data collected from
$ ./profiler.sh -d 30 {PID}under the same conditionsab -n 10000 -c 10 https://localhost:{PORT}/image: amd64/openjdk:8-jre
java version:
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
Profiler details
image: amd64/openjdk:11-jre
java version:
openjdk version "11.0.7" 2020-04-14
OpenJDK Runtime Environment 18.9 (build 11.0.7+10)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.7+10, mixed mode)
Profiler details
image: adoptopenjdk/openjdk8:debianslim-jre
java version:
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.252-b09, mixed mode)
Profiler details
image: adoptopenjdk/openjdk11:debianslim-jre
java version:
openjdk version "11.0.7" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)
Profiler details
andrievsky commentedon Jul 5, 2020
@hamdiallam Hello, would you like to be so kind to assist with the issue?
hamdiallam commentedon Jul 6, 2020
Hey! Sorry for not getting around to this.
Just synced with some more folks internally. Our current builds and runtime environments are targeted towards JDK 8. JDK 11 is currently experimental and we are working towards moving towards JDK11.
I'll spend some time to see if I can reproduce a regression in our integration environment using a JDK 11 build. If so, this is something that will have to be addressed as we move over.
hamdiallam commentedon Jul 6, 2020
Also thanks for the time it took to produce this data. It's super helpful
spockz commentedon Oct 30, 2020
@andrievsky what happens if you use the
SslContextServerEngineFactoryinstead? I noticed that for clients using the client variant improves the performance significantly, see #874.rpless commentedon Dec 4, 2020
Reading through #874, the suggestion seems to be that caching the SslContext might mitigate this issue. A similiar effect can be achieved with the ReloadingNetty4ServerEngineFactory which constructs the SslContext on an interval. Unlike the
SslContextServerEngineFactorythe Reloading engine factory can deal with most forms ofKeyCredentials, so you don't have to go through the pain of manually creating an SslContext. If you expose the ReloadingNetty4ServerEngineFactory (note its private to finagle as far as I can tell), and use it in the benchmark project it acts as a workaround for the issue presented with theJdkServerEngineFactory.When I run it locally I see the following results: https://gist.github.com/rpless/df3245f9b778335fc66ae2a5ea99c280. The
ReloadingNetty4ServerEngineFactoryon Java 11 yields similar results to the defaultJdkServerEngineFactoryon Java 8. I think there is still an issue with the defaultSslServerEngineFactoryon Java 11, however usingSslContextServerEngineFactoryorReloadingNetty4ServerEngineFactorymight be a good workaround.Assuming your SslContext does not change throughout the lifecycle of the process, there are some tradeoffs. The
SslContextServerEngineFactoryis probably better performance-wise because you do not have to reconstruct the context on an interval but requires you to correctly construct an SslContext. TheReloadingNetty4ServerEngineFactoryprobably wastes some effort because it reconstructs it on an interval, but handles creation for all the version of KeyCredentials.