Profiling casuing JVM to crash · Issue #11749 · DataDog/dd-trace-java · GitHub
Skip to content

Profiling casuing JVM to crash #11749

Open
Open
Profiling casuing JVM to crash#11749
Labels
type: bug reportBug report

Description

@snuderl
snuderl
opened on Jun 26, 2026

Tracer Version(s)

1.63.0

Java Version(s)

25.0.2.10.1, 26.0.1.8.1

JVM Vendor

Amazon Corretto

Bug Report

We are seeing tengs of crashes per day across our fleet. Started happening after upgrading 1.58.1 to 1.63.0. We see it on both java 25 and 26.

We also tried setting dd.profiling.ddprof.enabled=false but it didnt fix the crashes. Fully disabling the profiler makes them go away though

javadump.txt

Expected Behavior

Not crashing.

Reproduction Code

No response

Activity

  1. snuderl
    added
    type: bug fixBug fix
    on Jun 26, 2026

  2. rkennke commented on Jun 26, 2026

    @rkennke
    rkennke
    on Jun 26, 2026
    Contributor

    Summary: root cause is the experimental JEP 509 jdk.CPUTimeSample event on JDK 25+/Linux

    TL;DR — This is a JVM-level JFR crash triggered by the new experimental CPU-time profiler that dd-trace-java enables by default on JDK 25+/Linux. It is not a disk-full / out-of-space condition, and it is independent of the native ddprof profiler. The reliable workaround is to disable the jdk.CPUTimeSample / jdk.CPUTimeSamplesLost events.

    What crashed

    Internal Error (jfrStreamWriterHost.inline.hpp:84)
    guarantee(successful_write) failed: Not all the bytes got written, or os::write() failed
    V [libjvm.so] JfrStorage::write() <- JfrRecorderService::flush() <- flushpoint() (JFR Recorder Thread)

    The JVM's JFR recorder thread aborted while flushing buffered events to the on-disk chunk. The chunk writer was handed a corrupt (huge) length and os::write() failed.

    Why it's the profiler / JDK 25+ specific

    • On JDK 25+ and Linux, dd-trace-java switches CPU profiling from the stable jdk.ExecutionSample to the experimental jdk.CPUTimeSample event (JEP 509), at throttle=9ms — see OpenJdkController.java:170-175. This is the only JDK-25-gated behavior change, and the reporter only sees crashes on JDK 25 and 26.
    • jdk.CPUTimeSample is a brand-new, experimental SIGPROF-driven sampler that commits JFR events from contexts JFR historically didn't have to deal with (a signal handler / a non-Java sampler thread). We've traced a path in the JVM's JFR storage where a buffer's pos < top, which underflows to a ~2 GB write length and produces exactly this guarantee crash. (Full JDK-side analysis is being filed upstream.)

    Why the reporter's observations all line up

    • Only JDK 25 and 26 — jdk.CPUTimeSample doesn't exist before JDK 25.
    • dd.profiling.ddprof.enabled=false did not help — expected. The code that would suppress jdk.CPUTimeSample (OpenJdkOngoingRecording.java:58, disableOverriddenEvents) only runs when the native profiler is enabled. With ddprof.enabled=false, the experimental event is left enabled and unguarded.
    • Fully disabling the profiler fixes it — no JFR recording → no jdk.CPUTimeSample → no crash.
    • It is not disk-full — on a JDK with the JDK-8280684 fix (JDK 19+, so Corretto 25/26 both have it), an out-of-space condition aborts gracefully with "Failed to write to jfr stream because no space left on device". The reporter hit the guarantee instead, which only fires for a non-ENOSPC write failure. The host also had ~18 GB free RAM and no swap pressure.

    Workaround (recommended)

    Explicitly disable the experimental event (this override is applied after the JDK-25 switch, so it takes effect):

    -Ddd.profiling.disabled.events=jdk.CPUTimeSample,jdk.CPUTimeSamplesLost

    This keeps the rest of JFR-based profiling working; you lose the JDK-25 CPU-time sampling specifically. (Alternatively, re-enabling the native profiler — dd.profiling.ddprof.enabled=true, the default — also causes dd-trace-java to disable jdk.CPUTimeSample and use ddprof for CPU profiling instead.)

    Suggested dd-trace-java change

    jdk.CPUTimeSample is experimental in JDK 25/26 and can crash the JVM. dd-trace-java should not enable it by default on those versions. Options: gate the JDK-25+ switch behind an opt-in flag while JEP 509 is experimental; and/or disable jdk.CPUTimeSample whenever it won't be consumed (i.e. also when ddprof is disabled), so it's never left running purely as the JFR fallback. A root-cause fix for the underlying JVM bug is being pursued upstream in OpenJDK.

  3. snuderl commented on Jun 30, 2026

    @snuderl
    snuderl
    on Jun 30, 2026
    Author

    @rkennke we now see a different crash :) We are already fully JDK 26 so this report is valid only for that.

    dump.rtf

  4. rkennke commented on Jun 30, 2026

    @rkennke
    rkennke
    on Jun 30, 2026
    Contributor

    @rkennke we now see a different crash :) We are already fully JDK 26 so this report is valid only for that.

    dump.rtf

    Thank you! That looks like a different issue, but we don't know for sure.

    Is this still dd-trace-java 1.63.0 ?

  5. snuderl commented on Jul 1, 2026

    @snuderl
    snuderl
    on Jul 1, 2026
    Author

    yes nothing else changed since the previous report, but it does only happen when profiler is active

  6. rkennke commented on Jul 1, 2026

    @rkennke
    rkennke
    on Jul 1, 2026
    Contributor

    The last crash dump has been very useful to pin down the issue, and we now think it is a memory corruption.
    The mechanism is a use-after-free: a virtual thread caches a ThreadContext whose DirectByteBuffer points into a carrier thread's ProfiledThread/OTEP record; when that carrier's OS thread exits, the native record is freed and reallocated by glibc malloc, but the dangling buffer keeps being written to.

  7. rkennke commented on Jul 1, 2026

    @rkennke
    rkennke
    on Jul 1, 2026
    Contributor

    Fix is in review:

  8. PerfectSlayer
    added
    type: bug reportBug report
    and removed
    type: bug fixBug fix
    on Jul 6, 2026

  9. snuderl commented on Jul 14, 2026

    @snuderl
    snuderl
    on Jul 14, 2026
    Author

    Hey. I did test 1.64.0 and we still get a crash

    crash.txt

  10. rkennke commented on Jul 14, 2026

    @rkennke
    rkennke
    on Jul 14, 2026
    Contributor

    Hey. I did test 1.64.0 and we still get a crash

    crash.txt

    Thank you, that is interesting! Do you happen to have to core dump that the report says might be in /src/core.1 ?

  11. rkennke commented on Jul 14, 2026

    @rkennke
    rkennke
    on Jul 14, 2026
    Contributor

    Hey. I did test 1.64.0 and we still get a crash

    crash.txt

    And logs and/or stdout/stderr might also be useful, if you still have it.

  12. snuderl commented on Jul 14, 2026

    @snuderl
    snuderl
    on Jul 14, 2026
    Author

    Hmm can datadog pick those up? Unless it can we dont really get these now since pod just goes away after it crashes right now

  13. rkennke commented on Jul 16, 2026

    @rkennke
    rkennke
    on Jul 16, 2026
    Contributor

    what is the org-ID/service that is crashing?

  14. rkennke commented on Jul 16, 2026

    @rkennke
    rkennke
    on Jul 16, 2026
    Contributor

    do we know if the crash started to happen with a recent JDK version (e.g. 25 or 26)? or is this not known?

  15. rkennke commented on Jul 16, 2026

    @rkennke
    rkennke
    on Jul 16, 2026
    Contributor

    @snuderl Would you be able to run that service with a custom-built JDK26? I'd add some debug machinery that would give us a more useful stack-trace when it crashes again.

  16. snuderl commented on Jul 16, 2026

    @snuderl
    snuderl
    on Jul 16, 2026
    Author

    I dont have a good answer on when exactly is started happening. We were running into virtual thread deadlocks, so we had to upgrade jdk to 26, but that also meant upgrading datadog at the same time.

    For custom build, i dont think we are super comfortable with that and we also dont have a great staging environment where I could quickly reproduce.

  17. rkennke commented on Jul 16, 2026

    @rkennke
    rkennke
    on Jul 16, 2026
    Contributor

    I dont have a good answer on when exactly is started happening. We were running into virtual thread deadlocks, so we had to upgrade jdk to 26, but that also meant upgrading datadog at the same time.

    Ok, that's what I suspected.

    For custom build, i dont think we are super comfortable with that and we also dont have a great staging environment where I could quickly reproduce.

    In that case, the best that I can do is to fix the issue opportunistically - based on what we know - in upstream JDK. And that fix would land in JDK28. Without further testing and justification (e.g. 'we tried it in our failing workload, and the problem no longer shows up'), it would be hard to argue for a backport, and in any case, such a backport would land in JDK 25, not 26. This means you'd get the fix with JDK 28 GA earliest, which is March 2027.

    It also currently seems hard to come up with a mitigation on our side. So far this looks very much like a JDK bug, which is lingering since at least JDK 15, and something in your workload triggers it enough to go off the rails, the most likely suspect currently being high-rate JFR event churn, probably caused and/or amplified by virtual threads (but that is pure speculation).

    Being able to run with a custom JDK26 (preferably not in prod) would be immensely helpful.

  18. rkennke commented on Jul 16, 2026

    @rkennke
    rkennke
    on Jul 16, 2026
    Contributor

    What's happening — The JVM crashes on the "JFR Recorder Thread" with guarantee(successful_write) failed in JfrStorage::write(). Not disk-full, not a Datadog bug: a latent JVM JFR data race where a buffer's pos - top transiently underflows to a multi-GB size passed to os::write. Filed upstream as JDK-8387409. CPU profiling (ddprof native) is a separate writer and isn't involved.

    Fix status — JVM-side only. JDK 26 is non-LTS with a closed update train, so a real fix means a custom/patched JDK or a train that carries the backport. The steps below reduce recurrence, they don't eliminate it.

    Mitigation (deploy together):

    1. -XX:FlightRecorderOptions=threadbuffersize=1m (default 8k) — bigger buffers fill/promote far less often.
    2. Attach the override: -Ddd.profiling.jfr-template-override-file=/path/to/jfr-churn-mitigation-override.jfp (or DD_PROFILING_JFR_TEMPLATE_OVERRIDE_FILE). You may want to adjust the timings to your needs.
    3. Escalation if needed: hard-disable via -Ddd.profiling.disabled.events=jdk.CPUTimeSample,jdk.CPUTimeSamplesLost,jdk.SocketRead,jdk.SocketWrite,jdk.FileRead,jdk.FileWrite.

    Expectations — cuts trigger rate substantially; a burst can still hit it; there's a churn floor only the JVM fix removes.

    Separate item — one NoSuchMethodError: jdk.jfr.internal.JVM.getTicksFrequency()J, not from dd-agent → an app/third-party dependency with a stale JDK-internal reflection path on JDK 26; grep their deps for jdk.jfr.internal.JVM. (The MethodHandle$Holder NSMEs are benign — ignore.)

    jfr-churn-mitigation-override.jfp:

    # JFR template override — mitigation for the JFR chunk-writer crash (JDK-8387409)
# Raises thresholds/throttles on the highest-rate, stack-trace-bearing events to
# reduce thread-local buffer churn. MITIGATION, not a fix. Merged over dd.jfp.
# Pair with: -XX:FlightRecorderOptions=threadbuffersize=1m  (default is 8k)

# Network I/O: top churn source for a network service (was 10 ms)
jdk.SocketRead#threshold=100 ms
jdk.SocketWrite#threshold=100 ms

# File I/O (was 10 ms)
jdk.FileRead#threshold=50 ms
jdk.FileWrite#threshold=50 ms
jdk.FileForce#threshold=50 ms

# Lock/monitor events (was 10 ms)
jdk.JavaMonitorEnter#threshold=50 ms
jdk.JavaMonitorInflate#threshold=50 ms

# Allocation sampling: fewer stack traces/sec (was 300/s)
jdk.ObjectAllocationSample#throttle=50/s

# Datadog product events (was 10 ms) — HIGHER TRADEOFF: these feed the
# trace<->profile timeline / endpoint hotspots; raising them lowers resolution.
datadog.Scope#threshold=50 ms
datadog.Timeline#threshold=50 ms

  19. rkennke commented on Jul 22, 2026

    @rkennke
    rkennke
    on Jul 22, 2026
    Contributor

    @snuderl do you happen to have -Ddd.profiling.start-force-first=true anywhere in your configuration? I don't see it on the cmd-line, but maybe it is in /src/integrations/datadog/dd-java-agent.properties ? If so, that might explain why the 1.64 fix did not help.

  20. snuderl commented on Jul 24, 2026

    @snuderl
    snuderl
    on Jul 24, 2026
    Author

    nope, nothing really related to profiler in there

  21. Meemaw commented on Sep 21, 2026

    @Meemaw
    Meemaw
    on Sep 21, 2026
    Contributor

    Hey @rkennke. Do you believe this was fixed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bug reportBug report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions