Java Agent for OpenTracing

Automatically instruments 3rd-party libraries in Java applications

_{Note: The coverage statistic is not correct, because Jacoco cannot properly instrument code that is instrumented at the bytecode level.}

What is SpecialAgent?

SpecialAgent automatically instruments 3rd-party libraries in Java applications. The architecture of SpecialAgent was designed to involve contributions from the community, whereby its platform integrates and automates OpenTracing Instrumentation Plugins written by individual contributors. In addition to Instrumentation Plugins, the SpecialAgent also supports Tracer Plugins, which connect an instrumented runtime to OpenTracing-compliant tracer vendors, such as LightStep or Jaeger. Both the Instrumentation Plugins and the Tracer Plugins are decoupled from SpecialAgent -- i.e. neither kinds of plugins need to know anything about SpecialAgent. At its core, the SpecialAgent is itself nothing more than an engine that abstracts the functionality for automatic installation of Instrumentation Plugins, and then connecting them to Tracer Plugins. A benefit of this approach is that the SpecialAgent intrinsically embodies and encourages community involvement.

In additiona to its engine, the SpecialAgent packages a set of pre-supported Instrumentation Plugins and Tracer Plugins.

Table of Contents

  1 Introduction
  2 Quick Start
    2.1 Installation
       2.1.1 In Application
           2.1.1.1 Stable
           2.1.1.2 Development
       2.1.2 For Development
           2.1.2.1 Building
    2.2 Usage
       2.2.1 Static Attach
       2.2.2 Dynamic Attach
       2.2.3 Static Deferred Attach
  3 Configuration
    3.1 Overview
    3.2 Properties
    3.3 Selecting the Tracer Plugin
    3.4 Disabling Instrumentation Plugins
       3.4.1 Disabling All Instrumentation Plugins
       3.4.2 Disabling One Instrumentation Plugin
       3.4.3 Disabling AgentRules of an Instrumentation Plugin
    3.5 Disabling Tracer Plugins
  4 Definitions
    4.1 SpecialAgent
    4.2 Tracer
    4.3 Tracer Plugin
    4.4 Instrumentation Plugin
    4.5 Instrumentation Rule
  5 Objectives
    5.1 Goals
    5.2 Non-Goals
  6 Supported Plugins
    6.1 Instrumentation Plugins
    6.2 Tracer Plugins
    6.3 Instrumented libraries by existing rules
  7 Credits
  8 Contributing
  9 License

1 Introduction

This file contains the operational instructions for the use and development of SpecialAgent.

2 Quick Start

When SpecialAgent attaches to an application, either statically or dynamically, it will automatically load the Instrumentation Plugins explicitly specified as dependencies in its POM (Project Object Model).

Any exception that occurs during the execution of the bootstrap process will not adversely affect the stability of the target application. It is, however, possible that the Instrumentation Plugin code may result in exceptions that are not properly handled, and could destabilize the target application.

2.1 Installation

The SpecialAgent has 2 artifacts: main and test. These artifacts are built by Maven, and can be obtained by cloning this repository and following the Building instructions, or downloading directly from Maven's Central Repository.

2.1.1 In Application

To use the SpecialAgent on an application, first download the JAR:

2.1.1.1 Stable

The latest stable release is: 1.3.6.

wget -O opentracing-specialagent-1.3.6.jar "http://central.maven.org/maven2/io/opentracing/contrib/specialagent/opentracing-specialagent/1.3.6/opentracing-specialagent-1.3.6.jar"

2.1.1.2 Development

The latest development release is: 1.3.7-SNAPSHOT.

wget -O opentracing-specialagent-1.3.7-SNAPSHOT.jar "https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=io.opentracing.contrib.specialagent&a=opentracing-specialagent&v=LATEST"

This is the main artifact that contains within it the Instrumentation Plugins from the opentracing-contrib organization for which Instrumentation Rules have been implemented. This JAR can be specified as the -javaagent target for Static Attach to an application. This JAR can also be executed, standalone, with an argument representing the PID of a target process to which it should dynamically attach. Please refer to Usage section for usage instructions.

2.1.2 For Development

For development of Instrumentation Plugins, import the opentracing-specialagent-api and test-jar of the opentracing-specialagent.

<dependency>
  <groupId>io.opentracing.contrib.specialagent</groupId>
  <artifactId>opentracing-specialagent-api</artifactId>
  <version>1.3.6</version> <!--version>1.3.7-SNAPSHOT<version-->
  <scope>provided</scope>
</dependency>
<dependency>
  <groupId>io.opentracing.contrib.specialagent</groupId>
  <artifactId>opentracing-specialagent</artifactId>
  <version>1.3.6</version> <!--version>1.3.7-SNAPSHOT<version-->
  <type>test-jar</type>
  <scope>test</scope>
</dependency>

This is the test artifact that contains within it the AgentRunner, which is a JUnit runner class provided for testing of the ByteBuddy auto-instrumentation rules. This JAR does not contain within it any Instrumentation Plugins themselves, and is only intended to be applied to the test phase of the build lifecycle of a single plugin for an Instrumentation Plugin implementation. For direction with the AgentRunner, please refer to the opentracing-specialagent-api module.

2.1.2.1 Building

Prerequisite: The SpecialAgent requires Oracle Java to build. Thought the SpecialAgent supports OpenJDK for general application use, it only supports Oracle Java for building and testing.

The SpecialAgent is built in 2 passes that utilize different profiles:

1. The default profile is used for development of Instrumentation Rules. It builds and runs tests for each rule, but does not bundle the rules into opentracing-specialagent-1.3.6.jar

   To run this profile:

   mvn clean install

2. The assemble profile is used to bundle the Instrumentation Rules into opentracing-specialagent-1.3.6.jar. It builds each rule, but does not run tests. Once the build with the assemble profile is finished, the opentracing-specialagent-1.3.6.jar will contain the built rules inside it.

   Note: If you do not run this step, the opentracing-specialagent-1.3.6.jar from the previous step will not contain any Instrumentation Plugins within it!

   Note: It is important to not run Maven's clean lifecycle when executing the assemble profile.

   To run this profile:

   mvn -Dassemble package

• For a one-line build command to build SpecialAgent, its rules, run all tests, and create the assemble package:

  mvn clean install && mvn -Dassemble package

2.2 Usage

The SpecialAgent uses Java’s Instrumentation mechanism to transform the behavior of a target application. The entrypoint into the target application is performed via Java’s Agent convention. SpecialAgent supports both Static Attach and Dynamic Attach.

2.2.1 Static Attach

Statically attaching to a Java application involves the use of the -javaagent vm argument at the time of startup of the target Java application. The following command can be used as an example:

java -javaagent:opentracing-specialagent-1.3.6.jar -jar MyApp.jar

This command statically attaches SpecialAgent into the application in myapp.jar.

2.2.2 Dynamic Attach

Dynamically attaching to a Java application involves the use of a running application’s PID, after the application’s startup. The following commands can be used as an example:

1. Call this to obtain the PID of the target application:

   jps

2. For jdk1.8

   java -Xbootclasspath/a:$JAVA_HOME/lib/tools.jar -jar opentracing-specialagent-1.3.6.jar <PID>

3. For jdk9+

   java -jar opentracing-specialagent-1.3.6.jar <PID>

Note: If you encounter an exception stating Unable to open socket file, make sure the attaching VM is executed with the same permissions as the target VM.

2.2.3 Static Deferred Attach

With Static Attach, the application is executed with the -javaagent argument, and the agent initialization occurs before the application is started. This mode requires 1 command from the command line.

With Dynamic Attach, the application is allowed to start first, afterwhich an agent VM is dynamically attached to the application's PID. This mode requires 2 commands from the command line: the first for the application, and the second for the agent VM.

With Static Deferred Attach, the application is executed with the -javaagent argument, but the agent initialization is deferred until the application is started. This mode requires 1 command from the command line, and is designed specifically for Spring runtimes that have complex initialization lifecycles. The SpecialAgent relies on the ContextRefreshedEvent to signify that the application is ready, and thus to cue agent initialization. This approach works for all versions of Spring and Spring Boot.

The following command can be used as an example:

java -javaagent:opentracing-specialagent-1.3.6.jar -Dsa.spring -jar MySpringApp.jar

3 Configuration

3.1 Overview

The SpecialAgent exposes a simple pattern for configuration of SpecialAgent, the Instrumentation Plugins, as well as Tracer Plugins. The configuration pattern is based on system properties, which can be defined on the command-line, in a properties file, or in @AgentRunner.Config for JUnit tests:

1. Properties passed on the command-line via -D${PROPERTY}=... override same-named properties defined in layers below...

2. The @AgentRunner.Config annotation allows one to define log level and transformation event logging settings. Properties defined in the @Config annotation override same-named properties defined in layers below...

3. The -Dsa.config=${PROPERTIES_FILE} command-line argument can be specified for SpecialAgent to load property names from a ${PROPERTIES_FILE}. Properties defined in the ${PROPERTIES_FILE} override same-named properties defined in the layer below...

4. The SpecialAgent has a default.properties file that defines default values for properties that need to be defined.

3.2 Properties

The following properties are supported by all instrumentation plugins:

1. Logging:

   The -Dsa.log.level system property can be used to set the logging level for SpecialAgent. Acceptable values are: SEVERE, WARNING, INFO, CONFIG, FINE, FINER, or FINEST, or any numerical log level value is accepted also. The default logging level is set to WARNING.

   The -Dsa.log.events system property can be used to set the re/transformation events to log: DISCOVERY, IGNORED, TRANSFORMATION, ERROR, COMPLETE. The property accepts a comma-delimited list of event names. By default, the ERROR event is logged (only when run with AgentRunner).

   The -Dsa.log.file system property can be used to set the logging output file for SpecialAgent.

2. Verbose Mode: -Dsa.instrumentation.plugins.verbose, -Dsa.instrumentation.plugin.${PLUGIN_NAME}.verbose

   Sets verbose mode for all or one plugin (Default: false). This property can also be set in an AgentRunner JUnit test with the @AgentRunner.Config(verbose=true) for all tests in a JUnit class, or @AgentRunner.TestConfig(verbose=true) for an individual JUnit test method.

   Concurrent plugin supports verbose mode which is disabled by default. To enable set sa.concurrent.verbose=true. In non verbose mode parent span context (if exists) is propagating to task execution. In verbose mode parent span is always created on task submission to executor and child span is created when task is started.

3. Skip fingerprint verification: -Dsa.fingerprint.skip

   Tells the SpecialAgent to skip the fingerprint verification when linking Instrumentation Plugins into class loaders. This option allows one to work around an unexpected fingerprint verification failure, which can happen in complex runtimes that do not contain all class definitions on the class path. It must be noted, however, that if the fingerprint verification is disabled, the SpecialAgent will indiscriminately install all plugins regardless of library version compatibility issues, which may lead to NoClassDefFoundError, IllegalAccessError, AbstractMethodError, LinkageError, etc.

3.3 Selecting the Tracer Plugin

The SpecialAgent supports OpenTracing-compatible tracers. There are 2 ways to connect a tracer to the SpecialAgent runtime:

1. Internal Tracer Plugins

   The SpecialAgent includes the following Tracer Plugins:

   1. Jaeger Tracer Plugin
   2. LightStep Tracer Plugin

   The -Dsa.tracer=${TRACER_PLUGIN} property is used on the command-line to specify which Tracer Plugin will be used. The value of ${TRACER_PLUGIN} is the short name of the Tracer Plugin, i.e. jaeger or lightstep.

2. External Tracer Plugins

   The SpecialAgent allows external Tracer Plugins to be attached to the runtime.

   The -Dsa.tracer=${TRACER_JAR} property is used on the command-line to specify the JAR path of the Tracer Plugin to be used. The ${TRACER_JAR} must be a JAR that conforms to the TracerFactory API of the TracerResolver project.

NOTE: If a tracer is not specified with the -Dsa.tracer=... property, the SpecialAgent will present a warning in the log that states: Tracer NOT RESOLVED.

3.4 Disabling Instrumentation Plugins

The SpecialAgent has all of its Instrumentation Plugins enabled by default, and allows them to be disabled.

3.4.1 Disabling All Instrumentation Plugins

To disable all instrumentation plugins, specify a system property, either on the command-line or in the properties file referenced by -Dconfig=${PROPERTIES_FILE}.

sa.instrumentation.plugins.disable

3.4.2 Disabling One Instrumentation Plugin

To disable an individual instrumentation plugin, specify a system property, either on the command-line or in the properties file referenced by -Dconfig=${PROPERTIES_FILE}.

sa.instrumentation.plugin.${PLUGIN_NAME}.disable

The value of ${PLUGIN_NAME} is the name of the plugin as declared in the plugin's POM (Project Object Model). The names follow a consice pattern, such as okhttp for the specialagent-okhttp plugin artifactId, and web-servlet-filter for the specialagent-web-servlet-filter plugin artifactId.

3.4.3 Disabling AgentRules of an Instrumentation Plugin

To disable an individual AgentRule of an instrumentation plugin, specify a system property, either on the command-line or in the properties file referenced by -Dconfig=${PROPERTIES_FILE}.

sa.instrumentation.plugin.${PLUGIN_NAME}.${AGENT_RULE_SIMPLE_CLASS_NAME}.disable

The value of ${AGENT_RULE_SIMPLE_CLASS_NAME} is the simple class name of the AgentRule subclass that is to be disabled.

3.5 Disabling Tracer Plugins

The SpecialAgent has all of its Tracer Plugins enabled by default, and allows them to be disabled.

To disable all tracer plugins, specify a system property, either on the command-line or in the properties file referenced by -Dconfig=${PROPERTIES_FILE}.

sa.tracer.plugins.disable

To disable an individual tracer plugin, specify a system property, either on the command-line or in the properties file referenced by -Dconfig=${PROPERTIES_FILE}.

sa.tracer.plugin.${SHORT_NAME}.disable

The value of ${SHORT_NAME} is the short name of the plugin, such as lightstep or jaeger.

4 Definitions

The following terms are used throughout this documentation.

4.1 SpecialAgent

The SpecialAgent is software that attaches to Java applications, and automatically instruments 3rd-party libraries integrated in the application. The SpecialAgent uses the OpenTracing API for Instrumentation Plugins that instrument 3rd-party libraries, as well as Tracer Plugins that implement OpenTracing tracer service providers. Both the Instrumentation Plugins, as well as the Tracer Plugins are open-source, and are developed and supported by the OpenTracing community.

The SpecialAgent supports Oracle Java and OpenJDK. When building SpecialAgent from source, only Oracle Java is supported.

4.2 Tracer

Service provider of the OpenTracing standard, providing an implementation of the io.opentracing.Tracer interface.

Examples:

• Jaeger Tracer
• LightStep Tracer
• MockTracer

_{Tracers are not coupled to the SpecialAgent.}

4.3 Tracer Plugin

A bridge providing automatic discovery of Tracers in a runtime instrumented with the OpenTracing API. This bridge implements the TracerFactory interface of TracerResolver, and is distributed as a single "fat JAR" that can be conveniently added to the classpath of a Java process.

_{Tracer Plugins are not coupled to the SpecialAgent.}

4.4 Instrumentation Plugin

An OpenTracing Instrumentation project that exist as individual repositories under opentracing-contrib.

Examples:

• opentracing-contrib/java-okhttp
• opentracing-contrib/java-jdbc
• opentracing-contrib/java-jms

_{Instrumentation Plugins are not coupled to the SpecialAgent.}

4.5 Instrumentation Rule

A submodule of the SpecialAgent that implements the auto-instrumentation rules for Instrumentation Plugins via the opentracing-specialagent-api.

Examples:

• rules/specialagent-okhttp
• rules/specialagent-jdbc
• rules/specialagent-jms-1
• rules/specialagent-jms-2

_{Instrumentation Rules are coupled to the SpecialAgent.}

5 Objectives

5.1 Goals

1. The SpecialAgent must allow any Instrumentation Plugin available in opentracing-contrib to be automatically installable in applications that utilize a 3rd-party library for which an Instrumentation Plugin exists.
2. The SpecialAgent must automatically install the Instrumentation Plugin for each 3rd-party library for which a module exists, regardless in which class loader the 3rd-party library is loaded.
3. The SpecialAgent must not adversely affect the runtime stability of the application on which it is intended to be used. This goal applies only to the code in the SpecialAgent, and cannot apply to the code of the Instrumentation Plugins made available in opentracing-contrib.
4. The SpecialAgent must Static Attach and Dynamic Attach to applications running on JVM versions 1.7, 1.8, 9, and 11.
5. The SpecialAgent must implement a lightweight test methodology that can be easily applied to a module that implements instrumentation for a 3rd-party library. This test must simulate:
   1. Launch the test in a process simulating the -javaagent vm argument that points to the SpecialAgent (in order to test auto-instrumentation functionality).
   2. Elevate the test code to be executed from a custom class loader that is disconnected from the system class loader (in order to test bytecode injection into an isolated class loader that cannot resolve classes on the system classpath).
   3. Allow tests to specify their own Tracer instances via GlobalTracer, or initialize a MockTracer if no instance is specified. The test must provide a reference to the Tracer instance in the test method for assertions with JUnit.
6. The SpecialAgent must provide a means by which Instrumentation Plugins can be configured before use on a target application.

5.2 Non-Goals

1. The SpecialAgent is not designed to modify application code, beyond the installation of Instrumentation Plugins. For example, there is no facility for dynamically tracing arbitrary code.

6 Supported Plugins

6.1 Instrumentation Plugins

The following plugins have Instrumentation Rules implemented.

1. OkHttp3 [3.5.0,]
2. JDBC API (java.sql) [3.1,]
3. Concurrent API (java.util.concurrent)
4. Java Servlet API (javax.servlet) [2.3,] []
5. Mongo Driver [3.9.0,]
6. Apache Camel [2.24.0,]
7. AWS SDK [1.11.79,]
8. AWS SDK 2 [2.1.4,]
9. Cassandra Driver [3.0.0,]
10. JMS API (javax.jms v1 & v2)
11. JMS Spring [5.0.0.RELEASE,]
12. Elasticsearch Client [6.4.0,]
13. RxJava 2 [2.1.0,]
14. Kafka Client [1.1.0,]
15. Kafka Spring [2.2.0.RELEASE,]
16. AsyncHttpClient [2.7.0,]
17. RabbitMQ Client [5.0.0,]
18. RabbitMQ Spring [2.0.0.RELEASE,]
19. Thrift [0.12.0,]
20. GRPC [1.6.0,]
21. Jedis Client [2.7.0,]
22. Apache HttpClient [4.4,]
23. Lettuce Client [5.1.0.RELEASE,]
24. Spring Web [5.0.0.RELEASE,]
25. Spring Web MVC [5.0.0.RELEASE,]
26. Spring WebFlux [5.1.0.RELEASE,]
27. Spring WebSocket STOMP [5.1.0.RELEASE,]
28. Redisson [3.6.0,]
29. Grizzly HTTP Server [2.3.35,]
30. Grizzly AsyncHttpClient [1.15,]
31. Reactor [3.2.3.RELEASE,]
32. Hazelcast [3.7,]
33. Spymemcached [2.11.0,]
34. Feign [9.0.0,]
35. Zuul [1.0.0,]
36. Spring @Async and @Scheduled [5.0.0.RELEASE,]
37. Spring Messaging [5.1.1.RELEASE,]
38. Spring 3 Web [3.0.3.RELEASE,]
39. Spring 3 Web MVC [3.0.2.RELEASE,]
40. JAX-RS Client [2.0,]

6.2 Tracer Plugins

The following OpenTracing tracer service providers have Tracer Plugins implemented. Here is a demo.

1. Jaeger Tracer Plugin, Configuration reference
2. LightStep Tracer Plugin

6.3 Instrumented libraries by existing rules

The following libraries are instrumented by existing Instrumentation Rules.

1. Solr Client
2. JDBI
3. Hystrix
4. Spring Cloud

7 Credits

Thank you to the following contributors for developing instrumentation plugins:

• Sergei Malafeev
• Jose Montoya
• Przemyslaw Maciolek

Thank you to the following contributors for developing tracer plugins:

• Carlos Alberto Cortez

Thank you to the following developers for filing issues and helping us fix them:

• Louis-Etienne
• Marcos Trejo Munguia
• @kaushikdeb
• @deepakgoenka
• @etsangsplk
• @Vovan2006

Thank you to the following individuals for noticing typographic errors and sending PRs:

• Daniel Rodriguez Hernandez
• qudongfang

Finally, thanks for all of the feedback! Please share your comments as an issue!

8 Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

9 License

This project is licensed under the Apache 2 License - see the LICENSE.txt file for details.