Java Agent for OpenTracing,
with automatic instrumentation

Automatically instruments 3rd-party libraries in Java applications

Overview

Java SpecialAgent automatically instruments Java applications to produce trace events via the OpenTracing API. This file contains the operational instructions for the use of Java SpecialAgent.

Supported Instrumentation Plugins

1. OkHttp3
2. java.util.Concurrent
3. JDBC
4. Java Web Servlet Filter
5. Mongo Driver
6. Apache Camel

Operation

When Java SpecialAgent attaches to an application, either statically or dynamically, it will automatically load the OpenTracing instrumentation plugins explicitly specified as dependencies in its POM.

Any exception that occurs during the execution of the bootstrap process will not adversely affect the stability of the target application. It is, however, possible that the instrumentation plugin code may result in exceptions that are not properly handled, and would destabilize the target application.

Goals

1. The Java SpecialAgent must allow any Java instrumentation plugin available in opentracing-contrib to be automatically installable in applications that utilize a 3rd-party library for which an instrumentation plugin exists.
2. The Java SpecialAgent must automatically install the instrumentation plugin for each 3rd-party library for which a plugin exists, regardless in which ClassLoader the 3rd-party library is loaded.
3. The Java SpecialAgent must not adversely affect the runtime stability of the application on which it is intended to be used. This goal applies only to the code in the Java SpecialAgent, and cannot apply to the code of the instrumentation plugins made available in opentracing-contrib.
4. The Java SpecialAgent must support static and dynamic attach to applications running on JVM versions 1.7 to latest.
5. The Java SpecialAgent must implement a lightweight test methodology that can be easily applied to a module that implements instrumentation for a 3rd-party plugin. This test must simulate:
   1. Launch the test in a process with the -javaagent vm argument that points to the Java SpecialAgent (in order to test automatic instrumentation functionality of the otarules.btm file).
   2. Elevate the test code to be executed from a custom ClassLoader that is disconnected from the system ClassLoader (in order to test bytecode injection into an isolated ClassLoader that cannot resolve classes on the system classpath).
   3. Initialize a MockTracer as GlobalTracer, and provide a reference to the Tracer instance in the test method for assertions with JUnit. The Java SpecialAgent must provide a means by which instrumentation plugins can be configured before use on a target application.

Non-Goals

1. The Java SpecialAgent is not designed to modify application code, beyond the installation of OpenTracing instrumentation plugins. For example, there is no facility for dynamically tracing arbitrary code.

Installation

The Java SpecialAgent is built with Maven, and produces 2 artifacts:

1. opentracing-specialagent-<version>.jar

   This is the main artifact that contains within it all applicable instrumentation plugins from the opentracing-contrib project. This JAR can be specified as the -javaagent target for static attach to an application. This JAR can also be executed, standalone, with an argument representing the PID of a target process to which it should dynamically attach.

2. opentracing-specialagent-<version>-tests.jar

   This is the test artifact that contains within it the AgentRunner, which is a JUnit runner class provided for testing of otarules.btm files in instrumentation plugins. This JAR does not contain within it any instrumentation plugins themselves, and is only intended to be applied to the test phase of the build lifecycle of a single instrumentation plugin implementation. This JAR can be used in the same manner as the main JAR, both for static and dynamic attach.

Usage

The Java SpecialAgent uses Java’s Instrumentation interface to transform the behavior of a target application. The entrypoint into the target application is performed via Java’s Agent convention. Java SpecialAgent supports both static and dynamic attach.

Static Attach

Statically attaching to a Java application involves the use of the -javaagent vm argument at the time of startup of the target Java application. The following command can be used as an example:

java -javaagent:opentracing-specialagent.jar -jar myapp.jar

This command statically attaches Java SpecialAgent into the application in myapp.jar.

Dynamic Attach

Dynamically attaching to a Java application involves the use of a running application’s PID, after the application’s startup. The following commands can be used as an example:

jps # Call this to obtain the PID of the target application
java -jar opentracing-specialagent.jar <PID> # Replace <PID> with the PID from jps

Test Usage

The Java SpecialAgent uses the JUnit Runner API to implement a lightweight test methodology that can be easily applied to modules that implement instrumentation for 3rd-party plugins. This runner is named AgentRunner, and allows developers to implement tests using vanilla JUnit patterns, transparently providing the following behavior:

1. Launch the test in a process with the -javaagent vm argument that points to the Java SpecialAgent (in order to test automatic instrumentation functionality of the otarules.btm file).
2. Elevate the test code to be executed from a custom ClassLoader that is disconnected from the system ClassLoader (in order to test bytecode injection into an isolated ClassLoader that cannot resolve classes on the system classpath).
3. Initialize a MockTracer as GlobalTracer, and provide a reference to the Tracer instance in the test method for assertions with JUnit.

The AgentRunner is available in the test jar of the Java SpecialAgent module. It can be imported with the following dependency spec:

<dependency>
  <groupId>io.opentracing.contrib</groupId>
  <artifactId>opentracing-specialagent</artifactId>
  <version>${project.version}</version>
  <type>test-jar</type>
  <scope>test</scope>
</dependency>

To use the AgentRunner in a JUnit test class, provide the following annotation to the class in question:

@RunWith(AgentRunner.class)

In addition to the @RunWith annotation, each method annotated with @Test must declare a parameter of type MockTracer, as such:

@Test
public void test(MockTracer tracer) {}

Similarly, each method annotated with @Before, @After, @BeforeClass, and @AfterClass must declare a parameter of type MockTracer, as such:

@BeforeClass
public static void beforeClass(MockTracer tracer) {}

@AfterClass
public static void afterClass(MockTracer tracer) {}

@Before
public void before(MockTracer tracer) {}

@After
public void after(MockTracer tracer) {}

The MockTracer class can be referenced by importing the following dependency spec:

<dependency>
  <groupId>io.opentracing</groupId>
  <artifactId>opentracing-mock</artifactId>
  <version>${version.opentracing-mock}</version>
  <scope>test</scope>
</dependency>

Upon execution of the test class, in either the IDE or with Maven, the AgentRunner will execute each test method via the 3 step workflow described above.

Configuring AgentRunner

The AgentRunner can be configured via the @AgentRunner.Config(...) annotation. The annotation supports the following properties:

1. debug: If set to true, FINEST level root logging will be enabled. Default: false.
2. verbose: If set to true, Byteman verbose logging will be enabled. Default: false.
3. isolateClassLoader: If set to true, tests will be run from a ClassLoader that is isolated from the system ClassLoader. If set to false, tests will be run from the system ClassLoader. Default: true.

Developing Instrumentation Plugins for SpecialAgent

The opentracing-contrib repository contains 40+ OpenTracing instrumentation plugins for Java. Only a handful of these plugins are currently supported by SpecialAgent.

If you are interested in contributing to the SpecialAgent project by integrating support for existing plugins in the opentracing-contrib repository, or by implementing a new plugin with support for SpecialAgent, the following guide is for you:...

Implementing the Instrumentation Logic

The opentracing-contrib repository contains instrumentation plugins for a wide variety of 3rd-party libraries, as well as Java standard APIs. The plugins instrument a 3rd-party library of interest by implementing custom library-specific hooks that integrate with the OpenTracing API. To see examples, explore projects named with the prefix java-... in the opentracing-contrib repository.

Implementing the Auto-Instrumentation Script

The SpecialAgent uses Byteman to perform bytecode injection for the purpose of auto-instrumentation. Instrumentation scripts mustb e named otarules.btm, and be placed as a resource in the default package. Please refer to the following scripts as examples:

1. otarules.btm for OkHttp3
2. otarules.btm for java.util.Concurrent
3. otarules.btm for JDBC
4. otarules.btm for Java Web Servlet Filter
5. otarules.btm for Mongo Driver
6. otarules.btm for Apache Camel

Packaging

The SpecialAgent has specific requirements for packaging of instrumentation plugins:

1. If the library being instrumented is 3rd-party (i.e. it does not belong to the standard Java APIs), then the dependency artifacts for the library must be non-transitive (i.e. declared with <scope>test</scope>, or with <scope>provided</scope>).
   • The dependencies for the 3rd-party libraries are not necessary when the plugin is applied to a target application, as the application must already have these dependencies for the plugin to be used.
   • Declaring the 3rd-party libraries as non-transitive dependencies greatly reduces the size of the SpecialAgent package, as all of the instrumentation plugins as contained within it.
   • If 3rd-party libraries are not declared as non-transitive, there is a risk that target applications may experience class loading exceptions due to inadvertant loading of incompatibile classes.
   • Many of the currently implemented instrumentation plugins do not declare the 3rd-party libraries which they are instrumenting as non-transitive. In this case, an <exclude> tag must be specified for each 3rd-party artifact dependency when referring to the instrumentation plugin artifact. An example of this can be seen with the instrumentation plugin for the Mongo Driver here.
2. The package must contain a fingerprint.bin file. This file provides the SpecialAgent with a fingerprint of the 3rd-party library that the plugin is instrumenting. This fingerprint allows the SpecialAgent to determine if the plugin is compatible with the relevant 3rd-party library in a target application. To generate this file, include the following plugin in the project's POM:

   <plugin>
     <groupId>io.opentracing.contrib</groupId>
     <artifactId>specialagent-maven-plugin</artifactId>
     <version>0.0.1</version>
     <executions>
       <execution>
         <goals>
           <goal>fingerprint</goal>
         </goals>
         <phase>generate-resources</phase>
         <configuration>
           <destFile>${project.build.directory}/generated-resources/fingerprint.bin</destFile>
         </configuration>
       </execution>
     </executions>
   </plugin>

3. The package must contain a dependencies.tgf file. This file allows the SpecialAgent to distinguish instrumentation plugin dependency JARs from test JARs and API JARs. To generate this file, include the following plugin in the project's POM:

   <plugin>
     <groupId>org.apache.maven.plugins</groupId>
     <artifactId>maven-dependency-plugin</artifactId>
     <executions>
       <execution>
         <goals>
           <goal>tree</goal>
         </goals>
         <phase>generate-resources</phase>
         <configuration>
           <outputType>tgf</outputType>
           <outputFile>${project.build.directory}/generated-resources/dependencies.tgf</outputFile>
         </configuration>
       </execution>
     </executions>
   </plugin>

Testing

The SpecialAgent provides a convenient methodolofy for testing of the auto-instrumentation of plugins via AgentRunner. Please refer to the section on Test Usage for instructions.

Including the Instrumentation Plugin in the SpecialAgent

Instrumentation plugins must be explicitly packaged into the main JAR of the SpecialAgent. Please refer to the <id>deploy</id> profile in the POM for an example of the usage.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

License

This project is licensed under the Apache 2 License - see the LICENSE.txt file for details.