Automatically instruments 3rd-party libraries in Java applications
Note: The coverage statistic is not correct, because Jacoco cannot properly instrument code that is instrumented at the bytecode level.
SpecialAgent automatically instruments 3rd-party libraries in Java applications. The architecture of SpecialAgent was designed to involve contributions from the community, whereby its platform integrates and automates OpenTracing Instrumentation Plugins written by individual contributors. In addition to Instrumentation Plugins, the SpecialAgent also supports Tracer Plugins, which connect an instrumented runtime to OpenTracing-compliant tracer vendors, such as LightStep, Wavefront, or Jaeger. Both the Instrumentation Plugins and the Tracer Plugins are decoupled from SpecialAgent -- i.e. neither kinds of plugins need to know anything about SpecialAgent. At its core, the SpecialAgent is itself nothing more than an engine that abstracts the functionality for automatic installation of Instrumentation Plugins, and then connecting them to Tracer Plugins. A benefit of this approach is that the SpecialAgent intrinsically embodies and encourages community involvement.
In addition to its engine, the SpecialAgent packages a set of pre-supported Instrumentation Plugins and Tracer Plugins.
1 Introduction
2 Quick Start
2.1 Installation
2.1.1 In Application
2.1.1.1 Stable
2.1.1.2 Development
2.1.2 For Development
2.1.2.1 Instrumentation Plugins
2.1.2.2 Tracer Plugins
2.1.2.2.1 Short Name
2.2 Usage
2.2.1 Static Attach
2.2.2 Dynamic Attach
2.2.3 Static Deferred Attach
3 Configuration
3.1 Overview
3.2 Properties
3.3 Selecting the Tracer Plugin
3.4 Disabling Instrumentation Plugins
3.4.1 Disabling All Instrumentation Plugins
3.4.2 Disabling (or enabling) One Instrumentation Plugin
3.4.3 Disabling AgentRules of an Instrumentation Plugin
3.5 Disabling Tracer Plugins
3.6 Including custom Instrumentation Plugins
4 Definitions
4.1 SpecialAgent
4.2 Tracer
4.3 Tracer Plugin
4.4 Instrumentation Plugin
4.5 Instrumentation Rule
5 Objectives
5.1 Goals
5.2 Non-Goals
6 Supported Plugins
6.1 Instrumentation Plugins
6.2 Tracer Plugins
6.3 Instrumented libraries by existing rules
7 Credits
8 Contributing
9 License
This file contains the operational instructions for the use and development of SpecialAgent.
When SpecialAgent attaches to an application (either statically or dynamically), it will load the bundled Instrumentation Plugins.
The SpecialAgent is stable -- any exception that occurs during attachment of SpecialAgent will not adversely affect the stability of the target application. It is, however, important to note that SpecialAgent bundles Instrumentation Plugin that are developed by 3rd parties and individual contributors. We strive to assert the stability of SpecialAgent with rigorous integration tests, yet it is still possible that the code in a bundled Instrumentation Plugin may result in an exception that is not properly handled, which could potentially destabilize a target application.
The Maven build of the SpecialAgent project generates 2 artifacts: main and test. These artifacts can be obtained by cloning this repository and following the Development Instructions, or downloading directly from Maven's Central Repository.
The SpecialAgent is contained in a single JAR file. This JAR file is the main artifact that is built by Maven, and bundles the Instrumentation Plugins from the opentracing-contrib organization for which Instrumentation Rules have been implemented.
To use the SpecialAgent on an application, please download the stable or development main artifact.
The artifact JAR can be provided to an application with the -javaagent:${SPECIAL_AGENT_JAR} vm argument for Static Attach and Static Deferred Attach. The artifact JAR can also be executed in standalone fashion, which requires an argument to be passed for the PID of a target process to which SpecialAgent should dynamically attach. Please refer to Usage section for usage instructions.
The latest stable release is: 1.5.4
wget -O opentracing-specialagent-1.5.4.jar "http://central.maven.org/maven2/io/opentracing/contrib/specialagent/opentracing-specialagent/1.5.4/opentracing-specialagent-1.5.4.jar"The latest development release is: 1.5.5-SNAPSHOT
wget -O opentracing-specialagent-1.5.5-SNAPSHOT.jar "https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=io.opentracing.contrib.specialagent&a=opentracing-specialagent&v=LATEST"Note: Sometimes the web service call (in the line above) to retrieve the latest SNAPSHOT build fails to deliver the correct download. In order to work around this issue, please consider using the following command (for Linux and Mac OS):
wget -O opentracing-specialagent-1.5.5-SNAPSHOT.jar $(curl -s https://oss.sonatype.org/content/repositories/snapshots/io/opentracing/contrib/specialagent/opentracing-specialagent/1.5.5-SNAPSHOT/ | grep '".*\d\.jar"' | tail -1 | awk -F\" '{print $2}')The SpecialAgent is built in 2 passes that rely on different profiles:
-
The
defaultprofile is used for development of Instrumentation Rules. It builds and runs tests for each rule, but does not bundle the rules intoopentracing-specialagent-1.5.4.jarTo run this profile:
mvn clean install
-
The
assembleprofile is used to bundle the Instrumentation Rules intoopentracing-specialagent-1.5.4.jar. It builds each rule, but does not run tests. Once the build with theassembleprofile is finished, theopentracing-specialagent-1.5.4.jarwill contain the built rules inside it.Note: If you do not run this step, the
opentracing-specialagent-1.5.4.jarfrom the previous step will not contain any Instrumentation Plugins!Note: It is important to not run Maven's
cleanlifecycle when executing theassembleprofile.To run this profile:
mvn -Dassemble install
-
For a one-line build command to build SpecialAgent, its rules, run all tests, and create the
assemblepackage:mvn clean install && mvn -Dassemble install
2.1.2.1 Instrumentation Plugins
For development of Instrumentation Plugins, import the opentracing-specialagent-api and test-jar of the opentracing-specialagent.
<dependency>
<groupId>io.opentracing.contrib.specialagent</groupId>
<artifactId>opentracing-specialagent-api</artifactId>
<version>1.5.4</version> <!--version>1.5.5-SNAPSHOT<version-->
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.opentracing.contrib.specialagent</groupId>
<artifactId>opentracing-specialagent</artifactId>
<version>1.5.4</version> <!--version>1.5.5-SNAPSHOT<version-->
<type>test-jar</type>
<scope>test</scope>
</dependency>The test-jar is the test artifact that contains the AgentRunner class, which is a JUnit runner provided for testing of the ByteBuddy auto-instrumentation rules. This JAR does not contain Instrumentation Plugins themselves, and is only intended to be applied to the test phase of the build lifecycle of a single plugin for an Instrumentation Plugin implementation. For direction with the AgentRunner, please refer to the opentracing-specialagent-api module.
2.1.2.2 Tracer Plugins
Tracer Plugins integrate with the SpecialAgent via the OpenTracing TracerResolver, which connect the SpecialAgent to a Tracer. Tracer Plugins integrate to the SpecialAgent via the SPI mechanism defined in the TracerResolver, and are therefore not coupled to the SpecialAgent.
Tracer Plugins must be provided as "fat JARs" that contain the full set of all classes necessary for operation.
Tracer Plugins are integrated with the SpecialAgent by specifying a "provided" dependency in the !itest profile in the root POM. For instance, the dependency for the Jaeger Tracer Plugin is:
<dependency>
<groupId>io.opentracing.contrib</groupId>
<artifactId>jaeger-client-bundle</artifactId>
<version>0.0.3</version>
<scope>provided</scope>
</dependency>
Each Tracer Plugin integrated with the SpecialAgent must define a Short Name, which is a string that is used to reference the plugin with the -Dsa.plugin=<SHORT_NAME> system property. To provide a Short Name for the Tracer Plugin, you must define a Maven property in the root POM with the name matching the artifactId of the Tracer Plugin module. For instance, the Short Name for the Jaeger Tracer Plugin is defined as:
<properties>
...
<jaeger-client-bundle>jaeger</jaeger-client-bundle>
...
</properties>
The SpecialAgent is used by attaching to a target application. Once attached, the SpecialAgent relies on Java’s Instrumentation mechanism to transform the behavior of the application.
SpecialAgent supports the following attach modes:
| Attach Mode | Number of Required Commands to Attach |
Plugin and Rule Initialization Timeline |
|---|---|---|
| Static Attach |
1 (sync) |
Before app start (any application) |
| Dynamic Attach |
2 (async) |
After app start (any application) |
| Static Deferred Attach |
1 (sync) |
After app start (some applications) |
With Static Attach, the application is executed with the -javaagent argument, and the agent initialization occurs before the application is started. This mode requires 1 command from the command line.
Statically attaching to a Java application involves the use of the -javaagent vm argument at the time of startup of the target Java application. The following command can be used as an example:
java -javaagent:opentracing-specialagent-1.5.4.jar -jar MyApp.jarThis command statically attaches SpecialAgent into the application in MyApp.jar.
With Dynamic Attach, the application is allowed to start first, afterwhich an agent VM is dynamically attached to the application's PID. This mode requires 2 commands from the command line: the first for the application, and the second for the agent VM.
Dynamically attaching to a Java application involves the use of a running application’s PID, after the application’s startup. The following commands can be used as an example:
-
To obtain the
PIDof the target application:jps
-
To attach to the target
PID:-
For jdk1.8
java -Xbootclasspath/a:$JAVA_HOME/lib/tools.jar -jar opentracing-specialagent-1.5.4.jar <PID>
-
For jdk9+
java -jar opentracing-specialagent-1.5.4.jar <PID>
-
Note: If you encounter an exception stating Unable to open socket file, make sure the attaching VM is executed with the same permissions as the target VM.
With Static Deferred Attach, the application is executed with the -javaagent argument, but the agent initialization is deferred until the application is started. This mode requires 1 command from the command line, and is designed specifically for runtimes that have complex initialization lifecycles that may result in extraneously lengthy startup times when attached with Static Attach.
- Spring WebMVC (1.0 to LATEST).
- Spring Boot (1.0.0.RELEASE to LATEST).
If the above supported application environment is detected, Static Deferred Attach is automatically activated.
To deactivate Static Deferred Attach, specify the following system property on the command line:
-Dsa.init.defer=falseThe following command can be used as an example:
java -javaagent:opentracing-specialagent-1.5.4.jar -Dsa.init.defer=false -jar MySpringBootApp.jarThe SpecialAgent exposes a simple pattern for configuration of SpecialAgent, the Instrumentation Plugins, as well as Tracer Plugins. The configuration pattern is based on system properties, which can be defined on the command-line, in a properties file, or in @AgentRunner.Config for JUnit tests:
Configuration Layers
-
Properties passed on the command-line via
-D${PROPERTY}=...override same-named properties defined in the subsequent layers. -
The @AgentRunner.Config annotation allows one to define log level and re/transformation event logging settings. Properties defined in the
@Configannotation override same-named properties defined in the subsequent layers. -
The
-Dsa.config=${PROPERTIES_FILE}command-line argument can be specified for SpecialAgent to load property names from a${PROPERTIES_FILE}. Properties defined in the${PROPERTIES_FILE}override same-named properties defined in the subsequent layer. -
The SpecialAgent has a
default.propertiesfile that defines default values for properties that need to be defined.
The following properties are supported by all Instrumentation Plugins:
-
Logging:
-
-Dsa.log.levelSet the logging level for SpecialAgent. Acceptable values are:
SEVERE,WARNING,INFO,CONFIG,FINE,FINER, orFINEST, or any numerical log level value is accepted also. The default logging level is set toWARNING. -
-Dsa.log.eventsSet the re/transformation events to be logged:
DISCOVERY,IGNORED,TRANSFORMATION,ERROR,COMPLETE. The property accepts a comma-delimited list of event names. By default, theERRORevent is logged (only when run withAgentRunner). -
-Dsa.log.fileSet the logging output file for SpecialAgent.
-
-
Verbose Mode:
-Dsa.instrumentation.plugin.*.verbose
-Dsa.instrumentation.plugin.${RULE_NAME_PATTERN}.verboseSets verbose mode for all plugins (i.e.
*) or one plugin (i.e.${RULE_NAME_PATTERN}). This property can also be set in anAgentRunnerJUnit test with the@AgentRunner.Config(verbose=true)for all tests in a JUnit class, or@AgentRunner.TestConfig(verbose=true)for an individual JUnit test method.The Java Concurrent API plugin supports verbose mode, which is disabled by default. To enable, set
sa.concurrent.verbose=true. In non-verbose mode, parent span context is propagating to task execution (if a parent span context exists). In verbose mode, a parent span is always created upon task submission to the executor, and a child span is created when the task is started. -
Skip fingerprint verification:
-Dsa.fingerprint.skipTells the SpecialAgent to skip the fingerprint verification when linking Instrumentation Plugins into class loaders. This option allows one to work around an unexpected fingerprint verification failure, which can happen in complex runtimes that do not contain all class definitions on the class path. It must be noted, however, that if the fingerprint verification is disabled, the SpecialAgent will indiscriminately install all plugins regardless of library version compatibility issues, which may lead to
NoClassDefFoundError,IllegalAccessError,AbstractMethodError,LinkageError, etc.
3.3 Selecting the Tracer Plugin
The SpecialAgent supports OpenTracing-compatible tracers. There are 2 ways to connect a tracer to the SpecialAgent runtime:
-
Bundled Tracer Plugins
The SpecialAgent bundles the following Tracer Plugins:
The
-Dsa.tracer=${TRACER_PLUGIN}property specifies which Tracer Plugin is to be used. The value of${TRACER_PLUGIN}is the Short Name of the Tracer Plugin, i.e.jaeger,lightstep, orwavefront. -
External Tracer Plugins
The SpecialAgent allows external Tracer Plugins to be attached to the runtime.
The
-Dsa.tracer=${TRACER_JAR}property specifies the JAR path of the Tracer Plugin to be used. The${TRACER_JAR}must be a JAR that conforms to theTracerFactoryAPI of the TracerResolver project.
NOTE: If a tracer is not specified with the -Dsa.tracer=... property, the SpecialAgent will present a warning in the log that states: Tracer NOT RESOLVED.
3.4 Disabling Instrumentation Plugins
Instrumentation Plugins bundled with the SpecialAgent are enabled by default.
Multiple properties to disable or to enable all or individual plugins can be declared via the Configuration Pattern. The processing order of the properties is equal to the order of their declaration.
To disable all instrumentation plugins:
sa.instrumentation.plugin.*.disable
The suffix .disable is interchangeable with .enable=false.
To disable an individual instrumentation plugin:
sa.instrumentation.plugin.${RULE_NAME_PATTERN}.disable
The suffix .disable is interchangeable with .enable=false.
Conversely, to enable an individual instrumentation plugin.
sa.instrumentation.plugin.${RULE_NAME_PATTERN}.enable
The suffix .enable is interchangeable with .disable=false.
The value of ${RULE_NAME_PATTERN} represents the Rule Name, as specified in Instrumentation Plugins ("SpecialAgent Rule" column). The ${RULE_NAME_PATTERN} allows for the use of * and ? characters to match multiple rules simultaneously. For instance:
lettuce:5.?
Matches all Lettuce plugins, includinglettuce:5.0,lettuce:5.1, andlettuce:5.2.spring:web:*
Matches all Spring Web plugins, includingspring:web:3andspring:web:5.spring:web*
Matches all Spring Web and WebMVC plugins, includingspring:web:3,spring:web:5,spring:webmvc:3, andspring:webmvc:5.
If the version part of the ${RULE_NAME_PATTERN} does not end with a * or ? character, a * will be appended automatically. Therefore:
lettuce:5
Matches all Lettuce v5 plugins, includinglettuce:5.0,lettuce:5.1, andlettuce:5.2.spring:web
Matches all Spring Web plugins, includingspring:web:3andspring:web:5.spring
Matches all Spring plugins.spring:w
Does not match any plugins.
To disable an individual AgentRule of an instrumentation plugin:
sa.instrumentation.plugin.${PLUGIN_NAME}#${AGENT_RULE_SIMPLE_CLASS_NAME}.disable
The suffix .disable is interchangeable with .enable=false.
The value of ${AGENT_RULE_SIMPLE_CLASS_NAME} is the simple class name of the AgentRule subclass that is to be disabled.
3.5 Disabling Tracer Plugins
The SpecialAgent has all of its Tracer Plugins enabled by default, and allows them to be disabled.
To disable all tracer plugins:
sa.tracer.plugins.disable
The suffix .disable is interchangeable with .enable=false.
To disable an individual tracer plugin:
sa.tracer.plugin.${SHORT_NAME}.disable
The suffix .disable is interchangeable with .enable=false.
The value of ${SHORT_NAME} is the Short Name of the plugin, such as lightstep, wavefront, or jaeger.
Custom plugins and AgentRules can be implemented by following the SpecialAgent Rule API. JARs containing custom plugins and AgentRules can be loaded by SpecialAgent with:
-Dsa.instrumentation.plugin.include=<JARs>Here, <JARs> refers to a pathSeparator-delimited (: for *NIX, ; for Windows) string of JARs containing the custom rules.
The following terms are used throughout this documentation.
The SpecialAgent is software that attaches to Java applications, and automatically instruments 3rd-party libraries integrated in the application. The SpecialAgent uses the OpenTracing API for Instrumentation Plugins that instrument 3rd-party libraries, as well as Tracer Plugins that implement OpenTracing tracer service providers. Both the Instrumentation Plugins, as well as the Tracer Plugins are open-source, and are developed and supported by the OpenTracing community.
The SpecialAgent supports Oracle Java and OpenJDK.
4.2 Tracer
Service provider of the OpenTracing standard, providing an implementation of the io.opentracing.Tracer interface.
Examples:
Tracers are not coupled to the SpecialAgent.
4.3 Tracer Plugin
A bridge providing automatic discovery of Tracers in a runtime instrumented with the OpenTracing API. This bridge implements the TracerFactory interface of TracerResolver, and is distributed as a single "fat JAR" that can be conveniently added to the classpath of a Java process.
Tracer Plugins are not coupled to the SpecialAgent.
An OpenTracing Instrumentation project that exist as individual repositories under opentracing-contrib.
Examples:
Instrumentation Plugins are not coupled to the SpecialAgent.
A submodule of the SpecialAgent that implements the auto-instrumentation rules for Instrumentation Plugins via the opentracing-specialagent-api.
Examples:
Instrumentation Rules are coupled to the SpecialAgent.
- The SpecialAgent must allow any Instrumentation Plugin available in opentracing-contrib to be automatically installable in applications that utilize a 3rd-party library for which an Instrumentation Plugin exists.
- The SpecialAgent must automatically install the Instrumentation Plugin for each 3rd-party library for which a module exists, regardless in which class loader the 3rd-party library is loaded.
- The SpecialAgent must not adversely affect the runtime stability of the application on which it is intended to be used. This goal applies only to the code in the SpecialAgent, and cannot apply to the code of the Instrumentation Plugins made available in opentracing-contrib.
- The SpecialAgent must Static Attach and Dynamic Attach to applications running on JVM versions 1.7, 1.8, 9, and 11.
- The SpecialAgent must implement a lightweight test methodology that can be easily applied to a module that implements instrumentation for a 3rd-party library. This test must simulate:
- Launch the test in a process simulating the
-javaagentvm argument that points to the SpecialAgent (in order to test auto-instrumentation functionality). - Elevate the test code to be executed from a custom class loader that is disconnected from the system class loader (in order to test bytecode injection into an isolated class loader that cannot resolve classes on the system classpath).
- Allow tests to specify their own
Tracerinstances viaGlobalTracer, or initialize aMockTracerif no instance is specified. The test must provide a reference to theTracerinstance in the test method for assertions with JUnit.
- Launch the test in a process simulating the
- The SpecialAgent must provide a means by which Instrumentation Plugins can be configured before use on a target application.
- The SpecialAgent is not designed to modify application code, beyond the installation of Instrumentation Plugins. For example, there is no facility for dynamically tracing arbitrary code.
Intrinsically, the SpecialAgent includes support for the instrumentation of the following 3rd-party libraries. Each row refers to an Instrumentation Plugin, the Instrumentation Rule, and the minimum and maximum version tested by the build.
The following plugins have Instrumentation Rules implemented.
Direction for development of Instrumentation Rules is available in the opentracing-specialagent-api module.
6.2 Tracer Plugins
Intrinsically, the SpecialAgent includes support for the following Tracer Plugins. A demo can be referenced here.
The following libraries are instrumented by existing Instrumentation Rules.
Thank you to the following contributors for developing instrumentation plugins:
Thank you to the following contributors for developing tracer plugins:
Thank you to the following developers for filing issues and helping us fix them:
- Louis-Etienne
- Marcos Trejo Munguia
- @kaushikdeb
- @deepakgoenka
- @etsangsplk
- @Vovan2006
- Randall Theobald
- Jianshao Wu
Thank you to the following individuals for all other general contributions to the codebase:
Finally, thanks for all of the feedback! Please share your comments as an issue!
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please make sure to update tests as appropriate.
This project is licensed under the Apache 2 License - see the LICENSE.txt file for details.
