Agent Configuration
SDK Autoconfiguration
The SDK’s autoconfiguration module is used for basic configuration of the agent. Read the docs to find settings such as configuring export or sampling.
Here are some quick links into those docs for the configuration options for specific portions of the SDK & agent:
- Exporters
- Trace context propagation
- OpenTelemetry Resource and service name
- Batch span processor
- Sampler
- Span limits
- Using SPI to further configure the SDK
Enable Resource Providers that are disabled by default
In addition to the resource configuration from the SDK autoconfiguration, you can enable additional resource providers that are disabled by default:
System property:
otel.resource.providers.aws.enabled
Environment variable:
OTEL_RESOURCE_PROVIDERS_AWS_ENABLED
Default: false
Description:
Enables the AWS Resource Provider.
System property:
otel.resource.providers.gcp.enabled
Environment variable:
OTEL_RESOURCE_PROVIDERS_GCP_ENABLED
Default: false
Description:
Enables the GCP Resource Provider.
Configuring the agent
The agent can consume configuration from one or more of the following sources (ordered from highest to lowest priority):
- system properties
- environment variables
- the configuration file
- properties provided by the
AutoConfigurationCustomizer#addPropertiesSupplier()
function; using theAutoConfigurationCustomizerProvider
SPI
Configuring with Environment Variables
In some environments, configuring via Environment Variables is more preferred. Any setting configurable with a System Property can also be configured with an Environment Variable. Many settings below include both options, but where they don’t apply the following steps to determine the correct name mapping of the desired System Property:
- Convert the System Property to uppercase.
- Replace all
.
and-
characters with_
.
For example otel.instrumentation.common.default-enabled
would convert to
OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED
.
Configuration file
You can provide a path to agent configuration file by setting the following property:
System property:
otel.javaagent.configuration-file
Environment variable:
OTEL_JAVAAGENT_CONFIGURATION_FILE
Description: Path to valid Java properties file which contains the agent configuration.
Extensions
You can enable extensions by setting the following property:
System property:
otel.javaagent.extensions
Environment variable:
OTEL_JAVAAGENT_EXTENSIONS
Description: Path to an extension jar file or folder, containing jar files. If pointing to a folder, every jar file in that folder will be treated as separate, independent extension.
Java agent logging output
The agent’s logging output can be configured by setting the following property:
System property:
otel.javaagent.logging
Environment variable:
OTEL_JAVAAGENT_LOGGING
Description: The Java agent logging mode. The following 3 modes are supported:
simple
: The agent will print out its logs using the standard error stream. OnlyINFO
or higher logs will be printed. This is the default Java agent logging mode.none
: The agent will not log anything - not even its own version.application
: The agent will attempt to redirect its own logs to the instrumented application's slf4j logger. This works the best for simple one-jar applications that do not use multiple classloaders; Spring Boot apps are supported as well. The Java agent output logs can be further configured using the instrumented application's logging configuration (e.g.logback.xml
orlog4j2.xml
). Make sure to test that this mode works for your application before running it in a production environment.
Common instrumentation configuration
Common settings that apply to multiple instrumentations at once.
Peer service name
The
peer service name
is the name of a remote service to which a connection is made. It corresponds to
service.name
in the resource for the
local service.
System property:
otel.instrumentation.common.peer-service-mapping
Environment variable:
OTEL_INSTRUMENTATION_COMMON_PEER_SERVICE_MAPPING
Description:
Used to specify a mapping from host names or IP addresses to peer services, as a
comma-separated list of <host_or_ip>=<user_assigned_name>
pairs. The peer
service is added as an attribute to a span whose host or IP address match the
mapping.
For example, if set to the following:
1.2.3.4=cats-service,dogs-abcdef123.serverlessapis.com=dogs-api
Then, requests to 1.2.3.4
will have a peer.service
attribute of
cats-service
and requests to dogs-abcdef123.serverlessapis.com
will have an
attribute of dogs-api
.
Since Java agent version 1.31.0
, it is possible to provide a port and a path
to define a peer.service
.
For example, if set to the following:
1.2.3.4:443=cats-service,dogs-abcdef123.serverlessapis.com:80/api=dogs-api
Then, requests to 1.2.3.4
will have no override for peer.service
attribute,
while 1.2.3.4:443
will have have peer.service
of cats-service
and requests
to dogs-abcdef123.serverlessapis.com:80/api/v1
will have an attribute of
dogs-api
.
DB statement sanitization
The agent sanitizes all database queries/statements before setting the
db.statement
semantic attribute. All values (strings, numbers) in the query
string are replaced with a question mark (?
).
Note: JDBC bind parameters are not captured in db.statement
. See
the corresponding issue
if you are looking to capture bind parameters.
Examples:
- SQL query
SELECT a from b where password="secret"
will appear asSELECT a from b where password=?
in the exported span; - Redis command
HSET map password "secret"
will appear asHSET map password ?
in the exported span.
This behavior is turned on by default for all database instrumentations. Use the following property to disable it:
System property:
otel.instrumentation.common.db-statement-sanitizer.enabled
Environment variable:
OTEL_INSTRUMENTATION_COMMON_DB_STATEMENT_SANITIZER_ENABLED
Default: true
Description:
Enables the DB statement sanitization.
Capturing HTTP request and response headers
You can configure the agent to capture predefined HTTP headers as span attributes, according to the semantic convention. Use the following properties to define which HTTP headers you want to capture:
System property:
otel.instrumentation.http.client.capture-request-headers
Environment variable:
OTEL_INSTRUMENTATION_HTTP_CLIENT_CAPTURE_REQUEST_HEADERS
Description: A comma-separated list of HTTP header names. HTTP client instrumentations will capture HTTP request header values for all configured header names.
System property:
otel.instrumentation.http.client.capture-response-headers
Environment variable:
OTEL_INSTRUMENTATION_HTTP_CLIENT_CAPTURE_RESPONSE_HEADERS
Description: A comma-separated list of HTTP header names. HTTP client instrumentations will capture HTTP response header values for all configured header names.
System property:
otel.instrumentation.http.server.capture-request-headers
Environment variable:
OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_REQUEST_HEADERS
Description: A comma-separated list of HTTP header names. HTTP server instrumentations will capture HTTP request header values for all configured header names.
System property:
otel.instrumentation.http.server.capture-response-headers
Environment variable:
OTEL_INSTRUMENTATION_HTTP_SERVER_CAPTURE_RESPONSE_HEADERS
Description: A comma-separated list of HTTP header names. HTTP server instrumentations will capture HTTP response header values for all configured header names.
These configuration options are supported by all HTTP client and server instrumentations.
Note: The property/environment variable names listed in the table are still experimental, and thus are subject to change.
Capturing servlet request parameters
You can configure the agent to capture predefined HTTP request parameter as span attributes for requests that are handled by Servlet API. Use the following property to define which servlet request parameters you want to capture:
System property:
otel.instrumentation.servlet.experimental.capture-request-parameters
Environment variable:
OTEL_INSTRUMENTATION_SERVLET_EXPERIMENTAL_CAPTURE_REQUEST_PARAMETERS
Description: A comma-separated list of request parameter names.
Note: The property/environment variable names listed in the table are still experimental, and thus are subject to change.
Capturing consumer message receive telemetry in messaging instrumentations
You can configure the agent to capture the consumer message receive telemetry in messaging instrumentation. Use the following property to enable it:
System property:
otel.instrumentation.messaging.experimental.receive-telemetry.enabled
Environment variable:
OTEL_INSTRUMENTATION_MESSAGING_EXPERIMENTAL_RECEIVE_TELEMETRY_ENABLED
Default: false
Description:
Enables the consumer message receive telemetry.
Note that this will cause the consumer side to start a new trace, with only a span link connecting it to the producer trace.
Note: The property/environment variable names listed in the table are still experimental, and thus are subject to change.
Capturing enduser attributes
You can configure the agent to capture
general identity attributes
(enduser.id
, enduser.role
, enduser.scope
) from instrumentation libraries
like
JavaEE/JakartaEE Servlet
and
Spring Security.
Note: Given the sensitive nature of the data involved, this feature is turned off by default while allowing selective activation for particular attributes. You must carefully evaluate each attribute’s privacy implications before enabling the collection of the data.
System property:
otel.instrumentation.common.enduser.enabled
Environment variable:
OTEL_INSTRUMENTATION_COMMON_ENDUSER_ENABLED
Default: false
Description:
Common flag for enabling/disabling enduser attributes.
System property:
otel.instrumentation.common.enduser.id.enabled
Environment variable:
OTEL_INSTRUMENTATION_COMMON_ENDUSER_ID_ENABLED
Default: false
Description:
Determines whether to capture enduser.id
semantic attribute.
System property:
otel.instrumentation.common.enduser.role.enabled
Environment variable:
OTEL_INSTRUMENTATION_COMMON_ENDUSER_ROLE_ENABLED
Default: false
Description:
Determines whether to capture enduser.role
semantic attribute.
System property:
otel.instrumentation.common.enduser.scope.enabled
Environment variable:
OTEL_INSTRUMENTATION_COMMON_ENDUSER_SCOPE_ENABLED
Default: false
Description:
Determines whether to capture enduser.scope
semantic attribute.
Spring Security
For users of Spring Security who use custom
granted authority prefixes,
you can use the following properties to strip those prefixes from the
enduser.*
attribute values to better represent the actual role and scope
names:
System property:
otel.instrumentation.spring-security.enduser.role.granted-authority-prefix
Environment variable:
OTEL_INSTRUMENTATION_SPRING_SECURITY_ENDUSER_ROLE_GRANTED_AUTHORITY_PREFIX
Default: ROLE_
Description:
Prefix of granted authorities identifying roles to capture in the enduser.role
semantic attribute.
System property:
otel.instrumentation.spring-security.enduser.scope.granted-authority-prefix
Environment variable:
OTEL_INSTRUMENTATION_SPRING_SECURITY_ENDUSER_SCOPE_GRANTED_AUTHORITY_PREFIX
Default: SCOPE_
Description:
Prefix of granted authorities identifying scopes to capture in the enduser.scopes
semantic attribute.
Suppressing specific auto-instrumentation
Disabling the agent entirely
System property:
otel.javaagent.enabled
Environment variable:
OTEL_JAVAAGENT_ENABLED
Description:
Set the value to false
to disable the agent entirely.
Enable only specific instrumentation
You can disable all default auto instrumentation and selectively re-enable individual instrumentation. This may be desirable to reduce startup overhead or to have more control of which instrumentation is applied.
System property:
otel.instrumentation.common.default-enabled
Environment variable:
OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED
Description:
Set to
false
to disable all instrumentation in the agent.
System property:
otel.instrumentation.[name].enabled
Environment variable:
OTEL_INSTRUMENTATION_[NAME]_ENABLED
Description:
Set to true
to enable each desired instrumentation individually.
Note: Some instrumentation relies on other instrumentation to function properly. When selectively enabling instrumentation, be sure to enable the transitive dependencies too. Determining this dependency relationship is left as an exercise to the user.
Enable manual instrumentation only
You can suppress all auto instrumentations but have support for manual
instrumentation with @WithSpan
and normal API interactions by using
-Dotel.instrumentation.common.default-enabled=false -Dotel.instrumentation.opentelemetry-api.enabled=true -Dotel.instrumentation.opentelemetry-instrumentation-annotations.enabled=true
Suppressing specific agent instrumentation
You can suppress agent instrumentation of specific libraries.
System property:
otel.instrumentation.[name].enabled
Environment variable:
OTEL_INSTRUMENTATION_[NAME]_ENABLED
Description:
Set to false
to suppress agent instrumentation of specific libraries, where [name] is the
corresponding instrumentation name:
Library/Framework | Instrumentation name |
---|---|
Additional methods tracing | methods |
Additional tracing annotations | external-annotations |
Akka Actor | akka-actor |
Akka HTTP | akka-http |
Apache Axis2 | axis2 |
Apache Camel | camel |
Apache Cassandra | cassandra |
Apache CXF | cxf |
Apache DBCP | apache-dbcp |
Apache Dubbo | apache-dubbo |
Apache Geode | geode |
Apache HttpAsyncClient | apache-httpasyncclient |
Apache HttpClient | apache-httpclient |
Apache Kafka | kafka |
Apache MyFaces | jsf-myfaces |
Apache Pekko Actor | pekko-actor |
Apache Pekko HTTP | pekko-http |
Apache Pulsar | pulsar |
Apache RocketMQ | rocketmq-client |
Apache Struts 2 | struts |
Apache Tapestry | tapestry |
Apache Tomcat | tomcat |
Apache Wicket | wicket |
Armeria | armeria |
AsyncHttpClient (AHC) | async-http-client |
AWS Lambda | aws-lambda |
AWS SDK | aws-sdk |
Azure SDK | azure-core |
Couchbase | couchbase |
C3P0 | c3p0 |
Dropwizard Views | dropwizard-views |
Dropwizard Metrics | dropwizard-metrics |
Eclipse Grizzly | grizzly |
Eclipse Jersey | jersey |
Eclipse Jetty | jetty |
Eclipse Jetty HTTP Client | jetty-httpclient |
Eclipse Metro | metro |
Eclipse Mojarra | jsf-mojarra |
Eclipse Vert.x HttpClient | vertx-http-client |
Eclipse Vert.x Kafka Client | vertx-kafka-client |
Eclipse Vert.x RxJava | vertx-rx-java |
Eclipse Vert.x Web | vertx-web |
Elasticsearch client | elasticsearch-transport |
Elasticsearch REST client | elasticsearch-rest |
Google Guava | guava |
Google HTTP client | google-http-client |
Google Web Toolkit | gwt |
Grails | grails |
GraphQL Java | graphql-java |
GRPC | grpc |
Hibernate | hibernate |
HikariCP | hikaricp |
Java HTTP Client | java-http-client |
Java HttpURLConnection | http-url-connection |
Java JDBC | jdbc |
Java JDBC DataSource | jdbc-datasource |
Java RMI | rmi |
Java Runtime | runtime-telemetry |
Java Servlet | servlet |
java.util.concurrent | executors |
java.util.logging | java-util-logging |
JAX-RS (Client) | jaxrs-client |
JAX-RS (Server) | jaxrs |
JAX-WS | jaxws |
JBoss Logging Appender | jboss-logmanager-appender |
JBoss Logging MDC | jboss-logmanager-mdc |
JMS | jms |
Jodd HTTP | jodd-http |
JSP | jsp |
K8s Client | kubernetes-client |
kotlinx.coroutines | kotlinx-coroutines |
Log4j Appender | log4j-appender |
Log4j MDC (1.x) | log4j-mdc |
Log4j Context Data (2.x) | log4j-context-data |
Logback Appender | logback-appender |
Logback MDC | logback-mdc |
Micrometer | micrometer |
MongoDB | mongo |
Netflix Hystrix | hystrix |
Netty | netty |
OkHttp | okhttp |
OpenLiberty | liberty |
OpenTelemetry Extension Annotations | opentelemetry-extension-annotations |
OpenTelemetry Instrumentation Annotations | opentelemetry-instrumentation-annotations |
OpenTelemetry API | opentelemetry-api |
Oracle UCP | oracle-ucp |
OSHI (Operating System and Hardware Information) | oshi |
Play Framework | play |
Play WS HTTP Client | play-ws |
Quartz | quartz |
R2DBC | r2dbc |
RabbitMQ Client | rabbitmq |
Ratpack | ratpack |
ReactiveX RxJava | rxjava |
Reactor | reactor |
Reactor Netty | reactor-netty |
Redis Jedis | jedis |
Redis Lettuce | lettuce |
Rediscala | rediscala |
Redisson | redisson |
Restlet | restlet |
Scala ForkJoinPool | scala-fork-join |
Spark Web Framework | spark |
Spring Batch | spring-batch |
Spring Core | spring-core |
Spring Data | spring-data |
Spring JMS | spring-jms |
Spring Integration | spring-integration |
Spring Kafka | spring-kafka |
Spring RabbitMQ | spring-rabbit |
Spring RMI | spring-rmi |
Spring Scheduling | spring-scheduling |
Spring Web | spring-web |
Spring WebFlux | spring-webflux |
Spring Web MVC | spring-webmvc |
Spring Web Services | spring-ws |
Spymemcached | spymemcached |
Tomcat JDBC | tomcat-jdbc |
Twilio SDK | twilio |
Twitter Finatra | finatra |
Undertow | undertow |
Vaadin | vaadin |
Vibur DBCP | vibur-dbcp |
ZIO | zio |
Note: When using environment variables, dashes (-
) should be converted to
underscores (_
). For example, to suppress traces from akka-actor
library,
set OTEL_INSTRUMENTATION_AKKA_ACTOR_ENABLED
to false
.
Suppressing controller and/or view spans
Some instrumentations (e.g. Spring Web MVC instrumentation) produce
SpanKind.Internal spans to capture the
controller and/or view execution. These spans can be suppressed using the
configuration settings below, without suppressing the entire instrumentation
which would also disable the instrumentation’s capturing of http.route
and
associated span name on the parent
SpanKind.Server span.
System property:
otel.instrumentation.common.experimental.controller-telemetry.enabled
Environment variable:
OTEL_INSTRUMENTATION_COMMON_EXPERIMENTAL_CONTROLLER_TELEMETRY_ENABLED
Default: false
Description:
Set to true
to enable controller telemetry.
System property:
otel.instrumentation.common.experimental.view-telemetry.enabled
Environment variable:
OTEL_INSTRUMENTATION_COMMON_EXPERIMENTAL_VIEW_TELEMETRY_ENABLED
Default: false
Description:
Set to true
to enable view telemetry.
Instrumentation span suppression behavior
Some libraries that this agent instruments in turn use lower-level libraries, that are also instrumented. This would normally result in nested spans containing duplicate telemetry data. For example:
- Spans produced by the Reactor Netty HTTP client instrumentation would have duplicate HTTP client spans produced by the Netty instrumentation;
- Dynamo DB spans produced by the AWS SDK instrumentation would have children HTTP client spans produced by its internal HTTP client library (which is also instrumented);
- Spans produced by the Tomcat instrumentation would have duplicate HTTP server spans produced by the generic Servlet API instrumentation.
The Java agent prevents these situations by detecting and suppressing nested spans that duplicate telemetry data. The suppression behavior can be configured using the following configuration option:
System property:
otel.instrumentation.experimental.span-suppression-strategy
Environment variable:
OTEL_INSTRUMENTATION_EXPERIMENTAL_SPAN_SUPPRESSION_STRATEGY
Description: The Java agent span suppression strategy. The following 3 strategies are supported:
semconv
: The agent will suppress duplicate semantic conventions. This is the default behavior of the Java agent.span-kind
: The agent will suppress spans with the same kind (exceptINTERNAL
).none
: The agent will not suppress anything at all. We do not recommend using this option for anything other than debug purposes, as it generates lots of duplicate telemetry data.
For example, suppose we instrument a database client which internally uses the Reactor Netty HTTP client; which in turn uses Netty.
Using the default semconv
suppression strategy would result in 2 nested
CLIENT
spans:
CLIENT
span with database client semantic attributes emitted by the database client instrumentation;CLIENT
span with HTTP client semantic attributes emitted by the Reactor Netty instrumentation.
The Netty instrumentation would be suppressed, as it duplicates the Reactor Netty HTTP client instrumentation.
Using the suppression strategy span-kind
would result in just one span:
CLIENT
span with database client semantic attributes emitted by the database client instrumentation.
Both Reactor Netty and Netty instrumentations would be suppressed, as they also
emit CLIENT
spans.
Finally, using the suppression strategy none
would result in 3 spans:
CLIENT
span with database client semantic attributes emitted by the database client instrumentation;CLIENT
span with HTTP client semantic attributes emitted by the Reactor Netty instrumentation;CLIENT
span with HTTP client semantic attributes emitted by the Netty instrumentation.