Skip to content

Latest commit

 

History

History
747 lines (558 loc) · 53.7 KB

how_instrumentations_work.md

File metadata and controls

747 lines (558 loc) · 53.7 KB
Raw

How Instrumentations Work

Introduction

Around 120 integrations consisting of about 200 instrumentations are currently provided with the Datadog Java Trace Agent. An auto-instrumentation allows compiled Java applications to be instrumented at runtime by a Java agent. This happens when compiled classes matching rules defined in the instrumentation undergo bytecode manipulation to accomplish some of what could be done by a developer instrumenting the code manually. Instrumentations are maintained in /dd-java-agent/instrumentation/

Files/Directories

Instrumentations are in the directory:

/dd-java-agent/instrumentation/$framework/$framework-$minVersion

where $framework is the framework name, and $minVersion is the minimum version of the framework supported by the instrumentation. For example:

$ tree dd-java-agent/instrumentation/couchbase -L 2
dd-java-agent/instrumentation/couchbase
├── build.gradle
├── couchbase-2.0
│   ├── build.gradle
│   └── src
├── couchbase-2.6
│   ├── build.gradle
│   └── src
├── couchbase-3.1
│   ├── build.gradle
│   └── src
└── couchbase-3.2
    ├── build.gradle
    └── src

In some cases, such as Hibernate, there is a submodule containing different version-specific instrumentations, but typically a version-specific module is enough when there is only one instrumentation implemented ( e.g. Akka-HTTP)

Gradle

Instrumentations included when building the Datadog java trace agent are defined in /settings.gradle in alphabetical order with the other instrumentations in this format:

include ':dd-java-agent:instrumentation:$framework?:$framework-$minVersion'

Dependencies specific to a particular instrumentation are added to the build.gradle file in that instrumentation’s directory. Add necessary dependencies as compileOnly so they do not leak into the tracer.

Muzzle

Muzzle directives are applied at build time from the build.gradle file. OpenTelemetry provides some Muzzle documentation. Muzzle directives check for a range of framework versions that are safe to load the instrumentation.

See this excerpt as an example from rediscala

muzzle {
  pass {
    group = "com.github.etaty"
    module = "rediscala_2.11"
    versions = "[1.5.0,)"
    assertInverse = true
  }

  pass {
    group = "com.github.etaty"
    module = "rediscala_2.12"
    versions = "[1.8.0,)"
    assertInverse = true
  }
}

This means that the instrumentation should be safe with rediscala_2.11 from version 1.5.0 and all later versions, but should fail (and so will not be loaded), for older versions (see assertInverse). A similar range of versions is specified for rediscala_2.12. When the agent is built, the muzzle plugin will download versions of the framework and check these directives hold. To run muzzle on your instrumentation, run:

./gradlew :dd-java-agent:instrumentation:rediscala-1.8.0:muzzle
  • ⚠️ Muzzle does not run tests. It checks that the types and methods used by the instrumentation are present in particular versions of libraries. It can be subverted with MethodHandle and reflection, so muzzle passing is not the end of the story.

By default, all the muzzle directives are checked against all the instrumentations included in a module. However, there can be situations in which it’s only needed to check one specific directive on an instrumentation. At this point the instrumentation should override the method muzzleDirective() by returning the name of the directive to execute.

Instrumentation classes

The Instrumentation class is where the Instrumentation begins. It will:

  1. Use Matchers to choose target types (i.e., classes)
  2. From only those target types, use Matchers to select the members (i.e., methods) to instrument.
  3. Apply instrumentation code from an Advice class to those members.

Instrumentation classes:

  1. Must be annotated with @AutoService(InstrumenterModule.class)
  2. Should extend one of the six abstract TargetSystem InstrumenterModule classes
  3. Should implement one of the Instrumenter interfaces

For example:

import datadog.trace.agent.tooling.Instrumenter;
import datadog.trace.agent.tooling.InstrumenterModule;

@AutoService(InstrumenterModule.class)
public class RabbitChannelInstrumentation extends InstrumenterModule.Tracing
        implements Instrumenter.ForTypeHierarchy {/* */
}
TargetSystem Usage
InstrumenterModule.Tracing An Instrumentation class should extend an appropriate provided TargetSystem class when possible.
InstrumenterModule.Profiling
InstrumenterModule.AppSec
InstrumenterModule.Iast
InstrumenterModule.CiVisibility
InstrumenterModule.Usm
InstrumenterModule Avoid extending InstrumenterModule directly. When no other TargetGroup is applicable we generally default to InstrumenterModule.Tracing

Type Matching

Instrumentation classes should implement an appropriate Instrumenter interface that specifies how target types will be selected for instrumentation.
Instrumenter Interface Method(s) Usage(Example)
ForSingleType String instrumentedType() Instruments only a single class name known at compile time.(see Json2FactoryInstrumentation)
ForKnownTypes String[] knownMatchingTypes() Instruments multiple class names known at compile time.
ForTypeHierarchy String hierarchyMarkerType()``ElementMatcher<TypeDescription> hierarchyMatcher() Composes more complex matchers using chained HierarchyMatchers methods. The hierarchyMarkerType() method should return a type name. Classloaders without this type can skip the more expensive hierarchyMatcher() method. (see HttpClientInstrumentation)
ForConfiguredType Collection<String> configuredMatchingTypes() Do not implement this interface_._Use ForKnownType instead. ForConfiguredType is only used for last minute additions in the field - such as when a customer has a new JDBC driver that's not in the allowed list and we need to test it and provide a workaround until the next release.
ForConfiguredTypes String configuredMatchingType(); Do not implement this interface. __Like ForConfiguredType, for multiple classes

When matching your instrumentation against target types, prefer ForSingleType or ForKnownTypes over more expensive ForTypeHierarchy matching.

Consider adding an appropriate ClassLoaderMatcher so the Instrumentation only activates when that class is loaded. For example:

@Override
public ElementMatcher<ClassLoader> classLoaderMatcher() {
    return hasClassNamed("java.net.http.HttpClient");
}

The Instrumenter.ForBootstrap interface is a hint that this instrumenter works on bootstrap types and there is no classloader present to interrogate. Use it when instrumenting something from the JDK that will be on the bootstrap classpath. For example, ShutdownInstrumentation or UrlInstrumentation.

Note

Without classloader available, helper classes for bootstrap instrumentation must be place into the :dd-java-agent:agent-bootstrap module rather than loaded using the default mechanism.

Method Matching

After the type is selected, the type’s target members(e.g., methods) must next be selected using the Instrumentation class’s adviceTransformations() method. ByteBuddy’s ElementMatchers are used to describe the target members to be instrumented. Datadog’s DDElementMatchers class also provides these 10 additional matchers:

  • implementsInterface
  • hasInterface
  • hasSuperType
  • declaresMethod
  • extendsClass
  • concreteClass
  • declaresField
  • declaresContextField
  • declaresAnnotation
  • hasSuperMethod

Here, any public execute() method taking no arguments will have PreparedStatementAdvice applied:

@Override
public void adviceTransformations(AdviceTransformation transformation) {
    transformation.applyAdvice(
            nameStartsWith("execute")
                    .and(takesArguments(0))
                    .and(isPublic()),
            getClass().getName() + "$PreparedStatementAdvice"
    );
}

Here, any matching connect() method will have DriverAdvice applied:

@Override
public void adviceTransformations(AdviceTransformation transformation) {
    transformation.applyAdvice(
            nameStartsWith("connect")
                    .and(takesArgument(0, String.class))
                    .and(takesArgument(1, Properties.class))
                    .and(returns(named("java.sql.Connection"))),
            getClass().getName() + "$DriverAdvice");
}

Be precise in matching to avoid inadvertently instrumenting something unintended in a current or future version of the target class. Having multiple precise matchers is preferable to one more vague catch-all matcher which leaves some method characteristics undefined.

Instrumentation class names should end in Instrumentation.

Helper Classes

Classes referenced by Advice that are not provided on the bootclasspath must be defined in Helper Classes otherwise they will not be loaded at runtime. This included any decorators, extractors/injectors or wrapping classes such as tracing listeners that extend or implement types provided by the library being instrumented. Also watch out for implicit types such as anonymous/nested classes because they must be listed alongside the main helper class.

If an instrumentation is producing no results it may be that a required class is missing. Running muzzle

./gradlew muzzle

can quickly tell you if you missed a required helper class. Messages like this in debug logs also indicate that classes are missing:

[MSC service thread 1-3] DEBUG datadog.trace.agent.tooling.muzzle.MuzzleCheck - Muzzled mismatch - instrumentation.names=[jakarta-mdb] instrumentation.class=datadog.trace.instrumentation.jakarta.jms.MDBMessageConsumerInstrumentation instrumentation.target.classloader=ModuleClassLoader for Module "deployment.cmt.war" from Service Module Loader muzzle.mismatch="datadog.trace.instrumentation.jakarta.jms.MessageExtractAdapter:20 Missing class datadog.trace.instrumentation.jakarta.jms.MessageExtractAdapter$1"

The missing class must be added in the helperClassNames method, for example:

@Override
public String[] helperClassNames() {
    return new String[]{
            "datadog.trace.instrumentation.jakarta.jms.MessageExtractAdapter",
            "datadog.trace.instrumentation.jakarta.jms.JMSDecorator",
            "datadog.trace.instrumentation.jakarta.jms.MessageExtractAdapter$1"
    };
}

Enums

Use care when deciding to include enums in your Advice and Decorator classes because each element of the enum will need to be added to the helper classes individually. For example not just MyDecorator.MyEnum but also MyDecorator.MyEnum$1, MyDecorator.MyEnum$2, etc.

Decorator Classes

Decorators contain extra code that will be injected into the instrumented methods.

These provided Decorator classes sit in dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator
Decorator Parent Class Usage(see JavaDoc for more detail)
AppSecUserEventDecorator - Provides mostly login-related functions to the Spring Security instrumentation.
AsyncResultDecorator BaseDecorator Handles asynchronous result types, finishing spans only when the async calls are complete.
BaseDecorator - Provides many convenience methods related to span naming and error handling. New Decorators should extend BaseDecorator or one of its child classes.
ClientDecorator BaseDecorator Parent of many Client Decorators. Used to set client specific tags, serviceName, etc
DBTypeProcessingDatabaseClientDecorator DatabaseClientDecorator Adds automatic processDatabaseType() call to DatabaseClientDecorator.
DatabaseClientDecorator ClientDecorator Provides general db-related methods.
HttpClientDecorator UriBasedClientDecorator Mostly adds span tags to HTTP client requests and responses.
HttpServerDecorator ServerDecorator Adds connection and HTTP response tagging often used for server frameworks.
MessagingClientDecorator ClientDecorator Adds e2e (end-to-end) duration monitoring.
OrmClientDecorator DatabaseClientDecorator Set the span’s resourceName to the entityName value.
ServerDecorator BaseDecorator Adding server and language tags to the span.
UriBasedClientDecorator ClientDecorator Adds hostname, port and service values from URIs to HttpClient spans.
UrlConnectionDecorator UriBasedClientDecorator Sets some tags based on URI and URL values. Also provides some caching. Only used by UrlInstrumentation.

Instrumentations often include their own Decorators which extend those classes, for example:
Instrumentation Decorator Parent Class
JDBC DataSourceDecorator BaseDecorator
RabbitMQ RabbitDecorator MessagingClientDecorator
All HTTP Server frameworks various HttpServerDecorator

Decorator class names must be in the instrumentation's helper classes since Decorators need to be loaded with the instrumentation.

Decorator class names should end in Decorator.

Advice Classes

Byte Buddy injects compiled bytecode at runtime to wrap existing methods, so they communicate with Datadog at entry or exit. These modifications are referred to as advice transformation or just advice.

Instrumenters register advice transformations by calling

AdviceTransformation.applyAdvice(ElementMatcher, String) and Methods are matched by the instrumentation's adviceTransformations() method.

The Advice is injected into the type so Advice can only refer to those classes on the bootstrap class-path or helpers injected into the application class-loader. Advice must not refer to any methods in the instrumentation class or even other methods in the same advice class because the advice is really only a template of bytecode to be inserted into the target class. It is only the advice bytecode (plus helpers) that is copied over. The rest of the instrumenter and advice class is ignored. Do not place code in the Advice constructor because the constructor is never called.

You can not use methods like InstrumentationContext.get()outside of the instrumentation advice because the tracer currently patches the method stub with the real call at runtime. But you can pass the ContextStore into a helper/decorator like in DatadogMessageListener. This could reduce duplication if you re-used the helper. But unlike most applications, some duplication can be the better choice in the tracer if it simplifies things and reduces overhead. You might end up with very similar code scattered around, but it will be simple to maintain. Trying to find an abstraction that works well across instrumentations can take time and may introduce extra indirection.

Advice classes provide the code to be executed before and/or after a matched method. The classes use a static method annotated by @Advice.OnMethodEnter and/or @Advice.OnMethodExit to provide the code. The method name is irrelevant.

A method that is annotated with @Advice.OnMethodEnter can annotate its parameters with @Advice.Argument. @Advice.Argument will substitute this parameter with the corresponding argument of the instrumented method. This allows the @Advice.OnMethodEnter code to see and modify the parameters that would be passed to the target method.

Alternatively, a parameter can be annotated by Advice.This where the this reference of the instrumented method is assigned to the new parameter. This can also be used to assign a new value to the this reference of an instrumented method.

If no annotation is used on a parameter, it is assigned the n-th parameter of the instrumented method for the n-th parameter of the advice method. Explicitly specifying which parameter is intended is recommended to be more clear, for example:

@Advice.Argument(0) final HttpUriRequest request

All parameters must declare the exact same type as the parameters of the instrumented type or the method's declaring type for Advice.This. If they are marked as read-only, then the parameter type may be a super type of the original.

A method that is annotated with Advice.OnMethodExit can also annotate its parameters with Advice.Argument and Advice.This. It can also annotate a parameter with Advice.Return to receive the original method's return value. By reassigning the return value, it can replace the returned value. If an instrumented method does not return a value, this annotation must not be used. If a method throws an exception, the parameter is set to its default value (0 for primitive types and to null for reference types). The parameter's type must equal the instrumented method's return type if it is not set to read-only. If the parameter is read-only it may be a super type of the instrumented method's return type.

Advice class names should end in Advice.

Exceptions in Advice

Advice methods are typically annotated like

@Advice.OnMethodEnter(suppress = Throwable.class)

and

@Advice.OnMethodExit(onThrowable = Throwable.class, suppress = Throwable.class)

Using suppress = Throwable.class is considered our default for both unless there is a reason not to. It means the exception handler is triggered on any exception thrown within the Advice and the Advice method terminates. The opposite would be either no suppress annotation or equivalently suppress = NoExceptionHandler.class which would both allow exceptions in Advice code to surface and is usually undesirable.

If the Advice.OnMethodEnter method throws an exception, the Advice.OnMethodExit method is not invoked.

The Advice.Thrown annotation passes any thrown exception from the instrumented method to the Advice.OnMethodExit advice method. Advice.Thrown **** should annotate at most one parameter on the exit advice.

If the instrumented method throws an exception, the Advice.OnMethodExit method is still invoked unless the Advice.OnMethodExit.onThrowable() property is set to false. If this property is set to false, the Advice.Thrown annotation must not be used on any parameter.

If an instrumented method throws an exception, the return parameter is set to its default of 0 for primitive types or null for reference types. An exception can be read by annotating an exit method’s Throwable parameter with Advice.Thrown which is assigned the thrown Throwable or null if a method returns normally. This allows exchanging a thrown exception with any checked or unchecked exception. For example, either the result or the exception will be passed to the helper method here:

@Advice.OnMethodExit(onThrowable = Throwable.class, suppress = Throwable.class)
public static void methodExit(
        @Advice.Return final Object result,
        @Advice.Thrown final Throwable throwable
) {
    HelperMethods.doMethodExit(result, throwable);
}

InjectAdapters & Custom GETTERs/SETTERs

Custom Inject Adapter static instances typically named SETTER implement the AgentPropagation.Setter interface and are used to normalize setting shared context values such as in HTTP headers.

Custom inject adapter static instances typically named GETTER implement the AgentPropagation.Getter interface and are used to normalize extracting shared context values such as from HTTP headers.

For example google-http-client sets its header values using:

com.google.api.client.http.HttpRequest.getHeaders().put(key,value)

package datadog.trace.instrumentation.googlehttpclient;

import com.google.api.client.http.HttpRequest;
import datadog.trace.bootstrap.instrumentation.api.AgentPropagation;

public class HeadersInjectAdapter implements AgentPropagation.Setter<HttpRequest> {
    public static final HeadersInjectAdapter SETTER = new HeadersInjectAdapter();

    @Override
    public void set(final HttpRequest carrier, final String key, final String value) {
        carrier.getHeaders().put(key, value);
    }
}

But notice apache-http-client5 sets its header values using:

org.apache.hc.core5.http.HttpRequest.setHeader(key,value)

package datadog.trace.instrumentation.apachehttpclient5;

import datadog.trace.bootstrap.instrumentation.api.AgentPropagation;
import org.apache.hc.core5.http.HttpRequest;

public class HttpHeadersInjectAdapter implements AgentPropagation.Setter<HttpRequest> {
    public static final HttpHeadersInjectAdapter SETTER = new HttpHeadersInjectAdapter();

    @Override
    public void set(final HttpRequest carrier, final String key, final String value) {
        carrier.setHeader(key, value);
    }
}

These implementation-specific methods are both wrapped in a standard set(...) method by the SETTER.

To Wrap or Not To Wrap?

Typically, an instrumentation will use ByteBuddy to apply new code from an Advice class before and/or after the targeted code using @Advice.OnMethodEnter and @Advice.OnMethodExit.

Alternatively, you can replace the call to the target method with your own code which wraps the original method call. An example is the JMS Instrumentation which replaces the MessageListener.onMessage() method with DatadogMessageListener.onMessage(). The DatadogMessageListener then calls the original onMessage() method. Note that this style is not recommended because it can cause datadog packages to appear in stack traces generated by errors in user code. This has created confusion in the past.

Context Stores

Context stores pass information between instrumented methods, using library objects that both methods have access to. They can be used to attach data to a request when the request is received, and read that data where the request is deserialized. Context stores work internally by dynamically adding a field to the “carrier” object by manipulating the bytecode. Since they manipulate bytecode, context stores can only be created within Advice classes. For example:

ContextStore<X> store = InstrumentationContext.get(
        "com.amazonaws.services.sqs.model.ReceiveMessageResult", "java.lang.String");

It’s also possible to pass the types as class objects, but this is only possible for classes that are in the bootstrap classpath. Basic types like String would work and the usual datadog types like AgentSpan are OK too, but classes from the library you are instrumenting are not.

In the example above, that context store is used to store an arbitrary String in a ReceiveMessageResult class. It is used like a Map:

store.put(response, "my string");

and/or

String stored = store.get(response); // "my string"

Context stores also need to be pre-declared in the Advice by overriding the contextStore() method otherwise, using them throws exceptions.

@Override
public Map<String, String> contextStore() {
    return singletonMap(
            "com.amazonaws.services.sqs.model.ReceiveMessageResult",
            "java.lang.String"
    );
}

It is important to understand that even though they look like maps, since the value is stored in the key, you can only retrieve a value if you use the exact same key object as when it was set. Using a different object that is “.equals()” to the first will yield nothing.

Since ContextStore does not support null keys, null checks must be enforced before using an object as a key.

CallDepthThreadLocalMap

In order to avoid activating new spans on recursive calls to the same method a CallDepthThreadLocalMap is often used to determine if a call is recursive by using a counter. It is incremented with each call to the method and decremented ( or reset) when exiting.

This only works if the methods are called on the same thread since the counter is a ThreadLocal variable.

Span Lifecycle

In Advice classes, the @Advice.OnMethodEnter methods typically start spans and @Advice.OnMethodExit methods typically finish spans.

Starting the span may be done directly or with helper methods which eventually make a call to one of the various AgentTracer.startSpan(...) methods.

Finishing the span is normally done by calling span.finish() in the exit method;

The basic span lifecycle in an Advice class looks like:

  1. Start the span

  2. Decorate the span

  3. Activate the span and get the AgentScope

  4. Run the instrumented target method

  5. Close the Agent Scope

  6. Finish the span

@Advice.OnMethodEnter(suppress = Throwable.class)
public static AgentScope begin() {
    final AgentSpan span = startSpan(/* */);
    DECORATE.afterStart(span);
    return activateSpan(span);
}

@Advice.OnMethodExit(suppress = Throwable.class)
public static void end(@Advice.Enter final AgentScope scope) {
    AgentSpan span = scope.span();
    DECORATE.beforeFinish(span);
    scope.close();
    span.finish();
}

For example, the HttpUrlConnectionInstrumentation class contains the HttpUrlConnectionAdvice class which calls the HttpUrlState.start() and HttpUrlState.finishSpan() methods.

Continuations

  • AgentScope.Continuation is used to pass context between threads.
  • Continuations must be either activated or canceled.
  • If a Continuation is activated it returns a TraceScope which must eventually be closed.
  • Only after all TraceScopes are closed and any non-activated Continuations are canceled may the Trace finally close.

Notice in HttpClientRequestTracingHandler how the AgentScope.Continuation is used to obtain the parentScope which is finally closed.

Naming

  • Instrumentation names use kebab case. For example: google-http-client
  • Instrumentation module name and package name should be consistent. For example, the instrumentation google-http-client contains the GoogleHttpClientInstrumentation class in the package datadog.trace.instrumentation.googlehttpclient.
  • As usual, class names should be nouns, in camel case with the first letter of each internal word capitalized. Use whole words-avoid acronyms and abbreviations (unless the abbreviation is much more widely used than the long form, such as URL or HTML).
  • Advice class names should end in Advice.
  • Instrumentation class names should end in Instrumentation.
  • Decorator class names should end in Decorator.

Tooling

ignored_class_name.trie

The file ignored_class_name.trie lists classes that are to be globally ignored by matchers because they are unsafe, pointless or expensive to transform. If you notice an expected class is not being transformed, it may be covered by an entry in this list.

GraalVM

Instrumentations running on GraalVM should avoid using reflection if possible. If reflection must be used the reflection usage should be added to

dd-java-agent/agent-bootstrap/src/main/resources/META-INF/native-image/com.datadoghq/dd-java-agent/reflect-config.json

See GraalVM configuration docs.

Testing

Instrumentation Tests

Tests are written in Groovy using the Spock framework. For instrumentations, AgentTestRunner must be extended. For example, HTTP server frameworks use base tests which enforce consistency between different implementations ( see HttpServerTest). When writing an instrumentation it is much faster to test just the instrumentation rather than build the entire project, for example:

./gradlew :dd-java-agent:instrumentation:play-ws-2.1:test

Sometimes it is necessary to force Gradle to discard cached test results and rerun all tasks.

./gradle test --rerun-tasks

Running tests that require JDK-21 will require the JAVA_21_HOME env var set and can be done like this:

./gradlew  :dd-java-agent:instrumentation:aerospike-4:allLatestDepTests -PtestJvm=21

Latest Dependency Tests

Adding a directive to the build file gives early warning when breaking changes are released by framework maintainers. For example, for Play 2.5, we download the latest dependency and run tests against it:

latestDepTestCompile group: 'com.typesafe.play', name: 'play-java_2.11', version: '2.5.+'

latestDepTestCompile group: 'com.typesafe.play', name: 'play-java-ws_2.11', version: '2.5.+'

latestDepTestCompile(group: 'com.typesafe.play', name: 'play-test_2.11', version: '2.5.+') {
  exclude group: 'org.eclipse.jetty.websocket', module: 'websocket-client'
}

Dependency tests can be run like:

./gradlew :dd-java-agent:instrumentation:play-ws-2.1:latestDepTest

Additional Test Suites

The file dd-trace-java/gradle/test-suites.gradle contains these macros for adding different test suites to individual instrumentation builds. Notice how addTestSuite and addTestSuiteForDir pass values to addTestSuiteExtendingForDir which configures the tests.

ext.addTestSuite = (String testSuiteName) -> {
  ext.addTestSuiteForDir(testSuiteName, testSuiteName)
}

ext.addTestSuiteForDir = (String testSuiteName, String dirName) -> {
  ext.addTestSuiteExtendingForDir(testSuiteName, 'test', dirName)
}

ext.addTestSuiteExtendingForDir = (String testSuiteName, String parentSuiteName, String dirName) -> { /* */ }

For example:

addTestSuite('latestDepTest')

Also, the forked test for latestDep is not run by default without declaring something like:

addTestSuiteExtendingForDir('latestDepForkedTest', 'latestDepTest', 'test')

(also example vertx-web-3.5/build.gradle)

Smoke Tests

In addition to unit tests, Smoke tests may be needed. Smoke tests run with a real agent jar file set as the javaagent. These are optional and not all frameworks have them, but contributions are very welcome.

Summary

Integrations have evolved over time. Newer examples of integrations such as Spring and JDBC illustrate current best practices.

Additional Reading