Correlating Business Transactions with Business Transaction Logs in APM

Over the last two decades, software development in the enterprise has changed from applications in silos to distributed and composite applications. These new solutions connect multiple services, databases, and even mainframe computers to solve more complex business problems.

connectingIn this new solution development practice, the end users or computer-generated requests interact with a broad range of underlying systems that were once managed in a standalone fashion. From an operational support perspective, this can make troubleshooting complex when the connecting solution doesn’t make manageability an architectural guiding principle.

Now there is a convenient solution. We recommend the following method to correlate the system logs automatically to the business transactions via AppDynamics.

For those times when tracing application code doesn’t provide enough information to track down the cause of a bottleneck, APM provides visibility into the transaction logs that can be correlated to specific BT requests. Any additional information from application logs or machine logs can be helpful for your DevOps teams in investigating the problem. Keep in mind that log correlation visibility requires a license for both transaction analytics and log analytics.

By configuring BTs for a globally unique identifier (GUID) injection, you can correlate logs to specific BT requests. Correlating specific instances of your BTs to the related logs works by injecting the same request GUID of the BT into the associated logs through the Java agent. This helps to quickly find the relevant logs from multiple tiers and nodes for a specific BT.

Configuring GUID Injection

This feature is supported for two Java logging frameworks:

  • Apache log4j versions 1.2.12 to 1.2.17
  • Logback versions 1.0.0 to 1.1.3 (any version before 1.0.0 is not supported)

There are two steps to enabling transaction to log correlation. Both are required.

  • Select BTs for log correlation (this tells the Java agent which BTs you are interested in).
    1. In the Controller UI, click Analytics > Configuration.
    2. Select the application from the drop-down list.
    3. Scroll down to the section Configure Transactions for GUID Injection. Add transactions from the right-hand list to the left-hand list and click Save.
    4. Proceed to the next step. You must also configure the logging patterns before you will see correlated logs.
  • Define how the injected information appears in the logs. A pattern is responsible for formatting a logging request and an appender takes care of output destination. You need to know the appender name and pattern for your application logging framework, so you can properly configure this feature. You can configure this through the Controller UI as described here. You can also add the appender directly to the application code (described after this section).
    1. On the Analytics Configuration window, scroll to the section Configure Patterns for Logging format.
    2. Select application, tier, or node where you want to collect correlated logs.
    3. Enter the Appender name and choose the log framework.
    4. Enter the pattern and the request GUID string. You can add the request GUID anyplace in the pattern.

The request GUID must match the following exactly: [%X{AD.requestGUID}]

Add Appender Directly into the Source Code

If you have access to the source code for your application, you can also add the appender to the log4j.properties file directly. Here is an example of what that might look like with the request GUID in BOLD:

log4j.appender.order-file-appender=org.apache.log4j.FileAppender

log4j.appender.order-file-appender.File=logs/telecom-order.log

log4j.appender.order-file-appender.layout=org.apache.log4j.PatternLayout

log4j.appender.order-file-appender.layout.ConversionPattern=[%d]  [%-5p]  [%t]  [%c]  [%X{AD.requestGUID}]  %m%n

log4j.logger.com.appdynamics.order=DEBUG, order-file-appender

log4j.additivity.com.appdynamics.order=false

Viewing Correlated Logs and Transaction Data

Once you have configured GUID injection, you can search the log files from a number of points in the UI:

  • From a BT snapshot
  • Select relevant transactions from any transaction analytics search and see details
  • Select a log from any log analytics search and see the Event Details

 

Note: The information derived from this blog and illustrated examples are available at: https://docs.appdynamics.com/display/PRO42/Business+Transactions

Leave a comment