Test OpenTelemetry Observability of Your JVM Applications with Local Grafana Backend
Open Telemetry
Observability platforms are essential for understanding the internal state of our application and the infrastructure they run on. By leveraging Traces, Metrics and Logs data emitted by application and the infrastructure, we gain valuable insights into the application behaviour and performance. Given the numerous observability backends available out there, it is crucial to ensure that our applications and infrastructure when instrumented to generate Traces, Logs and Metrics are not locked into a single vendor. OpenTelemetry addresses this by providing a single set of APIs and specifications to send our instrumentation data to vendor of our choice.
Source code for this demo is available at:
https://github.com/achalise/springboot-observability
Micrometer: Enhancing Observability for JVM applications
For JVM applications, micrometer provides an additional level of abstraction with out of box instrumentation and a unified Observability API that is capable of produce traces, metrics and logs and making the integration process straight forward.
Setting Up a Local Observability Backend
Having a local set up for testing application observability is highly beneficial. grafana/otel-lgtm provides a complete observability backend in a single container which is ideal for demo and local development .
The OpenTelemetry Collector in the container receives OpenTelemetry signals on ports 4317 (gRPC) and 4318 (HTTP). It forwards
- metrics to a Prometheus database,
- spans to a Tempo database, and
- logs to a Loki database
Grafana has all three databases configured as data sources and exposes its Web UI on port 3000 for visualisation and analysis.
Application Configuration
In the set up for this article, we have two services — claimservice and paymentservice . We use micrometer to publish observability data via open telemetry to a local observability backend using grafana/otel-lgtm.
In claimservice we include the following dependencies to enable and export metrics and traces using micrometer to the local observability backend.
dependencies {
implementation("org.springframework.boot:spring-boot-starter-actuator")
implementation("org.springframework.boot:spring-boot-starter-web")
runtimeOnly ("io.micrometer:micrometer-registry-otlp")
implementation ("io.micrometer:micrometer-tracing-bridge-otel")
implementation ("io.opentelemetry:opentelemetry-exporter-otlp")
developmentOnly("org.springframework.boot:spring-boot-devtools")
developmentOnly("org.springframework.boot:spring-boot-docker-compose")
testImplementation("org.springframework.boot:spring-boot-starter-test")
testRuntimeOnly("org.junit.platform:junit-platform-launcher")
}The endpoints are defined in application.properties
management.otlp.metrics.export.url=http://localhost:4318/v1/metrics
management.otlp.tracing.endpoint=http://localhost:4318/v1/tracesIn payment-service , just for illustration, we make use of spring boot starter dependencies for convenience
dependencies {
implementation("org.springframework.boot:spring-boot-starter-actuator")
implementation("org.springframework.boot:spring-boot-starter-web")
implementation("com.grafana:grafana-opentelemetry-starter:1.4.0")
implementation("io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter")
testImplementation("org.springframework.boot:spring-boot-starter-test")
testRuntimeOnly("org.junit.platform:junit-platform-launcher")
}This set up will also publish metrics and traces to the local observability backend running on grafana/otel-lgtm by default.
Inside theclaimservice project, there is also a Docker Compose file compose.yaml used by development tools to start the container during application startup in development mode.
Custom instrumentation in our Application — Metrics and Traces Using Micrometer
We have defined a counter for every claim received successfully and also an error counter when there are exceptions. These counters are published as metrics to the observability backend by micrometer. We have also created two custom spans submit.claim and submit.payment .
@RestController
class ClaimController {
private Logger logger = LoggerFactory.getLogger(ClaimController.class);
private final ObservationRegistry observationRegistry;
private final Counter claimCounter;
private final Counter errorCounter;
private final RestClient.Builder restclientBuilder;
public ClaimController(ObservationRegistry observationRegistry, MeterRegistry meterRegistry, RestClient.Builder webClientBuilder) {
this.observationRegistry = observationRegistry;
this.claimCounter = Counter.builder("claim.counter")
.description("Counts Claim Submitted")
.tags("region", "us-east")
.register(meterRegistry);
this.errorCounter = Counter.builder("claim.error.counter")
.description("Counts Errors for Claim Submitted")
.tags("region", "us-east")
.register(meterRegistry);
this.restclientBuilder = webClientBuilder;
}
@PostMapping("/claim")
public Response submitClaim(@RequestBody Claim claim) {
logger.info("Received claim {}", claim);
Observation observation = Observation.createNotStarted("submit.claim", this.observationRegistry);
observation.lowCardinalityKeyValue("claimType", claim.type());
observation.highCardinalityKeyValue("claimAmount", claim.amount().toString());
AtomicReference<String> status = new AtomicReference<>("SUCCESS");
observation.observe(() -> {
// Execute business logic here
try {
Thread.sleep(30);
restclientBuilder.baseUrl("http://localhost:8081/payment").build().post().body(claim).retrieve();
claimCounter.increment();
if(RandomGenerator.getDefault().nextBoolean()) {
throw new RuntimeException("some error");
}
} catch (Exception _) {
errorCounter.increment();
observation.highCardinalityKeyValue("error", "error here");
status.set("ERROR");
}
});
return new Response(status.get(), UUID.randomUUID().toString());
}
}
record Response(String status, String correlationId){}
record Claim(String type, Double amount){}Running the Demo
To run the demo, follow these steps:
ClaimService
cd claimservice./gradlew bootRun
PaymentService
cd paymentservice./gradlew bootRun
Submit a few requests to generate data for metrics and traces:
POST localhost:8080/claim
Content-Type: application/json
{
"type": "FLOOD_DAMAGE",
"amount": 300.50,
"email": "testuser@email.com"
}Visualise in Grafana
We can now access grafana at http://localhost:3000 to explore observability data from our demo set up.
Conclusion
By combining OpenTelemetry and Micrometer, we achieve a flexible, vendor-agnostic observability solution for JVM applications. The grafana/otel-lgtm container further simplifies local development and testing, making it easier to integrate and visualise observability data.
Reference: