How to Send Traces Using Scala and OpenTelemetry

When working with distributed systems, traceability is crucial for understanding system behavior. If you are using Scala and OpenTelemetry to send traces, you may encounter some challenges, particularly if your traces are not appearing in the tracing backend like Zipkin or Tempo.

Why Might Traces Not Be Sent?

Traces not appearing in your tracing backend can be attributed to several issues. These issues might stem from misconfiguration in your OpenTelemetry setup, lack of proper attributes, or even potential threading issues with Scala. It’s essential to verify that the OpenTelemetry collector is up and running and that the endpoint is correctly configured, which you have indicated is ready.

In Scala, the ThreadLocal context propagation can behave differently compared to Java. If the Java code functions while the Scala code does not, there may be crucial differences in how contexts are managed. Let’s delve into the provided Scala code to ensure every detail is correct.

The Scala Code Explanation

Here’s the original Scala code you shared:

import io.opentelemetry.api.common.{AttributeKey, Attributes}
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter
import io.opentelemetry.sdk.OpenTelemetrySdk
import io.opentelemetry.sdk.trace.SdkTracerProvider
import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor
import java.util.concurrent.TimeUnit

object Main {
  def main(args: Array[String]): Unit = {
    try {
      // Configure the exporter with explicit endpoint and timeout
      val spanExporter = OtlpGrpcSpanExporter.builder()
        .setEndpoint("http://localhost:4317")
        .setTimeout(5, TimeUnit.SECONDS)
        .build()

      val sdkTracerProvider = SdkTracerProvider.builder()
        .addSpanProcessor(SimpleSpanProcessor.create(spanExporter))
        .build()

      val openTelemetry = OpenTelemetrySdk.builder()
        .setTracerProvider(sdkTracerProvider)
        .buildAndRegisterGlobal()

      val attributes = Attributes.of[java.lang.String, java.lang.Long](
        AttributeKey.stringKey("foo"), "bar",
        AttributeKey.longKey("code"), 42L
      )

      val tracer = openTelemetry.getTracer("my-notebook")
      val span = tracer.spanBuilder("My-span-ScalaEM").startSpan()
      val scope = span.makeCurrent()
      try {
        span.addEvent("ScalaEM span event", attributes)
        println("ScalaEM Log from span")
        Thread.sleep(1000L) // Simulating work
      }
      finally {
        scope.close()
        span.end()
      }
      println("trace sent")

      // Add a small delay to ensure the span is exported
      Thread.sleep(1000L)

      openTelemetry.shutdown()
      println("Done")
    } catch {
      case e: Exception =>
        println(s"Error occurred: ${e.getMessage}")
        e.printStackTrace()
    }
  }
}

Step-by-Step Breakdown

1. Configuring the Exporter: You have configured the OtlpGrpcSpanExporter with the correct endpoint and timeout. This is critical, and it looks properly set up.
2. Span Processor: The use of SimpleSpanProcessor to handle the span export is appropriate, ensuring that spans are sent to the defined exporter.
3. Creating Attributes: The attributes for the span are defined properly. Ensure that they are meaningful for your use case and that there are no typos.
4. Starting a Span: The span is correctly started and closed after the workload simulation.
5. Logging: The logging statements are within scope, ensuring traceability during the execution.

Potential Solutions for the Issue

1. Check the OpenTelemetry Collector Logs

Check the logs of your OpenTelemetry collector to see if there are any indications that spans are being received or if there are errors that might provide clues.

2. Context Propagation in Scala

Due to the differences in how contexts are handled in Scala, you need to ensure that the ThreadLocal variables (like span contexts) are properly managed. You might consider using the Scope API effectively to manage your span context even when calling other methods that could be interacting with different threads.

3. Add a Service Name to Your Tracer

Ensure that your tracer has a service name. This can help in the identification of traces in the backend. Modify your tracer initialization as follows:

val tracer = openTelemetry.getTracer("my-notebook", "1.0.0")

4. Ensure Correct Dependencies

Verify that you are including all necessary dependencies for OpenTelemetry Scala. Ensure there are no version mismatches.

Frequently Asked Questions (FAQ)

Q1: Will using Scala affect the performance of OpenTelemetry?

A1: Scala itself shouldn't impact the performance of OpenTelemetry. However, improper context management could lead to tracing issues.

Q2: Is there a difference in how Scala and Java manage tracing?

A2: Yes, Scala’s handling of threading can differ from Java’s, particularly with ThreadLocal variables and context propagation.

Q3: Can I debug my setup easily?

A3: Yes, debugging capabilities depend on your OpenTelemetry collector configuration and the backend you are using. Always enable verbose logging for better insights.

Conclusion

After reviewing your Scala code, the setup for sending traces with OpenTelemetry appears mostly correct. The crucial steps now involve verifying backend configurations, checking logs, and ensuring that context propagation aligns with what OpenTelemetry expects. Adjusting trace handling nuances between Scala and Java will likely lead to successful trace visibility in systems like Zipkin or Tempo in no time.