mirai - OpenTelemetry

1. Introduction

mirai provides comprehensive OpenTelemetry (otel) tracing for observing async operations and distributed computation.

With otel and otelsdk installed and tracing enabled, mirai automatically creates spans tracking daemon management, async operations, and task execution lifecycles.

Detailed monitoring of:

• Task submission and completion times
• Daemon lifecycle and performance
• Error tracking and debugging
• Distributed tracing across network boundaries

2. Automatic Tracing Setup

Tracing enables automatically when:

1. otel and otelsdk packages are installed
2. OpenTelemetry tracing is configured and enabled (see https://otelsdk.r-lib.org/reference/collecting.html)

No additional action needed - mirai automatically detects OpenTelemetry and begins tracing.

3. Span Types and Hierarchy

mirai creates several span types representing different operations.

3.1 Core Span Types

daemons set / daemons reset

Root span for a compute profile, created when daemons are set/reset. Span name includes URL for easy identification.

• Kind: internal
• Attributes:
  • server.address (e.g. ‘127.0.0.1’ or ‘hostname’)
  • server.port (where applicable)
  • network.transport (e.g. ‘tcp’ or ‘ipc’)
  • mirai.dispatcher (true/false)
  • mirai.compute (profile name)

daemon connect / daemon disconnect

Daemon process span, created when a daemon connects/disconnects. Span name includes URL for easy identification.

• Kind: internal
• Attributes:
  • server.address (e.g. ‘127.0.0.1’ or ‘hostname’)
  • server.port (where applicable)
  • network.transport (e.g. ‘tcp’ or ‘ipc’)

mirai_map

Parallel map operation span encompassing the entire map operation across multiple mirai tasks.

• Kind: internal

mirai

Client-side async task span. Created when mirai() is called, ends when it returns.

• Kind: client
• Attributes: mirai.id (unique task identifier)

daemon eval

Server-side task evaluation span. Tracks actual mirai evaluation duration on the daemon.

• Kind: server

3.2 Span Relationships and Context Propagation

Spans form a distributed structure tracing the complete async operation lifecycle:

daemons set (compute profile - top level)
daemon connect (daemon process 1 - top level)
...
daemon connect (daemon process N - top level)

mirai_map (top level) ──link→ daemons set
├── mirai (task 1) ──link→ daemons set
│   └── daemon eval ──link→ daemon connect
├── mirai (task 2) ──link→ daemons set
│   └── daemon eval ──link→ daemon connect
└── mirai (task N) ──link→ daemons set
    └── daemon eval ──link→ daemon connect
    
mirai (top level) ──link→ daemons set
└── daemon eval ──link→ daemon connect

daemons reset ──link→ daemons set
daemon disconnect (daemon process 1) ──link→ daemon connect
...
daemon disconnect (daemon process N) ──link→ daemon connect

Context Propagation: Context automatically packaged with each mirai() call and extracted on the daemon side, enabling proper parent-child relationships across process boundaries.

Span Links: Tasks link to their compute profile’s daemons set span (client-side) and each daemon connect span (server-side), showing exactly where evaluation occurred. When daemons reset and disconnect, new spans record these events linking back to the originals.

4. Status and Error Tracking

daemon eval spans automatically track operation success or failure.

Status Values:

• 'ok' or 'unset' - successful completion
• 'error' with description 'miraiError' - error failure
• 'error' with description 'miraiInterrupt' - interrupted

5. Monitoring and Observability

OpenTelemetry spans provide rich observability into mirai operations.

Performance Monitoring:

• Track task execution times (submission to completion)
• Monitor daemon utilization and load balancing
• Identify distributed computation bottlenecks

Error Analysis:

• Correlate errors with specific tasks and daemons
• Track error rates across operation types
• Debug distributed environment issues

Distributed Tracing:

• Follow task execution across network boundaries
• Understand complete async operation lifecycles
• Correlate client-side requests with server-side execution

6. Integration with Observability Platforms

mirai’s OpenTelemetry implementation works seamlessly with any compatible observability platform:

• Grafana
• Pydantic Logfire
• Jaeger
• Any listed at https://otelsdk.r-lib.org/reference/collecting.html

The tracer name org.r-lib.mirai enables easy filtering and identification of mirai-related traces.