Semantic Conventions for System Metrics

Status: Experimental

This document describes instruments and attributes for common system level metrics in OpenTelemetry. Consider the general metric semantic conventions when creating instruments not explicitly defined in the specification.

The system.* namespace SHOULD be exclusively used to report hosts’ metrics. The system.* namespace SHOULD only be used when the metrics are collected from within the target system. (physical servers, virtual machines etc). Metrics collected from technology-specific, well-defined APIs (e.g. Kubelet’s API or container runtimes) should be reported under their respective namespace (e.g. k8s., container.). Resource attributes related to a host, SHOULD be reported under the host.* namespace.

Processor Metrics

Description: System level processor metrics captured under the namespace system.cpu.

Metric: system.cpu.time

This metric is recommended.

system.cpu.state has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

Metric: system.cpu.utilization

This metric is recommended.

system.cpu.state has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

Metric: system.cpu.physical.count

This metric is recommended.

Metric: system.cpu.logical.count

This metric is recommended.

Metric: system.cpu.frequency

This metric is recommended.

Memory Metrics

Description: System level memory metrics capture under the namespace system.memory. This does not include paging/swap memory.

Metric: system.memory.usage

This metric is recommended.

[1]: The sum over all system.memory.state values SHOULD equal the total memory available on the system, that is system.memory.limit.

system.memory.state has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

Metric: system.memory.limit

This metric is opt-in.

[1]: Its value SHOULD equal the sum of system.memory.state over all states.

Metric: system.memory.utilization

This metric is recommended.

system.memory.state has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

Paging/Swap Metrics

Description: System level paging/swap memory metrics captured under the namespace system.paging.

Metric: system.paging.usage

This metric is recommended.

system.paging.state MUST be one of the following:

Metric: system.paging.utilization

This metric is recommended.

system.paging.state MUST be one of the following:

Metric: system.paging.faults

This metric is recommended.

system.paging.type MUST be one of the following:

Metric: system.paging.operations

This metric is recommended.

system.paging.direction MUST be one of the following:

system.paging.type MUST be one of the following:

Disk Controller Metrics

Description: System level disk performance metrics captured under the namespace system.disk.

Metric: system.disk.io

This metric is recommended.

disk.io.direction MUST be one of the following:

Metric: system.disk.operations

This metric is recommended.

disk.io.direction MUST be one of the following:

Metric: system.disk.io_time

This metric is recommended.

[1]: The real elapsed time (“wall clock”) used in the I/O path (time from operations running in parallel are not counted). Measured as:

• Linux: Field 13 from procfs-diskstats
• Windows: The complement of “Disk% Idle Time” performance counter: uptime * (100 - "Disk\% Idle Time") / 100

Metric: system.disk.operation_time

This metric is recommended.

[1]: Because it is the sum of time each request took, parallel-issued requests each contribute to make the count grow. Measured as:

• Linux: Fields 7 & 11 from procfs-diskstats
• Windows: “Avg. Disk sec/Read” perf counter multiplied by “Disk Reads/sec” perf counter (similar for Writes)

disk.io.direction MUST be one of the following:

Metric: system.disk.merged

This metric is recommended.

disk.io.direction MUST be one of the following:

Filesystem Metrics

Description: System level filesystem metrics captured under the namespace system.filesystem.

Metric: system.filesystem.usage

This metric is recommended.

system.filesystem.state MUST be one of the following:

system.filesystem.type has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

Metric: system.filesystem.utilization

This metric is recommended.

system.filesystem.state MUST be one of the following:

system.filesystem.type has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

Network Metrics

Description: System level network metrics captured under the namespace system.network.

Metric: system.network.dropped

This metric is recommended.

[1]: Measured as:

• Linux: the drop column in /proc/dev/net (source)
• Windows: InDiscards/OutDiscards from GetIfEntry2

network.io.direction MUST be one of the following:

Metric: system.network.packets

This metric is recommended.

network.io.direction MUST be one of the following:

Metric: system.network.errors

This metric is recommended.

[1]: Measured as:

• Linux: the errs column in /proc/dev/net (source).
• Windows: InErrors/OutErrors from GetIfEntry2.

network.io.direction MUST be one of the following:

Metric: system.network.io

This metric is recommended.

network.io.direction MUST be one of the following:

Metric: system.network.connections

This metric is recommended.

[1]: The value SHOULD be normalized to lowercase.

Consider always setting the transport when setting a port number, since a port number is ambiguous without knowing the transport. For example different processes could be listening on TCP port 12345 and UDP port 12345.

network.transport has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

system.network.state MUST be one of the following:

Aggregate System Process Metrics

Description: System level aggregate process metrics captured under the namespace system.process. For metrics at the individual process level, see process metrics.

Metric: system.processes.count

This metric is recommended.

system.processes.status has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

Metric: system.processes.created

This metric is recommended.

system.{os}. - OS Specific System Metrics

Instrument names for system level metrics that have different and conflicting meaning across multiple OSes should be prefixed with system.{os}. and follow the hierarchies listed above for different entities like CPU, memory, and network.

For example, UNIX load average over a given interval is not well standardized and its value across different UNIX like OSes may vary despite being under similar load:

Without getting into the vagaries of every Unix-like operating system in existence, the load average more or less represents the average number of processes that are in the running (using the CPU) or runnable (waiting for the CPU) states. One notable exception exists: Linux includes processes in uninterruptible sleep states, typically waiting for some I/O activity to complete. This can markedly increase the load average on Linux systems.

(source of quote, linux source code)

An instrument for load average over 1 minute on Linux could be named system.linux.cpu.load_1m, reusing the cpu name proposed above and having an {os} prefix to split this metric across OSes.

Metric: system.linux.memory.available

[1]: This is an alternative to system.memory.usage metric with state=free. Linux starting from 3.14 exports “available” memory. It takes “free” memory as a baseline, and then factors in kernel-specific values. This is supposed to be more accurate than just “free” memory. For reference, see the calculations here. See also MemAvailable in /proc/meminfo.