Semantic Conventions for System Metrics

Status: Experimental

This document describes instruments and attributes for common system level metrics in OpenTelemetry. Consider the general metric semantic conventions when creating instruments not explicitly defined in the specification.

The system.* namespace SHOULD be exclusively used to report hosts’ metrics. The system.* namespace SHOULD only be used when the metrics are collected from within the target system. (physical servers, virtual machines etc). Metrics collected from technology-specific, well-defined APIs (e.g. Kubelet’s API or container runtimes) should be reported under their respective namespace (e.g. k8s., container.). Resource attributes related to a host, SHOULD be reported under the host.* namespace.

Processor Metrics

Description: System level processor metrics captured under the namespace system.cpu.

Metric: system.cpu.time

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.cpu.timeCountersSeconds each logical CPU spent on each mode
AttributeTypeDescriptionExamplesRequirement Level
system.cpu.logical_numberintThe logical CPU number [0..n-1]1Recommended
system.cpu.statestringThe state of the CPUidle; interruptRecommended

system.cpu.state has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

ValueDescription
useruser
systemsystem
nicenice
idleidle
iowaitiowait
interruptinterrupt
stealsteal

Metric: system.cpu.utilization

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.cpu.utilizationGauge1Difference in system.cpu.time since the last measurement, divided by the elapsed time and number of logical CPUs
AttributeTypeDescriptionExamplesRequirement Level
system.cpu.logical_numberintThe logical CPU number [0..n-1]1Recommended
system.cpu.statestringThe state of the CPUidle; interruptRecommended

system.cpu.state has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

ValueDescription
useruser
systemsystem
nicenice
idleidle
iowaitiowait
interruptinterrupt
stealsteal

Metric: system.cpu.physical.count

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.cpu.physical.countUpDownCounter{cpu}Reports the number of actual physical processor cores on the hardware

Metric: system.cpu.logical.count

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.cpu.logical.countUpDownCounter{cpu}Reports the number of logical (virtual) processor cores created by the operating system to manage multitasking

Metric: system.cpu.frequency

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.cpu.frequencyGauge{Hz}Reports the current frequency of the CPU in Hz
AttributeTypeDescriptionExamplesRequirement Level
system.cpu.logical_numberintThe logical CPU number [0..n-1]1Recommended

Memory Metrics

Description: System level memory metrics capture under the namespace system.memory. This does not include paging/swap memory.

Metric: system.memory.usage

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.memory.usageUpDownCounterByReports memory in use by state. [1]

[1]: The sum over all system.memory.state values SHOULD equal the total memory available on the system, that is system.memory.limit.

AttributeTypeDescriptionExamplesRequirement Level
system.memory.statestringThe memory statefree; cachedRecommended

system.memory.state has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

ValueDescription
usedused
freefree
sharedshared
buffersbuffers
cachedcached

Metric: system.memory.limit

This metric is opt-in.

NameInstrument TypeUnit (UCUM)Description
system.memory.limitUpDownCounterByTotal memory available in the system. [1]

[1]: Its value SHOULD equal the sum of system.memory.state over all states.

Metric: system.memory.utilization

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.memory.utilizationGauge1
AttributeTypeDescriptionExamplesRequirement Level
system.memory.statestringThe memory statefree; cachedRecommended

system.memory.state has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

ValueDescription
usedused
freefree
sharedshared
buffersbuffers
cachedcached

Paging/Swap Metrics

Description: System level paging/swap memory metrics captured under the namespace system.paging.

Metric: system.paging.usage

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.paging.usageUpDownCounterByUnix swap or windows pagefile usage
AttributeTypeDescriptionExamplesRequirement Level
system.paging.statestringThe memory paging statefreeRecommended

system.paging.state MUST be one of the following:

ValueDescription
usedused
freefree

Metric: system.paging.utilization

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.paging.utilizationGauge1
AttributeTypeDescriptionExamplesRequirement Level
system.paging.statestringThe memory paging statefreeRecommended

system.paging.state MUST be one of the following:

ValueDescription
usedused
freefree

Metric: system.paging.faults

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.paging.faultsCounter{fault}
AttributeTypeDescriptionExamplesRequirement Level
system.paging.typestringThe memory paging typeminorRecommended

system.paging.type MUST be one of the following:

ValueDescription
majormajor
minorminor

Metric: system.paging.operations

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.paging.operationsCounter{operation}
AttributeTypeDescriptionExamplesRequirement Level
system.paging.directionstringThe paging access directioninRecommended
system.paging.typestringThe memory paging typeminorRecommended

system.paging.direction MUST be one of the following:

ValueDescription
inin
outout

system.paging.type MUST be one of the following:

ValueDescription
majormajor
minorminor

Disk Controller Metrics

Description: System level disk performance metrics captured under the namespace system.disk.

Metric: system.disk.io

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.disk.ioCounterBy
AttributeTypeDescriptionExamplesRequirement Level
disk.io.directionstringThe disk IO operation direction.readRecommended
system.devicestringThe device identifier(identifier)Recommended

disk.io.direction MUST be one of the following:

ValueDescription
readread
writewrite

Metric: system.disk.operations

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.disk.operationsCounter{operation}
AttributeTypeDescriptionExamplesRequirement Level
disk.io.directionstringThe disk IO operation direction.readRecommended
system.devicestringThe device identifier(identifier)Recommended

disk.io.direction MUST be one of the following:

ValueDescription
readread
writewrite

Metric: system.disk.io_time

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.disk.io_timeCountersTime disk spent activated [1]

[1]: The real elapsed time (“wall clock”) used in the I/O path (time from operations running in parallel are not counted). Measured as:

AttributeTypeDescriptionExamplesRequirement Level
system.devicestringThe device identifier(identifier)Recommended

Metric: system.disk.operation_time

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.disk.operation_timeCountersSum of the time each operation took to complete [1]

[1]: Because it is the sum of time each request took, parallel-issued requests each contribute to make the count grow. Measured as:

  • Linux: Fields 7 & 11 from procfs-diskstats
  • Windows: “Avg. Disk sec/Read” perf counter multiplied by “Disk Reads/sec” perf counter (similar for Writes)
AttributeTypeDescriptionExamplesRequirement Level
disk.io.directionstringThe disk IO operation direction.readRecommended
system.devicestringThe device identifier(identifier)Recommended

disk.io.direction MUST be one of the following:

ValueDescription
readread
writewrite

Metric: system.disk.merged

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.disk.mergedCounter{operation}
AttributeTypeDescriptionExamplesRequirement Level
disk.io.directionstringThe disk IO operation direction.readRecommended
system.devicestringThe device identifier(identifier)Recommended

disk.io.direction MUST be one of the following:

ValueDescription
readread
writewrite

Filesystem Metrics

Description: System level filesystem metrics captured under the namespace system.filesystem.

Metric: system.filesystem.usage

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.filesystem.usageUpDownCounterBy
AttributeTypeDescriptionExamplesRequirement Level
system.devicestringThe device identifier(identifier)Recommended
system.filesystem.modestringThe filesystem moderw, roRecommended
system.filesystem.mountpointstringThe filesystem mount path/mnt/dataRecommended
system.filesystem.statestringThe filesystem stateusedRecommended
system.filesystem.typestringThe filesystem typeext4Recommended

system.filesystem.state MUST be one of the following:

ValueDescription
usedused
freefree
reservedreserved

system.filesystem.type has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

ValueDescription
fat32fat32
exfatexfat
ntfsntfs
refsrefs
hfsplushfsplus
ext4ext4

Metric: system.filesystem.utilization

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.filesystem.utilizationGauge1
AttributeTypeDescriptionExamplesRequirement Level
system.devicestringThe device identifier(identifier)Recommended
system.filesystem.modestringThe filesystem moderw, roRecommended
system.filesystem.mountpointstringThe filesystem mount path/mnt/dataRecommended
system.filesystem.statestringThe filesystem stateusedRecommended
system.filesystem.typestringThe filesystem typeext4Recommended

system.filesystem.state MUST be one of the following:

ValueDescription
usedused
freefree
reservedreserved

system.filesystem.type has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

ValueDescription
fat32fat32
exfatexfat
ntfsntfs
refsrefs
hfsplushfsplus
ext4ext4

Network Metrics

Description: System level network metrics captured under the namespace system.network.

Metric: system.network.dropped

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.network.droppedCounter{packet}Count of packets that are dropped or discarded even though there was no error [1]

[1]: Measured as:

AttributeTypeDescriptionExamplesRequirement Level
network.io.directionstringThe network IO operation direction.transmitRecommended
system.devicestringThe device identifier(identifier)Recommended

network.io.direction MUST be one of the following:

ValueDescription
transmittransmit
receivereceive

Metric: system.network.packets

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.network.packetsCounter{packet}
AttributeTypeDescriptionExamplesRequirement Level
network.io.directionstringThe network IO operation direction.transmitRecommended
system.devicestringThe device identifier(identifier)Recommended

network.io.direction MUST be one of the following:

ValueDescription
transmittransmit
receivereceive

Metric: system.network.errors

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.network.errorsCounter{error}Count of network errors detected [1]

[1]: Measured as:

AttributeTypeDescriptionExamplesRequirement Level
network.io.directionstringThe network IO operation direction.transmitRecommended
system.devicestringThe device identifier(identifier)Recommended

network.io.direction MUST be one of the following:

ValueDescription
transmittransmit
receivereceive

Metric: system.network.io

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.network.ioCounterBy
AttributeTypeDescriptionExamplesRequirement Level
network.io.directionstringThe network IO operation direction.transmitRecommended
system.devicestringThe device identifier(identifier)Recommended

network.io.direction MUST be one of the following:

ValueDescription
transmittransmit
receivereceive

Metric: system.network.connections

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.network.connectionsUpDownCounter{connection}
AttributeTypeDescriptionExamplesRequirement Level
network.transportstringOSI transport layer or inter-process communication method. [1]tcp; udpRecommended
system.devicestringThe device identifier(identifier)Recommended
system.network.statestringA stateless protocol MUST NOT set this attributeclose_waitRecommended

[1]: The value SHOULD be normalized to lowercase.

Consider always setting the transport when setting a port number, since a port number is ambiguous without knowing the transport. For example different processes could be listening on TCP port 12345 and UDP port 12345.

network.transport has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

ValueDescription
tcpTCP
udpUDP
pipeNamed or anonymous pipe.
unixUnix domain socket

system.network.state MUST be one of the following:

ValueDescription
closeclose
close_waitclose_wait
closingclosing
deletedelete
establishedestablished
fin_wait_1fin_wait_1
fin_wait_2fin_wait_2
last_acklast_ack
listenlisten
syn_recvsyn_recv
syn_sentsyn_sent
time_waittime_wait

Aggregate System Process Metrics

Description: System level aggregate process metrics captured under the namespace system.process. For metrics at the individual process level, see process metrics.

Metric: system.processes.count

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.processes.countUpDownCounter{process}Total number of processes in each state
AttributeTypeDescriptionExamplesRequirement Level
system.processes.statusstringThe process state, e.g., Linux Process State CodesrunningRecommended

system.processes.status has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

ValueDescription
runningrunning
sleepingsleeping
stoppedstopped
defunctdefunct

Metric: system.processes.created

This metric is recommended.

NameInstrument TypeUnit (UCUM)Description
system.processes.createdCounter{process}Total number of processes created over uptime of the host

system.{os}. - OS Specific System Metrics

Instrument names for system level metrics that have different and conflicting meaning across multiple OSes should be prefixed with system.{os}. and follow the hierarchies listed above for different entities like CPU, memory, and network.

For example, UNIX load average over a given interval is not well standardized and its value across different UNIX like OSes may vary despite being under similar load:

Without getting into the vagaries of every Unix-like operating system in existence, the load average more or less represents the average number of processes that are in the running (using the CPU) or runnable (waiting for the CPU) states. One notable exception exists: Linux includes processes in uninterruptible sleep states, typically waiting for some I/O activity to complete. This can markedly increase the load average on Linux systems.

(source of quote, linux source code)

An instrument for load average over 1 minute on Linux could be named system.linux.cpu.load_1m, reusing the cpu name proposed above and having an {os} prefix to split this metric across OSes.

Metric: system.linux.memory.available

NameInstrument TypeUnit (UCUM)Description
system.linux.memory.availableUpDownCounterByAn estimate of how much memory is available for starting new applications, without causing swapping [1]

[1]: This is an alternative to system.memory.usage metric with state=free. Linux starting from 3.14 exports “available” memory. It takes “free” memory as a baseline, and then factors in kernel-specific values. This is supposed to be more accurate than just “free” memory. For reference, see the calculations here. See also MemAvailable in /proc/meminfo.