{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/zml-ai/2026-03-30-introducing-zml-smi/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "`zml-smi` is a universal diagnostic and monitoring tool for GPUs, TPUs and NPUs. It provides real-time insights into the performance and health of your hardware.\n\n![](https://zml.ai/posts/zml-smi/1.png)\n\nIt is a mix between `nvidia-smi` and `nvtop`.\n\nIt transparently supports all the platforms ZML supports. That is NVIDIA, AMD, Google TPU and AWS Trainium devices. It will be extended to support more platforms in the future as ZML continues to expand its hardware support.\n\n## Getting started\n\nYou can download [zml-smi from the official mirror](https://mirror.zml.ai/zml-smi/zml-smi-v0.2.tar.zst).\n\n```\n$ curl -LO 'https://mirror.zml.ai/zml-smi/zml-smi-v0.2.tar.zst'\n$ tar -xf zml-smi-v0.2.tar.zst\n$ ./zml-smi/zml-smi\n```\n\n## Listing devices\n\n```\n$ zml-smi\n```\n\n## Monitoring devices\n\nThe `--top` flag provides real-time monitoring of device performance, including utilization, temperature, and memory usage.\n\n```\n$ zml-smi --top\n```\n\n![](https://zml.ai/posts/zml-smi/2.png) ![](https://zml.ai/posts/zml-smi/3.png)\n\n## Completely sandboxed\n\n`zml-smi` doesn’t require any software on the target machine besides the device driver and the GLIBC (mostly due to the fact that some shared objects from vendors are loaded).\n\n## Metrics\n\n## Host\n\n`zml-smi` displays host-level metrics such as CPU model and utilization, memory usage, and temperature.\n\n### Available metrics\n\nHostname, Kernel, CPU Model, CPU Core Count, Memory Used / Total, Uptime, Load Average (1m / 5m / 15m), Device Count\n\n## Processes\n\n`zml-smi` also provides insights into the processes utilizing the devices, including their resource usage and command lines. This is available for all platforms.\n\n### Available metrics\n\nPID, Device Index, Device Utilization, Device Memory, Process Command Line\n\n## NVIDIA\n\nMetrics are given through the [NVML](https://developer.nvidia.com/management-library-nvml) library, which ships with the driver. As such, it is expected to be on the system.\n\n### Available metrics\n\nGPU Utilization, Temperature, Power Draw, Encoder Utilization, Decoder Utilization, VRAM Used, VRAM Total, Memory Bus Width, Temperature, Fan Speed, Power Draw, Power Limit, Graphics Clock, SM Clock, Memory Clock, Max Graphics Clock, Max Memory Clock, PCIe Link Generation, PCIe Link Width, PCIe TX Throughput, PCIe RX Throughput\n\n## AMD\n\nMetrics are provided through the [AMD SMI](https://rocm.docs.amd.com/projects/amdsmi/en/latest/) library. `zml-smi` ships with it in its sandbox.\n\nIn order to support the latest AMD GPUs, `zml-smi` at build time downloads the `amdgpu.ids` file from both Mesa and ROCm (7.2.1 at the time of this article) and merges them together. This allows `zml-smi` to recognize and report on the latest AMD GPU models, even if they are not yet included in the official ROCm release. This is the case for Ryzen AI Max+ 395 (Strix Halo) for instance.\n\nSandboxing that file turned somewhat tricky. Because `libdrm-amdgpu` expects to find it in `/opt/amdgpu/share/libdrm/amdgpu.ids`, we had to get a bit creative. We didn’t want to install anything outside the binary sandbox. Nor did we want to patch that string inside `libdrm`.\n\nSo we created a shared object named `zmlxrocm.so` that is added to the `DT_NEEDED` section of `libdrm_amdgpu.so.1`. Then, `fopen64` is renamed to `zmlxrocm_fopen64`, which is then provided by `zmlxrocm.so`. Since we now sit between `libdrm` and `fopen64`, we can intercept the call to `fopen64`, compare the path against `/opt/amdgpu/share/libdrm/amdgpu.ids` and redirect it to the sandboxed copy of the file.\n\n### Available metrics\n\nGPU Utilization, Memory Usage, Temperature, Power Draw, VRAM Used, VRAM Total, Temperature, Fan Speed, Power Draw, Power Limit, Graphics Clock, SoC Clock, Memory Clock, Max Graphics Clock, Max Memory Clock, PCIe Bandwidth, PCIe Link Generation, PCIe Link Width\n\n## TPU\n\nMetrics are provided via the local gRPC endpoint exposed by the TPU runtime. Those are the same metrics exposed to the [tpu-info](https://pypi.org/project/tpu-info/) tool from Google.\n\n### Available metrics\n\nTensorCore Duty Cycle, HBM Used, HBM Total\n\n## AWS Trainium\n\nMetrics are provided through a private API found in `libnrt.so`, which `zml-smi` embeds in its sandbox. Those are the same metrics provided by the [neuron-top](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-sys-tools/neuron-top-user-guide.html) utility.\n\n### Available metrics\n\nCore Utilization, HBM Used, HBM Total, Tensor Memory, Constant Memory, Model Code, Shared Scratchpad, Nonshared Scratchpad, Runtime Memory, Driver Memory, DMA Rings, Collectives, Notifications\n", "authors": [], "captured_at": "2026-09-08T10:17:10.166315+00:00", "category": "ai", "content_html": "

zml-smi is a universal diagnostic and monitoring tool for GPUs, TPUs and NPUs. It provides real-time insights into the performance and health of your hardware.

\n

\"\"

\n

It is a mix between nvidia-smi and nvtop.

\n

It transparently supports all the platforms ZML supports. That is NVIDIA, AMD, Google TPU and AWS Trainium devices. It will be extended to support more platforms in the future as ZML continues to expand its hardware support.

\n

Getting started

\n

You can download zml-smi from the official mirror.

\n
$ curl -LO 'https://mirror.zml.ai/zml-smi/zml-smi-v0.2.tar.zst'\n$ tar -xf zml-smi-v0.2.tar.zst\n$ ./zml-smi/zml-smi\n
\n

Listing devices

\n
$ zml-smi\n
\n

Monitoring devices

\n

The --top flag provides real-time monitoring of device performance, including utilization, temperature, and memory usage.

\n
$ zml-smi --top\n
\n

\"\" \"\"

\n

Completely sandboxed

\n

zml-smi doesn’t require any software on the target machine besides the device driver and the GLIBC (mostly due to the fact that some shared objects from vendors are loaded).

\n

Metrics

\n

Host

\n

zml-smi displays host-level metrics such as CPU model and utilization, memory usage, and temperature.

\n

Available metrics

\n

Hostname, Kernel, CPU Model, CPU Core Count, Memory Used / Total, Uptime, Load Average (1m / 5m / 15m), Device Count

\n

Processes

\n

zml-smi also provides insights into the processes utilizing the devices, including their resource usage and command lines. This is available for all platforms.

\n

Available metrics

\n

PID, Device Index, Device Utilization, Device Memory, Process Command Line

\n

NVIDIA

\n

Metrics are given through the NVML library, which ships with the driver. As such, it is expected to be on the system.

\n

Available metrics

\n

GPU Utilization, Temperature, Power Draw, Encoder Utilization, Decoder Utilization, VRAM Used, VRAM Total, Memory Bus Width, Temperature, Fan Speed, Power Draw, Power Limit, Graphics Clock, SM Clock, Memory Clock, Max Graphics Clock, Max Memory Clock, PCIe Link Generation, PCIe Link Width, PCIe TX Throughput, PCIe RX Throughput

\n

AMD

\n

Metrics are provided through the AMD SMI library. zml-smi ships with it in its sandbox.

\n

In order to support the latest AMD GPUs, zml-smi at build time downloads the amdgpu.ids file from both Mesa and ROCm (7.2.1 at the time of this article) and merges them together. This allows zml-smi to recognize and report on the latest AMD GPU models, even if they are not yet included in the official ROCm release. This is the case for Ryzen AI Max+ 395 (Strix Halo) for instance.

\n

Sandboxing that file turned somewhat tricky. Because libdrm-amdgpu expects to find it in /opt/amdgpu/share/libdrm/amdgpu.ids, we had to get a bit creative. We didn’t want to install anything outside the binary sandbox. Nor did we want to patch that string inside libdrm.

\n

So we created a shared object named zmlxrocm.so that is added to the DT_NEEDED section of libdrm_amdgpu.so.1. Then, fopen64 is renamed to zmlxrocm_fopen64, which is then provided by zmlxrocm.so. Since we now sit between libdrm and fopen64, we can intercept the call to fopen64, compare the path against /opt/amdgpu/share/libdrm/amdgpu.ids and redirect it to the sandboxed copy of the file.

\n

Available metrics

\n

GPU Utilization, Memory Usage, Temperature, Power Draw, VRAM Used, VRAM Total, Temperature, Fan Speed, Power Draw, Power Limit, Graphics Clock, SoC Clock, Memory Clock, Max Graphics Clock, Max Memory Clock, PCIe Bandwidth, PCIe Link Generation, PCIe Link Width

\n

TPU

\n

Metrics are provided via the local gRPC endpoint exposed by the TPU runtime. Those are the same metrics exposed to the tpu-info tool from Google.

\n

Available metrics

\n

TensorCore Duty Cycle, HBM Used, HBM Total

\n

AWS Trainium

\n

Metrics are provided through a private API found in libnrt.so, which zml-smi embeds in its sandbox. Those are the same metrics provided by the neuron-top utility.

\n

Available metrics

\n

Core Utilization, HBM Used, HBM Total, Tensor Memory, Constant Memory, Model Code, Shared Scratchpad, Nonshared Scratchpad, Runtime Memory, Driver Memory, DMA Rings, Collectives, Notifications

\n", "content_markdown": "`zml-smi` is a universal diagnostic and monitoring tool for GPUs, TPUs and NPUs. It provides real-time insights into the performance and health of your hardware.\n\n![](https://zml.ai/posts/zml-smi/1.png)\n\nIt is a mix between `nvidia-smi` and `nvtop`.\n\nIt transparently supports all the platforms ZML supports. That is NVIDIA, AMD, Google TPU and AWS Trainium devices. It will be extended to support more platforms in the future as ZML continues to expand its hardware support.\n\n## Getting started\n\nYou can download [zml-smi from the official mirror](https://mirror.zml.ai/zml-smi/zml-smi-v0.2.tar.zst).\n\n```\n$ curl -LO 'https://mirror.zml.ai/zml-smi/zml-smi-v0.2.tar.zst'\n$ tar -xf zml-smi-v0.2.tar.zst\n$ ./zml-smi/zml-smi\n```\n\n## Listing devices\n\n```\n$ zml-smi\n```\n\n## Monitoring devices\n\nThe `--top` flag provides real-time monitoring of device performance, including utilization, temperature, and memory usage.\n\n```\n$ zml-smi --top\n```\n\n![](https://zml.ai/posts/zml-smi/2.png) ![](https://zml.ai/posts/zml-smi/3.png)\n\n## Completely sandboxed\n\n`zml-smi` doesn’t require any software on the target machine besides the device driver and the GLIBC (mostly due to the fact that some shared objects from vendors are loaded).\n\n## Metrics\n\n## Host\n\n`zml-smi` displays host-level metrics such as CPU model and utilization, memory usage, and temperature.\n\n### Available metrics\n\nHostname, Kernel, CPU Model, CPU Core Count, Memory Used / Total, Uptime, Load Average (1m / 5m / 15m), Device Count\n\n## Processes\n\n`zml-smi` also provides insights into the processes utilizing the devices, including their resource usage and command lines. This is available for all platforms.\n\n### Available metrics\n\nPID, Device Index, Device Utilization, Device Memory, Process Command Line\n\n## NVIDIA\n\nMetrics are given through the [NVML](https://developer.nvidia.com/management-library-nvml) library, which ships with the driver. As such, it is expected to be on the system.\n\n### Available metrics\n\nGPU Utilization, Temperature, Power Draw, Encoder Utilization, Decoder Utilization, VRAM Used, VRAM Total, Memory Bus Width, Temperature, Fan Speed, Power Draw, Power Limit, Graphics Clock, SM Clock, Memory Clock, Max Graphics Clock, Max Memory Clock, PCIe Link Generation, PCIe Link Width, PCIe TX Throughput, PCIe RX Throughput\n\n## AMD\n\nMetrics are provided through the [AMD SMI](https://rocm.docs.amd.com/projects/amdsmi/en/latest/) library. `zml-smi` ships with it in its sandbox.\n\nIn order to support the latest AMD GPUs, `zml-smi` at build time downloads the `amdgpu.ids` file from both Mesa and ROCm (7.2.1 at the time of this article) and merges them together. This allows `zml-smi` to recognize and report on the latest AMD GPU models, even if they are not yet included in the official ROCm release. This is the case for Ryzen AI Max+ 395 (Strix Halo) for instance.\n\nSandboxing that file turned somewhat tricky. Because `libdrm-amdgpu` expects to find it in `/opt/amdgpu/share/libdrm/amdgpu.ids`, we had to get a bit creative. We didn’t want to install anything outside the binary sandbox. Nor did we want to patch that string inside `libdrm`.\n\nSo we created a shared object named `zmlxrocm.so` that is added to the `DT_NEEDED` section of `libdrm_amdgpu.so.1`. Then, `fopen64` is renamed to `zmlxrocm_fopen64`, which is then provided by `zmlxrocm.so`. Since we now sit between `libdrm` and `fopen64`, we can intercept the call to `fopen64`, compare the path against `/opt/amdgpu/share/libdrm/amdgpu.ids` and redirect it to the sandboxed copy of the file.\n\n### Available metrics\n\nGPU Utilization, Memory Usage, Temperature, Power Draw, VRAM Used, VRAM Total, Temperature, Fan Speed, Power Draw, Power Limit, Graphics Clock, SoC Clock, Memory Clock, Max Graphics Clock, Max Memory Clock, PCIe Bandwidth, PCIe Link Generation, PCIe Link Width\n\n## TPU\n\nMetrics are provided via the local gRPC endpoint exposed by the TPU runtime. Those are the same metrics exposed to the [tpu-info](https://pypi.org/project/tpu-info/) tool from Google.\n\n### Available metrics\n\nTensorCore Duty Cycle, HBM Used, HBM Total\n\n## AWS Trainium\n\nMetrics are provided through a private API found in `libnrt.so`, which `zml-smi` embeds in its sandbox. Those are the same metrics provided by the [neuron-top](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/tools/neuron-sys-tools/neuron-top-user-guide.html) utility.\n\n### Available metrics\n\nCore Utilization, HBM Used, HBM Total, Tensor Memory, Constant Memory, Model Code, Shared Scratchpad, Nonshared Scratchpad, Runtime Memory, Driver Memory, DMA Rings, Collectives, Notifications\n", "dateCreated": "2026-09-08T10:17:10.166315+00:00", "date_created": "2026-03-30T15:00:00+00:00", "date_modified": "2026-03-30T15:00:00+00:00", "date_published": "2026-03-30T15:00:00+00:00", "description": "zml-smi is a universal diagnostic and monitoring tool for GPUs, TPUs and NPUs. It provides real-time insights into the performance and health of your hardware. It is a mix between nvidia-smi and nvtop. It transparently supports all the…", "external_url": "https://zml.ai/posts/zml-smi/", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/zml-ai/2026/03/2026-03-30-introducing-zml-smi.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/zml-ai/2026/03/2026-03-30-introducing-zml-smi.md" }, "id": "urn:aggr:item:af51bfb7115c12559e8e5d412cb91ed69709c46f", "isBasedOn": { "@id": "https://zml.ai/posts/zml-smi/" }, "mainEntity": { "@id": "https://zml.ai/posts/zml-smi/", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/zml-ai/2026-03-30-introducing-zml-smi/#webpage" }, "dateModified": "2026-03-30T15:00:00+00:00", "datePublished": "2026-03-30T15:00:00+00:00", "headline": "Introducing zml-smi", "timeRequired": "PT3M", "url": "https://zml.ai/posts/zml-smi/", "wordCount": 639 }, "name": "Archived snapshot: Introducing zml-smi", "preview": null, "reading_minutes": 3, "replicated_at": null, "source": "zml-ai", "source_name": "ZML - Model to Metal", "summary": "zml-smi is a universal diagnostic and monitoring tool for GPUs, TPUs and NPUs. It provides real-time insights into the performance and health of your hardware. It is a mix between nvidia-smi and nvtop. It transparently supports all the platforms ZML supports. That is NVIDIA, AMD, Google TPU and AWS Trainium devices. It will be extended to support more platforms in the future as ZML continues to expand its hardware support.", "tags": [], "temporalCoverage": "2026-09-08T10:17:10.166315+00:00", "title": "Introducing zml-smi", "url": "https://aggr.aymericbeaumet.com/items/zml-ai/2026-03-30-introducing-zml-smi/", "word_count": 639 }