> ## Documentation Index
> Fetch the complete documentation index at: https://docs.nebius.com/llms.txt
> Use this file to discover all available pages before exploring further.

# NCCL Inspector in Soperator

The [NCCL Inspector](https://github.com/NVIDIA/nccl/blob/master/plugins/profiler/inspector/README.md) by NVIDIA® is a plug-in for the [NVIDIA Collective Communications Library](https://github.com/NVIDIA/nccl) (NCCL) that provides detailed, per-communicator, per-collective performance and metadata logging for distributed GPU training jobs. It helps you see how NCCL communication behaves inside a workload, without changing the training code.

If your Soperator cluster is prepared with NCCL Inspector support, you can enable it for specific jobs. When enabled for a job, the NCCL Inspector collects performance data from NCCL operations and sends the resulting metrics to the Grafana® installation that is part of your Soperator cluster. This gives you a job-level view of communication bandwidth, latency, message sizes and collective types.

<Note>
  The NCCL Inspector is available on demand for Soperator clusters deployed in Nebius AI Cloud. For details, [contact support](https://console.nebius.com/support/create-ticket) or your personal manager.
</Note>

## Enabling NCCL Inspector for jobs

On a Soperator cluster prepared with NCCL Inspector support, enable it in your Slurm job, for example, a [batch job](/slurm-soperator/jobs/index), by setting the `NCCL_INSPECTOR_ENABLE` environment variable:

```bash theme={null}
export NCCL_INSPECTOR_ENABLE=1
```

After that, run your workload as usual with `sbatch` or `srun`. You don't need to change the training application code.

<Note>
  To collect data for NCCL point-to-point (P2P) operations, the NCCL Inspector requires NCCL 2.30.3 or higher.
</Note>

To disable the NCCL Inspector for a `srun` call, add the `--snccliprecon-enabled=0` parameter to the `srun` command.

## Accessing Grafana dashboards

The following NCCL Inspector dashboards are available in Grafana:

* **NCCL Inspector Job Performance**: primary per-job view.
* **NCCL Inspector Metrics**: metric-level overview.
* **NCCL Inspector Raw Metrics**: raw metrics view.

For instructions about viewing dashboards in Grafana, see [Monitoring metrics of Soperator clusters](/slurm-soperator/monitoring/metrics).

## See also

* [Monitoring metrics of Soperator clusters](/slurm-soperator/monitoring/metrics)
* [Enhancing Communication Observability of AI Workloads with NCCL Inspector](https://developer.nvidia.com/blog/enhancing-communication-observability-of-ai-workloads-with-nccl-inspector/) in the NVIDIA Technical Blog
* [NCCL Inspector](https://github.com/NVIDIA/nccl/blob/master/plugins/profiler/inspector/README.md) in NCCL repository on GitHub

***

*The Grafana Labs Marks are trademarks of Grafana Labs, and are used with Grafana Labs' permission. We are not affiliated with, endorsed or sponsored by Grafana Labs or its affiliates.*