Skip to main content
You can use Slurm commands to view and manage jobs in your Soperator cluster. To run these commands, connect to the cluster’s login node.

​
How to view job list and details

​
Jobs in queue

To list all jobs that are currently in the queue, use the squeue command. You can use various parameters to specify the output format:
  • --long to include more details. For example:
    Output example:
  • --Format to customize output columns and their width. For example:
    Output example:
  • --steps to show job steps, that is, sets of tasks within a job. For example:
    Output example:

​
All jobs

The squeue command doesn’t list completed or failed jobs. To get the full details of all recently run jobs, use the scontrol command:
Output example:
By default, this command shows the jobs finished in the last 24 hours. The output is limited to 10,000 jobs. To find out how many jobs are displayed for your cluster, run the following command:
Output for default settings:
To list all jobs that were run on the cluster, use the sacct command.

​
Jobs and processes that run on specific nodes

To get the list of jobs running on particular nodes, run the following command:
Change the --nodelist parameter value to include the nodes that you need. Output example:
To get the job processes that are currently running on a given worker node, connect to this node and use scontrol listpids. Run the following command:
Output example:

​
Full details and batch script of a job

To get the full details of a job, run the following command:
Output example:
You can also retrieve the batch script used to run the job:
This command creates a slurm-<job_ID>.sh file with the contents of the script.

​
Job states

You can see the current job state in the STATE column when you list jobs with squeue or in the JobState parameter when you get job details with scontrol show job(s). Some of the common job states include: For a complete list of all possible job states, see the Slurm documentation.

​
How to manage jobs

The scontrol command lets you manage the jobs in the queue.

​
Suspend and resume a job

You can suspend a job, which means that the job processes are terminated, but resource allocations are retained. Run the following command:
The job is returned to the queue and waits in SUSPENDED status until you manually resume it:
The job is resumed and continues execution.

​
Requeue a job

You can requeue a job, which means that the job is terminated and returned to the queue. It restarts automatically when the resources are available. Run the following command to requeue a job:
A requeued job keeps the same job ID. If your job writes some data at paths that depend only on the job ID, the data from the previous attempt may be overwritten by the requeued job.
In Soperator clusters, some failed jobs are requeued by default. To check this setting for your cluster, run the following command:
Output for default settings:
You may want to prevent a requeued job from being scheduled again automatically. To requeue a running job and put it on hold until you explicitly allow it to be scheduled, run the following command:
To requeue and put on hold a job that hasn’t started yet, run the following command:
To allow the job to be scheduled again as soon as there are available resources, run the following command:

​
Cancel a job

You can cancel job execution. Run the following command:
The job is terminated and all resources are freed.