You can use Slurm commands to view and manage jobs in your Soperator cluster. To run these commands, connect to the cluster’s login node.
How to view job list and details
Jobs in queue
To list all jobs that are currently in the queue, use the squeue command. You can use various parameters to specify the output format:
-
--long to include more details. For example:
Output example:
-
--Format to customize output columns and their width. For example:
Output example:
-
--steps to show job steps, that is, sets of tasks within a job. For example:
Output example:
All jobs
The squeue command doesn’t list completed or failed jobs. To get the full details of all recently run jobs, use the scontrol command:
Output example:
By default, this command shows the jobs finished in the last 24 hours. The output is limited to 10,000 jobs. To find out how many jobs are displayed for your cluster, run the following command:
Output for default settings:
To list all jobs that were run on the cluster, use the sacct command.
Jobs and processes that run on specific nodes
To get the list of jobs running on particular nodes, run the following command:
Change the --nodelist parameter value to include the nodes that you need.
Output example:
To get the job processes that are currently running on a given worker node, connect to this node and use scontrol listpids. Run the following command:
Output example:
Full details and batch script of a job
To get the full details of a job, run the following command:
Output example:
You can also retrieve the batch script used to run the job:
This command creates a slurm-<job_ID>.sh file with the contents of the script.
Job states
You can see the current job state in the STATE column when you list jobs with squeue or in the JobState parameter when you get job details with scontrol show job(s). Some of the common job states include:
For a complete list of all possible job states, see the Slurm documentation.
How to manage jobs
The scontrol command lets you manage the jobs in the queue.
Suspend and resume a job
You can suspend a job, which means that the job processes are terminated, but resource allocations are retained. Run the following command:
The job is returned to the queue and waits in SUSPENDED status until you manually resume it:
The job is resumed and continues execution.
Requeue a job
You can requeue a job, which means that the job is terminated and returned to the queue. It restarts automatically when the resources are available. Run the following command to requeue a job:
A requeued job keeps the same job ID. If your job writes some data at paths that depend only on the job ID, the data from the previous attempt may be overwritten by the requeued job.
Cancel a job
You can cancel job execution. Run the following command:
The job is terminated and all resources are freed. 