You can extend the time limit of a running job by using the command
nsc-boost-timelimit on the login node.
nsc-boost-timelimit --help will show the available options.
Due to technical limitations in the job scheduler, boosting the time limit of a job is only possible once it has started. If you want to be notified of your job having started, you can use the
--mail-type=BEGIN option to sbatch.
The time limit you specify with the
-t option is the new total time limit (e.g to extend a one hour job by one hour, use
nsc-boost-timelimit -t 2:0:0).
You can also use the
-a option to add to the existing timelimit, e.g
nsc-boost-timelimit -a 1.5 to add 1.5 hours to the existing timelimit.
Example: check the cost of extending the time limit of a job to 8 hours
[x_makro@tetralith1 ~]$ nsc-boost-timelimit -c -t 8:0:0 14399940 The cost for boosting the timelimit of job 14399940 is 1200.0 tokens Project nsc-guest has 29554.8 tokens available
Example: extend the time limit of a job to 8 hours
[x_makro@tetralith1 ~]$ nsc-boost-timelimit -t 8:0:0 14399940 Timelimit changed, cost: 1200.0 tokens. Project nsc-guest now has 28354.8 tokens available
Example: extend the time limit of a job to 5 days
[x_makro@tetralith1 ~]$ nsc-boost-timelimit -t 5-0:0:0 14399940 Timelimit changed, cost: 17920.0 tokens. Project nsc-guest now has 10434.8 tokens available
The base cost of extending the time limit of a job is currently 10 tokens per core hour of extension1 (e.g extending the time limit of a 16-core, one hour job to two hours costs 160 tokens).
The timelimit can only be extended as far into the future as the cluster's normal limit on job length (3 days on Sigma and 7 days on Tetralith). This is for your own protection. NSC typically provides a week's warning of upcoming maintenance etc, and any longer-running jobs might then be killed at the start of the maintenance period. If you want to accept the risk of your job being killed by NSC in such situations, you can use the
--accept-the-risks option and set any time limit you want (as long as you have enough tokens). Another option is to extend your job several times (e.g 7 days at a time).
Example: fail to extend the time limit of a job to 8 days, then using the
--accept-the-risks option to extend it anyway
[kronberg@tetralith1 ~]$ nsc-boost-timelimit -t 8-0:0:0 17462797 ==== FAIL ==== Your request could not be fulfilled: You may only extend a job 168 hours into the future (unless using the "--accept-the-risks" option)! [kronberg@tetralith1 ~]$ nsc-boost-timelimit --accept-the-risks -t 8-0:0:0 17462797 Timelimit changed, cost: 30560.0 tokens. Project nsc now has 969440.0 tokens available
You cannot use the tool to extend the time limit on a job if one or more of the compute nodes it uses are being taken offline ("draining") for urgent security updates, or if one or more of the nodes is reserved for some other purpose. If you absolutely need such a job extended, contact NSC Support and explain why that job is extra important to you.
The reasoning behind this cost is: extending the time limit of a running job will cause the same inconvenience to other users as submitting a new job and boosting its priority.↩