Job Policies
The primary objective of the MSCF is to provide teraflop computing resources for grand-challenge computational problems. The job scheduling policy has been established to provide a higher priority on effective throughput and turnaround of large jobs. To maximize system flexibility, all jobs are submitted to a single queue. The job scheduler controls the allocation of compute processors to the users job and will place the job in one of the four available queues, short, normal, large, and idle. This allocation is governed by a number of policy constraints. All policy constraint values must be satisfied. For more information on MSCF policies, please see User Policies.
MSC News
- Call for White Papers on Scientific Computing
- EMSL introduces a new video about Chinook on EMSL's YouTube channel (script, .pdf, 9kb)
- Chinook - Living Life in the Fastlane [.pdf, 107kb]
Additional Information
- Meet the MSC Staff
- Related EMSL User Projects
- Completed Projects: Final Reports
- Help
- User Policies
- User Tips
- Accounts
- Sampling of Jobs Running
- Emergencies
- Glossary
- Training and Events
- Next-Generation Supercomputing (Greenbook) [.pdf, 2.4mb]
- FAQ
- Partners and Related Links
- Spokane Cluster for Biogeochemistry Research
Job Policy Constraints:
- Maximum number of running jobs: 3
- Maximum number of queued jobs: 8
- Maximum number of processors per job: 1800
- Minimum number of processors per job: 8
There is also a set of default values that limit the time a single job with a particular number of processors can have, which are shown below.
| Number of Processors in a Single Job | Time Limit | Notes |
|---|---|---|
| 512 - 1800 | 48 wall clock hours | These jobs will be placed ahead of the jobs in the queues below, i.e. they will receive highest priority. |
| 256-512 | 48 wall clock hours | These jobs will be placed ahead of the jobs in the queues below, i.e. they will receive higher priority. |
| 33 - 255 | 48 wall clock hours | Normal priority jobs. Note that many of these jobs will backfill with the large jobs in the larger queues. |
| 8 - 32 | 36 wall clock hours | Normal priority jobs. Note that many of these jobs will backfill with the large jobs in the larger queues. |
| 1-8 | 30 minutes | Test / Interactive queue, the 32 processors in this queue are reserved on the ThinNodes only. |
Idle queue:
The idle queue provides the opportunity for projects that have run out of their regular allocation to use processors that are idle on the machine. The primary purpose of this queue is to increase machine usage and help projects that have run out of their original allocation to get some computations done. The only limit on the "idle queue" is that jobs must be run for 90 minutes or less. Time used in the "idle queue" needs to be tracked in the GOLD accounting system and is designated as a "Charge Limit" for the project. Projects that qualify for the "idle queue" will have time assigned in the "CreditLimit" column of the "gbalance -h -u <UserID>" command. If your job will need to create a restart file, be sure it gets written before the 90 minute window terminates. To submit a job to the Idle queue, include the command: #BSUB -q "idle" and to see jobs in the "idle queue" you will need to use the -x flag for showq. For example: "showq -x | grep <UserID>" will find all of your jobs in all queues on MPP2. Send an e-mail to mscf-consulting@emsl.pnl.gov for more information or to answer any questions.
SIGHTS special purpose queue:
In addition to the queue limitations mentioned above the users can request access to a special purpose queue called Scientific Impact Generated by High Teraflop Simulations (SIGHTS). The SIGHTS queue is for compute jobs that require resources beyond the normal queue limits for MPP2, and serve uniquely impactful cutting-edge PNNL/EMSL mission science opportunities which cannot be performed at any other computing facility. SIGHTS jobs should require the use of 1024 processors or more, up to the capacity of MPP2. SIGHTS jobs are not automatically set in the MPP2 queue. SIGHTS jobs can be submitted anytime after approval and will be tended by an MSCF scientific consultant and operations personnel to assist in successful job completion . All requests for a week with a monthly outage will need to be submitted by 12 noon on Thursday before the scheduled outage.
Access to the SIGHTS is by request only and is subject to time availability. All requests are submitted to the MSCF consulting group for review. Please pick keyword "SIGHTS". In your request please provide a short (one-two page) description of what you plan to do and how you plan on doing it. Upon receipt of the request a consultant will be assigned to the job. The consultant will work with the users to be sure the job is ready. The consultants and operations staff will watch all SIGHTS jobs to be sure they are running correctly. Details about SIGHTS jobs are:
- Minimum number of processors is 1024.
- No time limit.
- Job can be started any time or after a scheduled system outage.
- Allocation costs for these jobs is none. All allocations for the job will be returned to the user.
Short pool:
The short pool of 16 reserved ThinNode processors allows users to run small and short jobs to test or debug their codes. Interactive or test jobs will be limited to a maximum of 8 processors and a 30 minutes time limit per job. Note: the reserved processors in this pool are ThinNodes. Hence, if you request FatNodes in your jobs the job you will have to wait for FatNode processors to become available.
These constraints are used as system default values. If you require resources beyond these limits (more processor, longer run times), please have your Principle Investigator contact MSCF Computer Projects Manager and the appropriate user account can be configured with exceptions to over-ride the default values.
