NSF Resource Allocations Policies

Home > User Support > Access > NSF Resource Allocations Policies

(Updated: November 2008)

1. Introduction

This document states the policies and instructions for preparing requests to use cyberinfrastructure resources on the TeraGrid. All interested persons are encouraged to review this guide prior to making submission; any questions regarding these policies should be directed to the TeraGrid Help Desk. Answers to Frequently Asked Questions (FAQs) are available at in the TeraGrid Knowledgebase.

TeraGrid resources are allocated as part of projects, where a project has a single principal investigator (PI) and may include allocations on one or more resources. There are three types of projects. Requests for "Research" projects are reviewed quarterly by the TeraGrid Resource Allocation Committee (TRAC), which consists of computational scientists who review these submissions and make allocation srecommendations for all TeraGrid resources.

Requests for "Startup" projects are accepted and reviewed continually, as are requests for "Education" projects, which support classroom instruction and training classes. Startup and Education projects are designed to be easy to request and allocate, and TeraGrid provides greater detail on special situations related to Startup and Education requests in Requesting a Startup or Education Allocation: Step-by-Step for First-Time Users.

Support for the acquisition of and/or operation of TeraGrid resources is provided by NSF under grant number OCI-0503697 to the Grid Infrastructure Group and grants to the TeraGrid Resource Providers. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NSF.

1.1. Summary of Major Changes

Language throughout this document has been revised and clarified wherever possible, and the focus of changes has been to streamline the allocations process for researchers across the board. As a result, this policy update reflects several significant changes from prior policies and request preparation instructions. These major changes are summarized here; all PIs should review the relevant sections before their next submission.

1. New limits and nomenclature have been set for Startup allocations. The "DAC" terminology has been replaced, and there is a new category for Education allocations. See Section 5.
2. Research allocations have replaced MRAC- and LRAC-level allocations. Requests of any size can now be submitted to each quarterly meeting of the TRAC. See Section 6.
3. The required and optional documents for Research (TRAC) submissions have changed. Failure to comply with the new and simplified page limits will result in submissions being returned without review. See Section 6.3.
4. Review criteria for scientific merit and the resource justification have been clarified. See Section 7.
5. Advances can now be requested for up to 25% of a pending resource request, subject to resource availability and staff approval. See Section 3.5.4.
6. Requests for Advanced User Support should now be justified in the Main Document of the request, not within a separate appendix. See Section 6.3.2.
7. Supplemental requests will now generally be reviewed and processed within two weeks. See Section 3.5.3.

In an effort to ensure that current and potential users benefit as soon as possible, these policy changes are being rolled out while related changes to the TeraGrid's resource request system (POPS) are being implemented. Users should watch for changes to POPS and pay close attention to instructions within the POPS data entry forms.

2. Appropriate Uses for TeraGrid Allocations

TeraGrid provides access to resources in high-performance computing, scientific visualization, data storage, and advanced user support, to enhance scientific discovery. These policies and procedures are followed to ensure a fair and efficient allocation of these resources.

The TeraGrid encourages and supports projects in all disciplines for a wide range of purposes. In general, TeraGrid allocations policies provide for projects that (1) permit researchers or research groups to evaluate TeraGrid capabilities, port codes, and otherwise get started in the TeraGrid environment; (2) support a range of educational objectives, from classroom instruction to training classes; and (3) allow individuals or groups to pursue their science and engineering research objectives.

2.1. Categories of Research Activities

Within this broad scope, TeraGrid projects support various categories of research consortiums and services. Most often, TeraGrid projects can be classified in one of the following categories:

1. Single Principal Investigator (PI): Most projects support a single PI and possibly a small research group working closely and co-authoring papers.
2. Multi-PI, Large Research Collaborations: Projects of this type are characterized by a single PI representing a large group of collaborating co-PIs who are working on subprojects within the overall collaboration. A single request is submitted, and a single project is allocated. The management of the allocated resources is left to the discretion of the principals on the request.
3. Large-scale Consortiums: Projects of this type are intended to support large-scale, funded projects that work together as a consortium. Often in these cases, a mechanism already exists for allocating community or project resources (e.g., an instrument such as a telescope or detector), and that mechanism will also be used to make allocations from the time granted to the community project to the individual investigators. Requests for this type of project typically describe the internal processes for managing access of individual investigators within the consortium.
4. Gateways or Community Services: Projects of this type provide services to a large community of users who are typically not directly collaborating with the project PI. An example of such a project would be an application portal service providing access to software and computer time to a community of biology researchers via a web-based interface. Requests to provide such a service must describe the details of the services provided, the methods used, the expected consumption of resources, and mechanisms for monitoring the users and usage of the service. Statistics of community usage should be reported quarterly and in renewal requests for resources, progress reports, and end-of-project reports. TeraGrid provides How to Write a Winning Gateway Proposal to assist in writing these requests.

2.2. PI Eligibility

The principal investigator (PI) of an allocated project is the person responsible for the accuracy of the resource request and the management of the ensuing allocation. This person must be a researcher or educator at a U.S. academic or non-profit research institution. A PI may not be a high school or undergraduate student; a qualified advisor, for example, a high school teacher or faculty member, must serve in this capacity. In most cases, a graduate student may not be a PI; however, see Section 8.1 for an exception for NSF Graduate Research Fellows and Honorable Mention awardees. A post-doctoral researcher is eligible to be a PI.

A U.S.-based scientist, engineer, or educator who has a joint appointment with a university or non-profit research institution may submit a request using that affiliation. The appointment may be adjunct, instructional, or any other official position.

Research staff from federal and state agencies or federally funded research and development centers (FFRDCs) are also welcome to apply for a TeraGrid allocation if their agency or center does not typically provide research staff with access to non-TeraGrid HPC resources of adequate scope for the planned research. The lack of access to CI resources should be documented in the request, as described in Section 7.3. As above, work not funded by a peer-reviewed grant will be subject to scientific review.

Section 8 lists further categories of individuals who are eligible to request TeraGrid allocations. The most common class of ineligible PIs are researchers based outside of the U.S. While TeraGrid supports collaborations between U.S.-based PIs and their foreign colleagues, PIs on TeraGrid requests and allocations must be based at a U.S. institution.

Allocated projects need not be supported by NSF grants; PIs may have support from any funding agency or funding source. Furthermore, while most requests do have peer-reviewed funding awards, a PI need not have such a grant in order to submit resource requests; however, the scientific merit of requests for use in research not supported by peer-reviewed funding will be evaluated by TeraGrid reviewers. See Section 7.1.

3. General Allocation Policies

To support the wide range of research objectives and project uses described above, the TeraGrid allocations process defines three types of projects.

1. Startup. Startup projects target new users planning to submit more substantial requests in the future as well as users who have modest computational requirements that, nevertheless, cannot be satisfied by their campus resources.
2. Education. Education projects target faculty or teachers intending to use TeraGrid resources for classroom instruction or training classes related to the use of advanced cyberinfrastructure technologies.
3. Research. Research projects are designated for projects that have progressed beyond the startup phase and are conducting "production" usage of TeraGrid resources in pursuit of their research objectives.

Details specific to Startup and Education projects, including allocation size restrictions, are described in Section 5. Details specific to Research requests and projects are covered in Section 6. However, projects of all three types are subject to a common, basic set of policies and procedures, which are summarized in this section.

3.1. TeraGrid Resources

TeraGrid makes available a set of cyberinfrastructure resources for allocation requests and awards. In general, these resources can be categorized into computational, storage, specialized, and grid resources. For a full listing of resources and architecture details, please refer to the TeraGrid Resource Catalog.

Computational resources are typically HPC systems oriented toward batch job submission. TeraGrid includes systems of various architectures and from many vendors. Note that computational resources typically have associated disk systems for short-term data storage by users.

Storage resources include both disk-based systems for medium- and long-term storage of data and tape-based systems for long-term storage. Some storage resources available to users are not part of the allocations process.

Specialized resources include dedicated visualization systems, systems for virtual hosting, and high-throughput Condor pools.

Advanced user support resources include staff experts from the TeraGrid partner sites whose time is devoted to working with single allocated projects to accelerate the progress of each project toward its research objectives. Advanced user support may include working on code performance, adapting codes to TeraGrid capabilities, improving gateway capabilities, and so on.

Grid resources are sets of (generally) computational resources that are allocated as a single unit and for which allocations can be spent on any resource in the set, at the user's discretion at any time during an allocation award. TeraGrid Roaming (see Section 4.1.1) includes many computational resources. Smaller grids include the "TeraGrid Clusters" (IA-64 "DTF" clusters at NCSA, SDSC and ANL) or the Abe/Queen Bee grid (Xeon clusters at NCSA and LONI/LSU).

3.2. Submission of Allocation Requests

Allocation requests for TeraGrid resources are only accepted electronically and must be made through the POPS system, which is also accessible through the TeraGrid User Portal. Requests include annual submissions to initiate or continue allocations as well as manage allocations during an allocation period.

The POPS system recognize seven types of requests, which in general can apply to all three types of projects:

1. New: A submission is considered "new" if it is (a) a PI's first submission to the TeraGrid, (b) a request in a completely different area of science than a PI's previously submitted requests, or (c) a PI's first request for a particular project type. For example, a PI with a prior or current Startup allocation but now submitting a Research request is considered a new Research request. NOTE: PIs continuing prior MRAC or LRAC projects should submit renewal requests to the TRAC.
2. Renewal: A submission should be a "renewal" if a PI (a) has had an allocation of the same request type, which was active within the past two years, (b) is continuing the same or similar line of research, and (c) is in the same field of science. A renewal request must address the progress of the prior allocation's research.
3. Progress Report: This POPS request applies only to PIs who have received a multi-year Research allocation (resulting from a full "New" or "Renewal" request submission). Such PIs must submit a Progress Report that details successful use of the prior year's allocation in order to receive the subsequent year's allocation. The submission should also indicate the resource request for the forthcoming year. Detailed justification is not required, but significant changes to the original request should be explained. This type of request is distinct from the mandatory annual progress report that forms part of all Renewal requests. This request does not apply to Startup or Education projects.
4. Supplement: A supplemental request asks for additional resources during an existing allocation's one-year time frame for projects proceeding more rapidly than anticipated or requiring more resources than anticipated (see Section 3.5.3).
5. Transfer: Transfers (Section 3.5.5) of allocated units between TeraGrid resources are requested through POPS. No formal documentation is required.
6. Extension: Extensions (Section 3.5.6) of allocation periods beyond the normal 12-month duration are requested through POPS. A brief reason is required but no formal documentation is needed.
7. Justification (Appeal): If the TRAC has requested that the PI give more details or justification before the project can be allocated any time or any additional time, the PI should submit a justification (appeal). This type of submission should specifically address information that the reviewers have requested. (See Section 6.4.4.) This request type does not apply to Startup or Education requests.

3.3. One Project Per PI

An individual should generally be PI on only one active TeraGrid resource request/project at a given time. Several distinct research activities can be combined in a single resource request, however. The resource request for each activity must be justified, and any allocation-size limits apply to the aggregate request.

The single-project rule is designed to minimize the effort required by PIs for submitting resource requests and the overhead to the TeraGrid process for reviewing those requests. While PIs may have several different funded grants that require computational support, these should be included as subprojects within a single request.

There are several exceptions to this rule:

• A PI may submit a Research request at any time during an active Startup allocation. A successful Research allocation will supersede any Startup allocations and start a new 12-month allocation period.
• A PI with an active Research project may submit a Startup request for access to a specific resource or set of resources that he or she has not used before.
• If a PI has research activities that are in significantly different fields of science and, therefore, cannot be reviewed by the same set of reviewers, multiple requests/projects will be considered.
• If a PI is leading an individual research activity as well as a community consortium activity or community service activity, separate requests for these distinct activities will be considered.
• A PI with active Startup or Research projects may submit requests for any number of Education projects to keep classroom and training users separate from research activities.

Similarly, to minimize the effort required to gain access to TeraGrid resources, closely collaborating researchers should submit a single collaborative request rather than several individual requests. For example, a PI and associated post-doctoral researchers; investigators supported by the same funding grant; and researchers in the same lab group should consider submitting a request describing and justifying the various subactivities. One of the collaborators is designated as the PI, and others can be designated as co-PIs. For further guidance on whether multiple requests or a single, collaborative request is appropriate in a specific case, please contact the TeraGrid Help Desk .

3.4. Allocation Duration

All allocations for TeraGrid resources are made for a 12-month period. PIs can continue their activities in subsequent years through annual renewal requests (Section 3.5.1) or, in the case of Research projects, through a multi-year allocation.

3.4.1. Multi-Year Allocations

The TeraGrid provides PIs who have long-term research activities with the option of submitting requests for multi-year Research allocations. These requests directly support a scientific grant with multi-year funding; allocations may be requested for the duration of that support. The submission must indicate the supporting grants for the computational activities. Multi-year requests are held to the same review criteria as single-year allocations.

IMPORTANT: While a commitment is made to provide resources for the approved time period, actual allocations are made for 12-month periods. Each year, a Progress Report and streamlined subsequent year request are required. Continued allocations depend on the TRAC determination of sufficient progress and justification.

Before requesting a multi-year Research allocation, a PI must demonstrate that all prerequisite and preliminary activity (code porting or code development, for example) has been completed and that the project is ready to conduct its research activities. Furthermore, it is highly recommended that all new PIs submit single-year requests until they are familiar with the process and review criteria, have received favorable reviewer feedback, and can demonstrate solid progress.

3.4.2. Forfeiture of Unused Allocation Units

At the end of each 12-month allocation period, single- and multi-year projects forfeit any unused compute SUs. For storage allocations that have used less than their allocated space, a PI must justify continued need for the extra space or risk a reduced allocation amount in subsequent years. However, see Section 3.5.6 below.

3.5. Allocation Management

Once a project has received its allocation, the TeraGrid gives the PI and any co-PIs a number of options to ensure the flexibility needed to complete their research objectives.

3.5.1. Project Continuation

The most important consideration for PIs is whether they wish to continue computing after the expiration of their current allocation period.

In general, Startup allocations are not eligible for continuation or renewal; PIs are expected to submit Research requests to a TRAC meeting during their Startup allocation period to continue pursuing their research objectives. Certain exceptions apply; see Section 5.2.

For single-year Research awards, the PI should submit a Renewal request to the TRAC for consideration. In most cases, they should submit this request approximately one year after their initial request submission, so that it can be reviewed and awarded to avoid any interruption. In general, the appropriate submission period is the one approximately three months prior to the expiration of their allocation.

For multi-year Research allocations, the PI should submit a Progress Report approximately one year after their initial request submission, so that it can be reviewed and allocated to avoid any interruption. In other words, the timing of a Progress Report submission for a multi-year allocation is the same as the timing for submitting a renewal request for a single-year allocation.

3.5.2. Additional Users

A PI may share his/her allocation by establishing accounts under the allocation with any number of collaborators, including graduate or undergraduate students. Research collaborators may include colleagues based outside of the U.S. Additional users can be added through the TeraGrid User Portal (under the My TeraGrid tab).

PIs on Education allocations can contact the TeraGrid Help Desk for assistance in establishing accounts for large groups of students under their allocated project.

3.5.3. Supplemental Requests

To request additional resources during an allocation award period, a PI may submit a Supplemental request for resources using POPS. This type of request should explain what has been accomplished with the allocations that have already been made, state exactly how many units are needed, and justify how the PI intends to use the additional resources. These requests are accepted at any time and are generally considered within two weeks of submission. Supplemental allocation amounts are added to a project's current allocations and expire at the end of the current allocation period; unused SUs are forfeited. In most cases, a project cannot receive a Supplemental allocation and an Extension during the same allocation period.

In practice, to ensure a group can continue computing without an interruption, a PI should not wait until an allocation is completely spent before submitting a Supplemental request. Please allow at least two weeks for review and processing.

Supplemental allocations to Startup and Education projects are constrained by the allocation-size restrictions on these projects (see Section 5).

PIs should submit Supplemental requests via the POPS system.

3.5.4. Advances

Upon submitting a new or renewal Research request, PIs are allowed to ask for an immediate advance of up to 25% of the resources requested in anticipation of favorable review. The size of the advance will depend on current availability on the target resource and staff review. Any usage of this advance will be debited against the eventual allocation. Requests for advances should be sent to the TeraGrid Help Desk (POPS does not currently handle advances).

3.5.5. Transfers

PIs can request that allocated time be transferred from one platform to another, subject to resource availability on the target resource. On Research allocations, such transfers are subject to conversion of SUs according to established weighting factors (see Section 4.2.2). On Startup allocations, transfers are subject to the size restrictions for the target resource (see Section 5).

PIs should submit transfer requests via the POPS system.

3.5.6. Extensions

Projects that encounter problems in consuming their allocations, such as unexpected staffing changes, can request an extension to their allocation end date. In such instances, PIs may request a single extension of an allocation, extending the expiration date by a maximum of six months The PI will be asked to specify the project number and the length of the extension (1-6 months), along with a brief reason for the extension.

Note that Extensions apply to all current resource allocations on the same project; PIs cannot extend some allocations and submit overlapping Renewal requests for other resources.

If a project still has unused allocation units at the end of the extended allocation period, the PI must submit a Renewal request to continue their TeraGrid project; the request should explain the reasons for the unused allocation.

Extension requests should be submitted via the POPS system.

3.6. Acknowledging Support

An acknowledgement of support from the appropriate resource provider and the TeraGrid should appear in any publication of material, whether copyrighted or not, that describes work which benefited from access to TeraGrid cyberinfrastructure resources. For suggested language, see Acknowledging TeraGrid Support.

PIs should include a bibliography of articles or other manuscripts —published, accepted, submitted or in preparation— that benefited from support by TeraGrid resources as part of their annual Progress Reports and Final Reports (see Section 6.3.2).

4. Selecting Resources and Calculating Allocation Sizes

PIs who need the special capabilities of particular TeraGrid resources can specify those resources in their requests. This section contains information related to selecting resources and calculating allocation amounts for all requests.

4.1. Recommended Use Guidelines

For each resource available for allocation, a statement of recommended use is available in the TeraGrid Resource Catalog. These guidelines describe Resource Provider policies or usage models that may be emphasized or, conversely, may not be permitted on certain resources. PIs are advised to review these guidelines before submitting requests for specific resources.

4.1.1. TeraGrid Roaming Access

In addition to specific resources, a large subset of TeraGrid resources can also be requested as a collective resource. An allocation for "TeraGrid Roaming" can be used across a large group of TeraGrid compute resources: those that are available for Roaming Access. The resources available for roaming access are indicated in the TeraGrid Resource Catalog. A more detailed description is available in TeraGrid Roaming Allocations.

In calculating roaming requests, 1 SU for TeraGrid Roaming is equivalent to 1 SU on the IA64 Clusters at NCSA (Mercury), SDSC, and ANL. Usage on other resources is converted using the standard conversion rates (see Section 4.2.2).

In requesting TeraGrid Roaming, in addition to the standard resource justification, PIs should also describe how the research plan would take advantage of roaming access capabilities. For example, a code may run on many TeraGrid compute resources, and the group may plan to submit runs to the shortest queue at any given time. The group may also plan to take advantage of grid capabilities for runs that span several resources.

If a request for TeraGrid Roaming does not demonstrate a need or plans to take advantage of these roaming capabilities, the TRAC may recommend the request be allocated on a specific resource.

4.2.Calculating Resource Requests

The requested allocation amounts for TeraGrid resources need to be detailed and justified in the supporting documents. Allocation amounts for compute, storage, and/or advanced support resources should be clearly linked to the scientific goals and the proposed research plan.

• For compute resources, calculations should be based on small-scale test runs on the resources of interest and scaling tests, as well as the number and type of runs to be made.
• For storage resources, calculations should describe the data to be stored and the amount of storage required for the data (record/file size, number of files/records, and/or other relevant characteristics).
• For advanced support, requests should describe how the amount of support time (months) was estimated; the submission should also describe the corresponding effort to be contributed by the requesting research team and the expected improvements to be achieved by this support.

4.2.1. Definitions for Allocation Units

Different resource types define different units for making resource requests. The TeraGrid Resource Catalog shows the allocation request unit for each resource. Throughout the remainder of this document, the following resource types and allocation units are used:

Compute resources (including most visualization resources) are requested in terms of "service units" (SUs). In general, 1 SU equals 1 processor core-hour, or one wallclock-hour on one processor core, on a given resource. Local resource policies may define more complex formulas for SUs that include such factors as queue priority and so on. A submission that requests allocations on several resources should specify each of those requests in terms of local SUs.

Storage resources are requested in terms of terabytes (TBs). Some smaller-scale, specialized resources may require that requests be made in gigabytes (GBs).

Advanced Support is requested in terms of FTE-Months, generally an estimate of the expected duration and level of TeraGrid staff support needed. These estimates may later be revised upon interaction with the relevant TeraGrid staff.

Specialized resources may define specialized request units. Refer to the TeraGrid Resources Catalog for the units used in requesting allocations on these resources.

4.2.2. Converting Units Between Compute Resources

When allocated units are transferred from one compute resource to another, a conversion is applied to normalize the relative computational power of the two resources. The most common situation is a transfer of SUs from one compute resource to another. In this case, the standard conversion is based on a formula derived from the resource's performance on the HPL benchmark. Users can review the standard conversions using the SU Conversion Calculator.

Alternate conversion rates. If a PI has performance results of his/her group's own code on the source and target resources in a transfer request, the PI can request a conversion factor based on that code's performance be applied. The alternate conversion rate should be documented and noted in the transfer request.

At this time, it is not possible to convert allocable units on one resource type to units on another resource type. For example, storage allocations cannot be converted to compute allocations.

5. Startup and Education Requests

To enable researchers to quickly gain access to TeraGrid resources, TeraGrid provides two small-scale project types, both of which require only a project abstract.

• Startup projects target researchers getting started on TeraGrid, developing code, testing applications, and other similar purposes. PIs making Startup requests can select from a full list of TeraGrid resources, with TeraGrid Roaming as one of the possible resources.
• Education projects are specifically designated for class instruction or training activities.

Education requests present requesters with those TeraGrid resources that Resource Providers have made available for classroom or training activities.

Given the increased size of the new TeraGrid HPC systems, the default request limits for Startup and Education allocations has been increased for compute resources to 200,000 SUs. TeraGrid Roaming allocations are limited to 50,000 SUs, since most projects use only a small number of resources. The Startup and Education limit for disk allocations is 5 TB and for tape allocations, 25 TB.

RPs, however, may limit the size of a Startup allocation for a specific resource where 200,000 SUs (or the default storage limits) would represent a prohibitively large fraction of the resource. The Startup allocation limit will be posted in the TeraGrid Resource Catalog for each resource. In addition, transfers to a single resource on a Startup allocation cannot exceed that resource's Startup limit. The TeraGrid reviewers and allocations staff will enforce these limits.

Startup and Education allocations are for a period of one year beginning at the soonest practical date following a review of the request. They end approximately 12 months later, on the last day of the twelfth full month.

5.1. Submission

Startup and Education requests can be submitted at any time via the POPS system. The requests are reviewed and processed continually. An individual request is typically reviewed and processed within two weeks of submission.

Within POPS, Startup and Education requests primarily require basic information about the PI, any co-PIs, and funding grants supporting the work being conducted. The POPS forms also request an abstract describing the work to be conducted.

An abstract with a general description of the work is typically sufficient, and often an existing abstract can be used. Additional information that is welcome includes explanations for why the project needs access to TeraGrid resources, how the estimate of resources request was reached, and how the resources requested were selected.

Curricula vitae (CVs) for the PI and any co-PIs are required documents and must be uploaded to POPS. CVs should be in standard NSF or NIH formats (two page maximum); the CVs for the PI and all co-PIs should be assembled into a single document for uploading.

5.2. Renewal

In general, Startup allocations are presumed to be for one year only. However, renewals will be permitted with appropriate justification and subject to TeraGrid reviewer approval; researchers may be directed to submit Research project requests to the TRAC. Valid rationales for renewing Startup projects may include progress on Startup work that has proceeded more slowly than anticipated, or lack of access to campus resources to support the research activities being proposed even at the scale of TeraGrid Startup allocations. This may include the need to access specialized resources or larger-scale resources not typically available on campuses.

Education projects can be renewed indefinitely as long as the education activity —for example, a class repeated each academic year— is continued.

In submitting a renewal for a Startup or Education project, a PI is required to include a Progress Report describing how the prior year's allocation was used and a listing of any publications (published, accepted, submitted or in preparation) that resulted from the prior year's allocation.

5.3. Review and Processing

Startup and Education requests are accepted at anytime and reviewed by TeraGrid staff. Please allow two weeks from submission for review and project creation.

The review process for Startup and Education requests is as follows: Each request will be reviewed by at least one TeraGrid staff reviewer, and if the first review rejects the request, a second review will automatically be conducted. For Startup and Education requests, each RP will define a primary reviewer, and the Allocations Coordinator or designee will assign the reviewer from each RP whose resource has been selected by the PI. The EOT Area Director will appoint reviewers for Education requests; TeraGrid Advanced User Support Area Director will appoint reviewers for TeraGrid Roaming requests.

PIs for Startup and Education requests are notified by email when reviews are completed. A subsequent postal mail confirmation with user information for new accounts will follow soon thereafter.

6. Submitting Research Requests

For projects that have progressed beyond the Startup phase, a Research request is appropriate. Research requests include a well-documented resource-use plan that describes how the requested allocations are necessary and sufficient to accomplish the project's research objectives. A single Research request and resulting project may combine compute, storage and Advanced User Support Program requests.

Research requests of all sizes are submitted for consideration to the TeraGrid Resource Allocation Committee (TRAC). There is no limit to the size of allocations that can be provided to Research requests; however, the Allocation Coordinator may determine that some smaller-scale Research requests can be processed via staff-only review.

6.1. Deadlines for Submissions

The TRAC submission deadlines are January 15, April 15, July 15, and October 15. These deadlines are approximately six weeks prior to the meeting at which they will be reviewed, and 10 weeks prior to the allocation start date. The corresponding allocations begin April 1, July 1, October 1, and January 1, respectively, and the corresponding allocation end dates are therefore March 31, June 30, September 30, and December 31.

In general, once a PI enters the allocations process, the PI should plan to submit either a Renewal request or Progress Report at the same deadline each year to avoid interruption in the project's allocations, assuming they consume their allocation within the 12-month period. (See, however, Section 3.5.6 on extending projects that are proceeding more slowly than anticipated.) All potential and continuing Research projects are encouraged to keep these deadlines in mind for their planning purposes.

6.2. POPS Entry Forms

The following elements are entered directly into POPS web forms as part of the submission. Since they are entered in the POPS forms, these elements need not be repeated within the request documents.

• PI and Co-PI Information: The entries consist of the standard cover sheet information regarding the PI and any co-PIs.
• Project Information: The project title, relevant field(s) of science, the project abstract and keywords. Accurate primary and secondary fields of science ensure that a request can be assigned to experts within the appropriate domain.
• Supporting Grants: Information regarding any grants, awards, funding or other support provided to accomplish the goals of the submitted request. These should be the annual dollar amounts that have been provided for support to the entire project, not just the budget for computing-related activities. If the supporting grant funds activities beyond those proposed for TeraGrid resources (such as experimental work, or work by other collaborators), the PI can specify the percentage of the annual funding for the grant that will be associated with the activities to be supported by TeraGrid resources.
• Resource Request: The resources being requested, along with a summary of the resource needs (i.e., responses to the questions associated with each resource). For a multi-year request, the resource request data entry should only reflect needs for the coming 12 months; out-year resource requirements should be detailed in the Request Documents.
• Request Documents: For Research submissions, specific documents are required as described in Section 6.3. While POPS accepts several file formats, PDF files are recommended.

6.3. Research Request Documents

The supplied documents must discuss the work planned, justifies the resource request, and provides any additional information. The Main Document must adhere to page length rules. See Section 7 for guidelines in crafting a successful submission.

6.3.1. Document Formatting

While readability is of greatest importance, documents must satisfy the following minimum requirements. Documents that conform to NSF proposal format guidelines will satisfy these guidelines.

Margins: Documents must have 2.5-cm (1-inch) margins at the top, bottom, and sides.

Fonts and Spacing: The type size used throughout the documents must conform to the following three requirements:

• The font size must not be smaller than 10 points.
• Type density must be no more than 15 characters per 2.5 cm (1 inch).
• No more than 6 lines must be within a vertical space of 2.5 cm (1 inch).

Page Numbering: Page numbers should be included in each file by the submitter. Page numbering is not provided by POPS.

File Format: POPS accepts PDF and Microsoft Word file formats. PDF files are recommended.

6.3.2. Required and Optional Documents

Submissions should include the following required documents and may include several optional documents. No other documents will be reviewed. This set of information corresponds to what is currently requested by TRAC reviewers and largely to what is provided in current submissions, but it does expand the total permissible page count for many types of submissions. Many projects are not expected to use the full page-limit in all sections. Concise submissions are appreciated.

IMPORTANT: If page limits for the Main Document are exceeded, the Allocation Coordinator will return the submission without review.

POPS NOTE: The Main Document should be uploaded as the Proposal Document in POPS, and all other documents should be uploaded as "optional attachments" in POPS. (POPS will be modified to provide more explicit support for specific documents in the near future.)

Main Document–Required, 10 or 15 pages. This document should have the scientific background, research objectives, and the justification of the request for all resources and resource types. The reviewers will focus on this document, and the document should include sufficient detail to permit reviewers to make a recommendation. The following page limits apply:

• Multi-Year Request OR Single-Year Request for more than 10 million SUs: 15 pages.
• All Other Requests: 10 pages.

Progress Report–Required for renewals and supplements, 5 pages. For renewal requests and continuations of multi-year allocations, the reviewers also consider a PI's record of using their allocations and achieving meaningful results. This document should report on progress made during prior (current) allocation period. This should be submitted as the Main Document to continue a Multi-Year Allocation. The reviewers will consider this document for rating the successful use of prior allocations.

Publications Resulting from TeraGrid Support–Required for Renewals and Progress Reports, no page limit. This document should contain a listing only of bibliographic citations for reviewed publications that benefitted from access to TeraGrid resources. These publications, authored by the PI or users of the allocation, may be in preparation, submitted, accepted, or published. This information is separated for TeraGrid metrics purposes and should not include citations for literature cited elsewhere in the submission (see "References and Figures" below).

CVs–Required, no limit. This is a single document that includes CVs for the PI and all co-PIs. The two-page NSF or NIH format is highly recommended. For supplemental or justification (rebuttals) submissions for which CVs for the same PI and co-PIs have already been submitted in the original request, this document can be omitted.

Special Requirements–Optional, 1 page. This document should be included if the completion of the research plan requires capabilities that fall outside an RP’s standard user and usage environment. For example, jobs longer than the standard queue lengths, specific scheduling demands, or dedicated file system space. The TRAC is not required to read this document; the intended audience is the staff at the relevant RP(s).

Code Performance and Scaling–Optional, 5 pages. This document can contain information to support claims of code performance and scaling that does not fit within the Main Document. Reviewers should not be required to closely scrutinize this document to evaluate the overall request.

References–Optional, no limit. A PI may separate a lengthy bibliography (for literature cited in the Main Document) in order to take full advantage of the maximum page limits. Please note that this bibliography is different from the "Publications Resulting from TeraGrid Support" described above.

Summary of Document Requirements for TeraGrid Research Allocation Resource Requests

na = not applicable

Request Type in POPS	Main Document	Progress Report	Publications Resulting from TeraGrid Support	CVs	Special Requirements	Code Performance and Scaling	References
Page Limits	10 pp.; 15 pp. for multi-year or >10 million SU requests	5 pp.	no limit	2 pp.-format per CV; single combined document	1 p.	5 pp.	no page limit
New	Required	na	na	Required	Optional	Optional	Optional
Renewal	Required	Required	Required	Required	Optional	Optional	Optional
Supplemental	Required	Required	na	Not required if no change in PI/CoPIs	Optional	Optional	Optional
Justification	Required	na	na	Not required if no change in PI/CoPIs	Optional	Optional	Optional
Progress Report	Progress Report is submitted as the Main Document to continue a multi-year allocation	na	Required	Required	Optional	Optional	Optional
Extension	na	na	na	na	na	na	na
Transfer	na	na	na	na	na	na	na
Advance	na	na	na	na	na	na	na

6.4. Review and Processing

Research requests are reviewed quarterly by the TeraGrid Resource Allocations Committee (TRAC). The TRAC consist of volunteer experts from the faculty and staff of U.S. universities, laboratories, and other research institutions. All committee members have expertise in at least one area of computational science or engineering and serve a term of approximately three years.

The TRAC will ensure that:

• All requests for resources that exceed a threshold level determined by the TeraGrid shall be peer-reviewed.
• Written reviews of the resource requests shall be completed in a timely way and made available to the requestors and the allocation coordinators.
• Recommendations to the TeraGrid for the allocation of resources based on the requests, reviews, and the available resources shall be developed and documented.
• The environment for this process shall be consistent with the conflict-of-interest policy and shall maintain confidentiality of requestors and their reviews.

For details on the selection of TRAC members and the TRAC review and meeting process, please refer to the TRAC Policies and Procedures web page [URL forthcoming].

6.4.1. Conflicts of Interest

Every effort is made to avoid conflicts of interest. TRAC members are not allowed to review or be present for the discussion of requests from their home institution; former students, postdocs, or advisors; or current and recent collaborators. In addition, TRAC members who are also PIs or co-PIs on TeraGrid resource requests do not review requests or participate in TRAC deliberations for the meeting to which they submit their requests.

If in the opinion of a PI, a certain individual has a conflict of interest, the PI may request that the individual not act as reviewer on their request or potential subsequent appeal. The Allocations Coordinator will consider such requests for the particular review. Such a request should be included as part of a PI's resource request submission in the Special Requirements document. (POPS will be modified to accept these names directly in the near future.)

The full TRAC conflict of interest policy is at [URL forthcoming] and is incorporated by reference into this document.

6.4.2. Notification of Results

PIs will be notified of the results of the review of Research requests via an email notification within three weeks following the TRAC meeting. A subsequent postal mail confirmation with user information for new accounts will follow soon thereafter.

6.4.3. Confidentiality of Reviews and Requests

All TRAC reviews are considered confidential and are only made available to:

• the PI and co-PIs of the submitted request;
• members of the TRAC without a conflict of interest;
• TRAC reviewers of appeals (justifications), supplemental requests, progress reports, or subsequent renewals for continuing allocations; and
• TeraGrid allocations officers; and
• NSF staff.

TRAC requests and documentation are generally considered confidential, with access limited to the same sets of individuals. However, the requests may be made available to TeraGrid User Services or Advanced User Support staff to assist a research group or evaluate their needs for advanced support.

TeraGrid allocation amounts and descriptive project information (PI name and institution, title, and abstract) are not considered to be confidential.

6.4.4. Appeal Process

PIs who wish to appeal the recommendations of the TRAC may do so by submitting an Appeal. The Appeal should be submitted via POPS within four weeks of the PI being notified of the review results.

The PI may use the Appear to supply additional information or clarification requested by the reviewers in order to make a recommendation. A PI may also use the Appeal to specify why, in the PI's opinion, the reviews of the original request and the resulting allocation were incorrect or unfair. The Appeal should not be used to re-submit a poor request; in such cases, the PI should submit a revised request to the next available TRAC meeting.

If possible, the TRAC members who reviewed the original proposal will consider the Appeal. If there is no response after two weeks from the reviewers, the Allocations Coordinator will make a determination as to any increase to be made to any initial allocation, if resources are available. If the resources to be allocated are not available on the requested resources, the allocation may be made on another resource. Alternatively, the allocation may be made at the beginning of the next allocation cycle on the requested resource.

7. Preparing a Successful Request Document

On average, more than 80% of Research requests receive allocations. However, some allocations may be for less than the level requested. To help ensure that you receive the allocation needed for you to successfully complete your research project, it is important that you provide an appropriate level of justification. The single most important review criterion is the justification of the resource request in the Main Document of the request. Inadequate justification for requested resources is often the reason for reduced or denied allocations. This section provides policies and guidelines to help you craft a successful submission.

In short, the bulk of the Main Document should focus on the resource request, whether for compute, storage, or advanced support.

• For compute resources, the request should address the appropriateness of the computational methodology, the algorithmic capabilities that make this problem a good fit for the requested systems, and the rationale for the number of runs being proposed.>
• For storage resources, the request should address the appropriateness of the storage resource for the data sharing or management needs, the planned use and usage patterns for the data to be stored, and the rationale for the amount of storage space requested.
• For advanced support, the request should address the work plan and the computational objectives that will be targeted with the requested support. There should also be a metric for success of the advanced support and statements on how the advanced support will impact a community beyond the investigator's own research program.

7.1. Scientific Merit

The Main Document of a request for resources will succinctly state the scientific impact of the research to be conducted, and the existing merit-reviewed supporting grants for the research should be listed in the POPS forms.

If the research has had independent review and has been deemed worthy of scientific support as demonstrated by current financial support from a national agency or foundation, the scientific merit and approach will not be subject to further review by the TRAC.

In those circumstances where independent merit-review has not resulted in current financial support from a national agency or foundation, the Allocations Coordinator will highlight this to the reviewers and the TRAC will review the scientific merit and approach of the proposed work. For ongoing computational activities, the TRAC will also consider the progress made using prior allocations, including the publication of peer-reviewed manuscripts and other communications within the community.

7.2. Review Criteria

The Main Document of the resource request will be reviewed against three criteria, which apply across all types of resources, with the level of detail of the review rising with the size of the requested resources:

1. Appropriateness of Methodology: For compute requests, the choice of applications, methods, algorithms and techniques to be employed to accomplish the stated scientific objectives should be reasonably described and motivated. For storage requests, the data usage, access methods, algorithms and techniques to be employed to accomplish the stated research objectives should be reasonably described and motivated. For shared collections, requestors should describe the public or community access methods to be provided.
2. Appropriateness of Research Plan: The steps in the research plan should explain how the research objectives would be achieved. For compute resources, the proposed computations should encompass simulation parameters (step size, time scale, ensemble parameters, etc.) that are needed to obtain accurate and meaningful results, as well as the human resources that can be devoted to the task. The amount of resources requested should be derived from the methodology and the research plan. If there are serious concerns about the research plan, reviewers will document those concerns in their reviews, and the PI may choose to address those concerns in an Appeal.
3. Efficient Use of Resources: The resources selected should be used as efficiently as is reasonably possible and in accordance with the recommended use guidelines of those resources. For computational resources, performance and parallel scaling data should be provided along with a discussion of optimization and/or parallelization work to be done to improve the applications. If the reviewers conclude that the request is more appropriate on TeraGrid resources other than those requested, they may recommend an allocation on those other resources instead.

In considering these criteria, the TRAC recognizes that scientific productivity is the end goal. If adapting to less familiar but, in the view of the panel, better architectures or algorithms requires a significant learning curve for the proposer, with a concomitant interruption of scientific productivity, the TRAC may suggest the alternatives, but nevertheless grant the requested resources with the proposed architecture and methods. In exceptional cases, where the reviewers conclude "beyond a reasonable doubt" that the proposed methods are so inefficient that they amount to a waste of public resources, they should not approve the request until their concerns are addressed by the proposer. Reviewers should be encouraged to recommend advanced support, even for those proposers who have not requested it, as part of their recommendations for addressing shortcomings. For storage resources, information on required performance and expected access patterns should be provided for all data and collections to be stored and used along with a discussion of work done or planned to improve the efficiency of the data use.

7.3. Access to Other CI Resources

In addition, PIs should describe their local cyberinfrastructure resources and other non-TeraGrid resources available to them.

Local Computing Environment: Requests should describe briefly the local (e.g., campus or lab) computing environment available to a research team and, most significantly, how TeraGrid resources will provide capabilities beyond those of local resources or why the requested TeraGrid resources are required in addition to those resources.

Other Supercomputer Resources: If a PI has access to other supercomputer resources or is logically expected to have such access, as with researchers who are affiliated with or are collaborating with other agencies with resource facilities, the document should describe why those resources are not available to the group or why the requested TeraGrid resources are required in addition to those resources.

7.4. Documenting Multi-Year Requests

Multi-year requests must present a program to use resources across two or more consecutive years. Typically, the duration of a multi-year allocation aligns with a PI's or group's supporting grants. If this is not the case, the Main Document should describe how funding will be made available for the duration of the multi-year allocation.

Activities encompassed by multi-year requests should be mature enough to begin research runs at the start of the allocation period. Significant start-up efforts (code development and testing) should be completed via Startup allocations or single-year Research allocations prior to submission of a multi-year request. As noted previously, recipients of multi-year allocations must submit annual Progress Reports to receive each subsequent year's allocation.

7.5. Samples of Successful Request Documents

To assist PIs in writing their documents, Examples of Well-Written Research Allocation Requests are available as models for others to follow.

With these samples, reviewers have expressed the view that these were excellent examples of how an allocation request should be presented. In particular, the reviewers noted that in each case:

• The request summarizes the research in context of the current state-of-the-art, outlines the computational algorithms to be used, and relates these algorithms to the research subsections.
• The reviewers were provided sufficient information but not overwhelmed by details.
• Justification for the allocation request was clear and closely coupled to computational experiments or projects. As such, if the committee needed to reduce the original request, the reduction could have been done rationally with minimum disruption to the investigator.
• Results from previous allocations were summarized and related to this request.

8. Further Details on Eligibility

A PI who applies for a TeraGrid resource allocation must be a researcher or educator at a U.S. academic or non-profit research institution. A principal investigator (PI) may not be a high school, undergraduate, or graduate student (but see Section 8.1); a qualified advisor must serve in this capacity. A postdoc is eligible to serve as a PI. TeraGrid welcomes allocation requests from throughout the science and engineering community. This section clarifies and makes explicit the eligibility rules that apply to specific classes of researchers and educators.

8.1. NSF Grad Student Fellows and Honorable Mentions

While in most cases, a graduate student is ineligible to be PI of an allocation request, an exception is made for NSF Graduate Student Fellows and Honorable Mention recipients. Recipients of these NSF awards can submit requests for Startup allocations for compute time, per arrangement with NSF. In the POPS Startup submission form a checkbox is provided for a person to identify him- or herself as a fellow/honorable mention recipient. Include a CV and supporting documentation (grant number or an award letter) as part of the request submission.

8.2. Other Federal Agencies

Research staff employed by federal agencies or non-NSF FFRDCs are eligible to apply for a TeraGrid allocation if their agency or center does not typically provide research staff with access to non-TeraGrid HPC resources of adequate scope for the planned research.

8.2.1. Military Service Academies

Allocation requests submitted on behalf of faculty members at the military service academies, including the U.S. Military Academy, the U.S. Naval Academy, the U.S. Air Force Academy, the U.S. Coast Guard Academy, the U.S. Naval Postgraduate School, and the Uniformed University of the Health Sciences, will be accepted on the same basis as requests from other academic institutions.

8.3. State and Local Governments

State educational offices or organizations and local school districts may submit allocation requests intended to broaden the impact, accelerate the pace, and increase the effectiveness of improvements in science, mathematics, and engineering education in both K-12 and post-secondary levels. A teacher or educator at an accredited public or private K-12 school is eligible to apply for an allocation as PI.

8.4. Non-Profit, Non-Academic Organizations

Independent museums, observatories, libraries, research laboratories, professional societies and similar organizations in the United States that are directly associated with educational or research activities are eligible.

8.5. Unaffiliated Individuals

Scientists, engineers or educators located within the U.S. may be eligible for support, even if the individual is not employed by or affiliated with an organization provided that

• the project is sufficiently meritorious and otherwise complies with the conditions of any allocation request;
• the requestor has demonstrated the capability to carry out the project.

Unaffiliated individuals should contact the TeraGrid Help Desk before preparing a request for submission.

8.6. Foreign Collaborators

A PI on a request must be a researcher or educator employed at a U.S. institution. Allocation requests with "foreign components" may be made through a domestic PI (with a foreign Co-PI). A foreign component is defined as the performance of any part of the project outside the U.S. either by the PI or a researcher, or researchers employed by a foreign institution. The TeraGrid will not provide an allocation to a PI that does not have a substantive role in the project; that is the PI may not simply serve as a proxy for a foreign researcher.

Collaborative projects involving non-U.S. researchers are encouraged as long as they include substantive intellectual participation by the U.S. researchers. In joint research projects, foreign collaborators are eligible to make use of that allocation in a manner consistent with the request.

Allocations requests involving foreign collaborators will be evaluated using the standard review criteria.

8.7. For-Profit Organizations

U.S. commercial organizations, especially small businesses with strong capabilities in scientific or engineering research or education may apply for an allocation. An allocation request from a commercial organization may be granted when the project is of special concern from a national point of view, special resources are available for the work, or the proposed project is especially meritorious. The TeraGrid is interested in supporting projects that couple industrial research resources and perspectives with those of universities; therefore, it especially welcomes requests for cooperative projects involving both universities and the private commercial sector. It is necessary for these organizations to report their work in an open forum, or make the work readily available to the public.

In addition to supporting scientific research by commercial organizations under the terms described above using the normal TeraGrid resource allocation process, many of the TeraGrid sites have active industrial partnership programs, including funded access to resources without the restrictions associated with free allocations; for additional information see the TeraGrid Industry Partnerships page.