Data: Where to Initiate File Transfers

Home > User Info > Data > Where to Initiate File Transfers

A system that is configured to automatically start GridFTP server processes is sometimes referred to as a GridFTP server. A GridFTP server process understands requests that adhere to the GridFTP protocol and performs authentication and data transfer operations based on those requests. Using a GridFTP server will speed up data transfers.

See the table GridFTP Server Deployment by Site for GridFTP hostnames at each site.

Dedicated GridFTP Servers

Many sites have nodes that are dedicated GridFTP servers. These nodes utilize the full TeraGrid backbone bandwidth (10-30 Gb/sec), so the Wide Area Network is rarely a bottleneck. Because fewer tasks are running, there should be better file transfer performance. GSI authentication and proxy certificates provide security for transfers.

While all sites have at least one GridFTP server, not all nodes on TeraGrid machines are GridFTP servers. When moving data, request a GridFTP server from the table below. The transfer command will be initiated on a GridFTP server via a round robin assignment.

Use the syntax shown on the Transfer Examples page when integrating transfer requests into job execution scripts.

Third Party Transfers

A GridFTP transfer between two GridFTP servers, rather than between a server and a client, is called a third-party transfer. A third-party transfer occurs when the GridFTP client that is initiating the transfer is run on a system that is neither the source nor the destination of the transfer operation. This allows use of dedicated transfer nodes without having to log in to the transfer node and simplifies the user's workflow. See the Transfer Examples page for specific details.

GridFTP/RFT Server Deployment by Site

The table below provides a list of gridFTP and RFT resources by site. This table provides information on where to start a gridFTP transfer and where to send the transfer (see the file system bandwidth ceiling column for transfer performance details).

Note: Any of the servers listed in the GridFTP Server column can be used as the "from" or "to" location using the tgcp file transfer command. For those resources that do not have servers login to that resource to copy files.

PSC GridFTP Server Deployment

PSC runs GridFTP servers on BigBen (Cray XT3) and on the front-end cache servers for the PSC archiver. No GridFTP servers are provided directly on Rachel (HP GS1280); use gridftp to first send large data files to the archiver and subsequently retrieve them from Rachel using the "far get" command. Similarly, large files to be transferred out of Rachel should first be transferred to the archiver via the "far store" command, after which they may be retrieved using GridFTP.

BigBen:

The GridFTP service on BigBen is comprised of two striped servers, each with 10Gb/s network interfaces. Direct your large-file transfers to/from

gsiftp://gridftp.bigben.psc.teragrid.org

Rachel:

Direct your transfers via GridFTP to either

gsiftp://gridftp.archiver.psc.teragrid.org

or

gsiftp://gridftp.psc.teragrid.org

or

gsiftp://gridftp.rachel.psc.teragrid.org

Note that in all cases, the target is in fact the PSC archiver and not Rachel itself.

To transfer a file from the PSC archiver to Rachel, login to Rachel and use the far get command.

To transfer a file from Rachel to the PSC archiver, login to Rachel and use the far store command.

PSC archiver:

The GridFTP service on the PSC archiver runs on six archiver cache nodes. These are not striped, and each have a 1Gb network interface. Servers are assigned on a round-robin basis as transfers are initiated to/from:

gsiftp://gridftp.archiver.psc.teragrid.org

or

gsiftp://gridftp.psc.teragrid.org

SDSC HPSS Archival Storage

HPSS at SDSC does not have a GridFTP front end. HPSS requires the use of HSI interface to move data in and out of HPSS. Therefore, an extra step is required to transfer between compute resource file systems and another TeraGrid site using the GridFTP server.

1. Move data onto SDSC TG cluster parallel filesystem with hsi.
2. Use a grid-ftp tool to move data elsewhere [to and from SDSC]