Data: Transfer Overview

Home > User Info > Data > Transfer Overview

The data transfer section provides information on the two types of transfers made using the TeraGrid:

• moving data between locations on the TeraGrid. A number of clients are available at the RP sites, most of which are based on gridFTP. You may choose an appropriate client depending on the location and size of your transfer. This page defines the available gridFTP clients. Additional information regarding where to initiate transfers are available via the links in the right side text menu. (see How to Initiate Transfers)
• moving data to the TeraGrid from a local machine. This might include observational data from sensors, surveys, etc., that will be analyzed on TG computing resources. Basic command line commands will work for most transfers. Some sites have clients to assist in moving your data to the TG more efficiently. (see Moving Local Files to the TeraGrid

Transferring data between your local client machine is typically done using common Unix commands. While these commands can be used to move data between TeraGrid resources, it is highly recommended that the GridFTP transfer methods described below be used for optimum performance.

Choosing the Right Data Transfer Method

Although each data transfer may be unique, following a few rules-of-thumb will greatly increase efficiency of transfer:

• GridFTP page for syntax on using GridFTP Clients,
• Where to Initiate File Transfers for information on GridFTP servers,
• Data Storage for information on choosing fast file systems, and the
• Examples parameter optimization syntax page.

There are three GridFTP clients available on TeraGrid for data transfer, uberftp, tgcp, and globus-url-copy. Both uberftp and tgcp utilize and improve on globus-url-copy. Each offers different features; in general, tgcp is recommended for batch jobs and uberftp is an interactive client; that is, it provides step-by-step instructions for the transfer. Please view the table below for a comparison, then click on the name of the client for a detailed description and examples of how to use the client.

GridFTP Client Recommendations

	tgcp	globus-url-copy	uberftp
interface	command line	command line	interactive
use fast file system	yes	yes	yes
tar multiple files	yes	yes	yes
GridFTP servers	automatic	user specified	user specified
optimize parameters	automatic	tcpbuf = 8 parallel = 1	tcpbuf = 8 parallel = 1

The following guidelines are recommended for each client:

1. Use a gridFTP client: tgcp, globus-url-copy, or uberFTP.
2. Utilize dedicated gridFTP servers wherever possible using a third-party transfer.
3. Use optimization parameters: tcp buffer size=8 MB and parallel=1.
4. Plan ahead and send results of your jobs to fast file systems, where available.
5. If you are transferring multiple small files, create a tarball.

While some basic file copy commands may be used for very small files, in general they are markedly less efficient than utilizing gridFTP. An alphabetized list of basic and recommended commands is available in the Data Glossary for quick reference.

Because the TeraGrid is constantly evolving, users should check back for changes in data transfer recommendations. For individualized help with file transfers, please contact the TeraGrid Help Desk.

Data Transfer Performance

What impacts transfer rates?

• Disk speed
• Connectivity of disk to node
• Node characteristics & load
• Connectivity of node to WAN
• For all networks
  • Bandwidth
  • Latency
  • Buffer Size
  • Protocol
  • Load
  • Encryption

Many of these factors are intrinsic to the system and not in the control of the user. However, by using the recommendations above, choices made by the user can significantly influence the speed of data transfers—in some cases resulting in 50 times the transfer speed that results from default settings and non-gridFTP commands. (N.B., Because of the factors above, even with the best optimization, don't expect 40 Gb/sec; performance is usually limited by end node connectivity, not WAN bandwidth.)

Reporting Performance

• PSC daily transfer page