Data: How to Initiate Transfers: GridFTP Clients

Home > User Info > Data > How to Initiate Transfers: GridFTP Clients

GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth, wide-area networks. TeraGrid has three clients which utilize GridFTP (click on the client name to see more information and examples of that method):

• UberFTP (recommended)
• globus-url-copy (preferred for scripting)
• tgcp (deprecated)

Refer to the GridFTP Server Deployment by Site table for GridFTP information for each site, including host names and bandwidth information. For more information on CTSS kits that support GridFTP, see the CTSS documentation.

GridFTP provides the following protocol features:

• GSI security on control and data channels
• Multiple data channels for parallel transfers
• Partial file transfers
• Third-party (direct server-to-server) transfers
• Authenticated data channels
• Reusable data channels
• Command pipelining

What GridFTP is Not

GridFTP is not a command; there is no client program named GridFTP.  Some confusion may arise from recommendations to "use GridFTP to transfer your files."   This actually means that the user should choose a client that uses the GridFTP protocol.  (See UberFTP, globus-url-copy, or tgcp below.)

The source and destination arguments may be URLs, local files, or standard input/output.  The following table shows some acceptable values for these arguments:

Source/Destination	Description
file:<fullpath> file://<host><fullpath>	For local files -- relative pathnames not allowed
gsiftp://<host><path>	For remote files -- relative paths allowed
http://<path to file> https://<path to file>	For accessing web files—relative paths allowed
-  (dash)	Standard input and output

UberFTP

UberFTP is an interactive GridFTP file transfer client.  In addition to standard FTP client mechanics, UberFTP supports GSI authentication, parallel data channels and striping. Using UberFTP opens a session within which files may be transfered and directories and files may be manipulated. UberFTP on the TeraGrid requires GSI authentication; that is, the user must use an active proxy session. Some users prefer UberFTP over globus-url-copy because the syntax is less verbose and less prone to typographical errors. However, globus-url-copy is more versatile for embedding data transfers into job scripts.

To find TeraGrid resources that support UberFTP, and for details about specific versions, please view the CTSS documentation.

Usage

       uberftp [-?] [-d]
               [ host [-P port] [-a auth] [-c N] [-t type] ]

       uberftp [-?] [-d]  host  [-P port] [-a auth]
               [-c N] [-t type] "cmd"

       uberftp -v

Options

       host      Connect to host
       -P port   Connect to port (default 2811 for GSI)
       -a auth   Use auth authentication (default is GSI)
       -c N      Use N parallel data connections. Defaults to 1.
       -t type   Use ASCII or BINARY transfer type. Valid type options
                 are 'a' (ASCII) or 'b' (BINARY). Default is BINARY.
       cmd       This specifies the FTP commands to run once the control
                 connection is establish. This list must be enclosed
                 in quotes. Multiple commands are semicolon delimited
                 uberftp will execute these commands and then exit
       -v        Print version information and exit
       -?        Print this help message
       -d        Turn on debug (verbose) output

A few commands are listed below for convenient reference.

put - send a single file to the remote service get - retrieve a single file from the remote service mput - send multiple files that match the given expression mget - retrieve multiple files that match the expression

ascii/binary - change transfer type blksize - change the block size used with extended block mode passive - toggle between active (PORT) and passive (PASV) parallel N - set N number of parallel data streams tcpbuf - change the TCP buffer size

Optimization Parameters

The example below shows starting an UberFTP session and settiing transfer parameters for normal network traffic:

tg-login1> uberftp
uberftp> parallel 1
uberftp> tcpbuf 8388608
TCP buffer set to 8388608 bytes

Example showing parameters when experiencing high network traffic:

tg-login1> uberftp
uberftp> parallel 2
uberftp> tcpbuf 4194304
TCP buffer set to 4194304 bytes

For more information on UberFTP, please see the NCSA uberFTP home page.

See additional DMOVER documentation.

globus-url-copy

The globus-url-copy client program is a GridFTP client for transfering files from the command line. It is not an interactive command. Because it is a command line client, no feedback is provided after the command is executed. It is the client of choice for embedding transfers in job scripts. It is a part of globus-client, a required component of the CTSS Application Development and Runtime Support Capability Kit. As such, it is available on all resources which support that kit. You can search for more information in the CTSS documentation.

Usage

% globus-url-copy <source_url> <destination_url>

where <source> or <destination> is of the format:

• if local file, file:<full path>
• if remote file, gsiftp://<hostname>/<full path>

Optimization Parameters

To optimize performance of data transfers using this client:

• specify the size (in bytes) of the buffer to be used by the underlying ftp data channels

-tcp-bs <size> | -tcp-buffer-size <size>

• specify the number of streams to be used in the ftp transfer

-p <parallelism> | -parallel <parallelism>

Based on file transfer trials, using 1 parallel stream and 8 MB buffer should provide optimum performance.  For extremely large files (>X GB) or with high network traffic, you may wish to change the number of parallel streams to 2 and the buffer size to 4MB.  (N.B., that the product of these two parameters should remain 8.)

Examples

Copying a local text file to SDSC:

% globus-url-copy file:`pwd`/prog.c \
gsiftp://tg-login.sdsc.teragrid.org/~/run1/prog.c

Getting a file from NCSA's mass storage system, (command issued from tg-login.ncsa.teragrid.org):

globus-url-copy gsiftp://mss.ncsa.uiuc.edu/~/prog.c \
file:`pwd`/prog.c

tgcp

The transfer command tgcp is a command-line user tool intended to provide high-performance file transfer, while simplifying efficient copying of files and directories between and within GridFTP enabled clusters. tgcp is a wrapper for globus-url-copy and RFT (Reliable Transfer Service), providing options such as '-big', for a striped globus-url-copy, and '-rft', to manage transfers using RFT. The tgcp wrapper invokes third-party transfers between GridFTP servers at the TeraGrid source and the TeraGrid destination sites. Because an administrator maintains configuration files describing optimal TCP buffer sizes for specific TeraGrid host/domain source-destination pairs, users do not need to enter these values.

Striped GridFTP, RFT, and tgcp clients are available on all TeraGrid resources.

Please be advised that tgcp, although currently a required component of the CTSS Application and Development Runtime Kit, is deprecated, and therefore not guaranteed to work correctly on all TeraGrid resources. It will soon become an optional component and disappear from some resources. The recommended GridFTP copy clients to use for most resources are UberFTP and globus-url-copy.

Examples

The following examples show four methods of using tgcp to transfer files.

Transfer of files larger than 100 MB using the "-big" option

The following example describes a transfer of the file "bigfile" from the local site to the site specified in the command (in this case, SDSC). The option "-big" designates a striped transfer and should always be used to transfer files larger than 100 MB.

tgcp -big FILENAME tg-login.sdsc.teragrid.org:/users/USERNAME/FILENAME.dest

Using Globus RFT (Reliable Transfer Service)

RFT is a service that performs third party transfers on behalf of the requester. The benefits to using RFT are performing third party GridFTP transfers for the user on a separate server, transfering entire directories, performing multiple transfers in parallel, and automatically retrying individual transfers if they fail. This last feature ensures that the TGCP operation completes regardless of whether the user waits for it to complete or not.

tgcp -rft FILENAME /disks/scratchgpfs1/USERNAME/FILENAME.onscratch

Transfer of a small file

The following example describes a transfer of the file "smallfile" from the local site to the site specified in the command (in this case, SDSC):

tgcp FILENAME tg-login.sdsc.teragrid.org:/users/USERNAME/FILENAME.dest

When transferring large files the '-big' option should be used as a hint that tgcp should use high-performance file transfer methods. Since high-performance file transfer methods may have a higher startup cost, this option should not be used for small transfers.

Local home to local scratch transfer

The following example describes a transfer of the file "bigfile" from the local site to local scratch. The option "-stripe" designates a striped transfer:

tgcp -stripe FILENAME /disks/scratchgpfs1/USERNAME/FILENAME.onscratch

The current method for improving performance when '-big' is specified is using striped GridFTP, which is equivalent to the '-striped' option in globus-url-copy. tgcp supports the '-stripe' option for users who are familiar with globus-url-copy.