Globus User Guide

Globus is software-as-a-service for file transfer and sharing. Designed specifically for researchers, Globus provides fast, reliable, and secure file transfer among XSEDE resources or between an XSEDE resource and another machine (such as a campus cluster, lab server, or personal computer). Globus is core campus bridging technology that enables researchers to scale their computational research from the desktop, across campus, and to national cyberinfrastructure. Beyond file transfer, Globus allows researchers to securely share data with collaborators directly from existing systems, without investing in additional campus or cloud storage just for the purposes of sharing.

Globus Use Cases

Globus is used for moving data between any two resources, whether it is a small number of very large (even terabyte-sized) files or a very large number of small files. Note: A resource is represented in Globus as an endpoint, identified by a unique name (e.g., the endpoint for TACC's Stampede is "xsede#stampede", and NICS's Kraken is "xsede#kraken").

Globus can be used to move data:

Globus also provides an easy way to share data with collaborators directly from an existing storage system. There is no need to move data to a different system to public cloud service just for the purposes of sharing.

Transferring and Sharing Files

Signing up to use Globus

In order to use Globus you must have a Globus account, that you can create by going to https://www.globus.org/signup. After signing up you will receive an e-mail request to confirm your e-mail address; click on the link to confirm and you will then be able to access the file transfer and sharing service.

Note: You may also, optionally, link your Globus account to your XSEDE User Portal (XUP) identity – this will enable you to access the Globus service using your XSEDE user name and password.

Moving data between XSEDE resources

Globus can be used to move data between any two XSEDE resources. All XSEDE resources are already configured as Globus endpoints.

Moving data between XSEDE and a local machine

It's easy to set up a Globus endpoint on a user's machine and upload or download files between this machine and an XSEDE resource. Globus makes it possible to easily transfer files to and from any machine – even if behind a firewall or NAT without any administrative privileges – using Globus Connect Personal.

Moving data between XSEDE and another HPC system

Any multi-user HPC or shared storage system can be configured as a Globus endpoint using Globus Connect Server. On most servers, installing Globus Connect Server requires just a few commands. Once the software is installed, any user with a local account on that system can move and share files between it and an XSEDE resource. Information on Globus Connect Server is available on the Globus support web site.

Moving data between two non-XSEDE machines

Globus can also be used to move data between any two systems such as a campus cluster, a non-XSEDE computing facility, or a machine that captures data from an instrument – even if the machine is behind a firewall or NAT. You can easily turn any system into a Globus endpoint using Globus Connect.

Sharing data from your Globus endpoint

You can share files from an XSEDE resource or any other Globus endpoint that has been enabled for sharing. Your collaborators need only a Globus account to access files that you have shared with them. Globus ensures that access controls on the local system are enforced while allowing other Globus users to easily access and download the data.

A Globus endpoint can be enabled for sharing by subscribing to Globus Plus (for personal computers) or to a Globus Provider plan (for multi-user servers). As of May 2014 only the XSEDE resources managed by the San Diego Supercomputing Center are enabled for sharing. If you would like to enable your resource for sharing please contact the Globus team about a subscription.

Globus Command Line Interface

Globus provides a command line interface (CLI) that may be accessed using any standard SSH terminal client. Prior to running shell commands, you must upload your public SSH key to your Globus account. See the instructions in this post for more details.

Once your key is uploaded, to enter a secure Globus shell type:

% ssh globususername@cli.globusonline.org

You will see the Globus command prompt:

Welcome to globus.org, .  Type "help" for help.
$ _

Commonly used commands include:

  • Display recently used endpoints with endpoint-list
  • Start a transfer with scp
  • Display the transfer tasks status with status

A complete list of Globus commands, along with with detailed descriptions, is available on the CLI support forum.

Tips

Users can turn off email notifications by running the following command:

login1$ profile -n off

This command will also disable alert emails, such as a notification that your credentials have expired, requiring you to keep a slightly closer watch on long-running transfer requests.

Accessing Globus via a REST API

In addition to the Web and command line interfaces described above, Globus provides programmatic access the file transfer via a REST-style API. The API is a great way to integrate Globus services into your custom portal or science gateway. The API provides Python and Java client libraries.

More information is available on the Transfer API web site. You are also encouraged to use the API mailing list to engage directly with the API developers.

Reference

The Globus service is quite powerful and includes many features not mentioned above (e.g., group management, monitoring, endpoint management, etc.) You can learn more using the following resources:

Last update: May 27, 2016