Remote Data Download

Portion of NE-CAT Remote landing page, showing location of Sync button in list of Remote sessions

If you collected your data via remote, you can download your data using the sync script.  Click 'sync' button next to the session you want to download on the homepage of the NE-CAT Remote GUI.  This will download the sync script to your machine.  

You run the script from a terminal window on Linux or Apple systems using Python3; i.e.  'python <name of script>.py'.  Running the command with no options will print a usage message, showing a set of options which you can use for downloading your data.  To download your data, run the script with your chosen options. This script runs rsync to transfer data to your filesystem, so if the transfer is interrupted you can simply restart the script in the same location and it will continue copying data that was not transferred.

You may download specific data files or from a target directory.  Use '-da' (or '-dar') to select specific files/directories (the directory tree may take a couple of seconds to load). Select single or multiple files (e.g., 6, 8, ...) to download. Resume interrupted transfers by restarting the script. Use '-t' to download from a specific directory.

The sync script that you download expires after 72 hours. If you still need to download data from a session after your download script has expired, simply return to the NE-CAT Remote GUI and download the sync script again, this resets the time limit.

* * * * * * * * * *

NE-CAT now offers Globus for secure and reliable data transfer, in addition to the existing Python script.  Globus is a non-profit service for secure, reliable research data management developed and operated by University of Chicago (https://www.globus.org/our-story). You can move or share your data between two endpoints (file transfer locations) using the Globus interface. 
 
Step 1: Create a Globus Account
  • Visit globus.org and click Login in the top-right corner.
  • Follow the prompts to create an account. Refer to the Globus manual for detailed instructions.

IMPORTANT: NE-CAT uses the Globus Guest Collection option. After creating your account, please send your Globus ID or the email associated with your account to Ali Kaya.

You will receive two separate emails:

  • One email invites you to join your group. Click "Click here to apply for membership" to accept it.
  • The second email provides access to your group's shared directory. Follow the instructions provided in the email to complete the process.

Step 2: Access and Transfer Data

Check Endpoint Availability:

  • Globus supports transferring data between a source endpoint (e.g., NE-CAT) and your destination endpoint (e.g., local storage or lab server).
  • If you are using your institution's or department's data storage, ensure it has a Globus endpoint. Contact your IT department to obtain the endpoint name, if necessary.

Setting Up a New Endpoint:

  • If transferring data to personal or lab storage, set up an endpoint:
    • Log in to your Globus account.
    • Navigate to the Collections tab on the left.
    • Click Get Globus Connect Personal and follow the setup instructions (see the Globus manual for details).

File Transfer:

  • Follow the prompts to create an account. Refer to the Globus manual for detailed instructions.
  • Open the File Manager in the Globus web app and enable the dual-panel view for source and destination endpoints.
  • Locate the Source Endpoint:
    • Search for the collection name provided by NE-CAT.
    • The collection name remains the same for all your experiments.
    • You will see folders resembling the following format: [example needed here if applicable].
  • Set Up the Destination Endpoint:
    • On the right side, select your endpoint and navigate to the desired directory path.
  • Once both endpoints are selected, start the transfer.

For more information, refer to the Globus manual. If you need assistance, contact Ali Kaya.