Remote Data Download

Portion of NE-CAT Remote landing page, showing location of Sync button in list of Remote sessions

If you collected your data via remote, you can download your data using the sync script.  Click 'sync' button next to the session you want to download on the homepage of the NE-CAT Remote GUI.  This will download the sync script to your machine.  

You run the script from a terminal window on Linux or Apple systems using python; i.e.  'python <name of script>.py'.  Running the command with no options will print a usage message, showing a set of options which you can use for downloading your data.  To download your data, run the script with your chosen options. This script runs rsync to transfer data to your filesystem, so if the transfer is interrupted you can simply restart the script in the same location and it will continue copying data that was not transferred.

The sync script that you download expires after a week. If you still need to download data from a session after your download script has expired, simply return to the NE-CAT Remote GUI and download the sync script again, this resets the time limit.

* * * * * * * * * *

NE-CAT is starting to use Globus and sftp to transfer data in addition to the legacy Python script. 

Globus is a non-profit service for secure, reliable research data management developed and operated by University of Chicago (https://www.globus.org/our-story). You can move or share your data between two endpoints (file transfer locations) using the Globus interface. 

To transfer data from NE-CAT using Globus you must have or setup: a Globus account, Globus connect personal (Globus Desktop/Laptop App), a Globus Endpoint, your ANL Domain or APS login credentials, and you must be listed as an Experimenter on the ESAF associated with the trip from which you are trying to download data.

You can find more information at the Globus and sftp manual. If you need assistance, please contact with Ali Kaya

If you are already using Globus and you have an endpoint setup, please follow the instructions below. Otherwise please see the manual (available from this page as a PDF) for detail instruction. 

  1. Go to www.globus.org
  2. Click 'Log In'
  3. Follow the prompt, log in with your home institution credential.
    • You must use your own institution. When you select your own institution, it will bring you to your institutional Globus-related login web page where you need to enter your institutional username and password. Then, your institution will authenticate you with Globus.

  4. After log in, you'll be at the File Manager page. Search aps#data for APS endpoint at File manager connection bar.
  5. APS will ask for your credentials. You can use either ANL domain or APS Web Account credentials. To access with your APS Web account, please enter your badge number starting with d, such as d12345 and enter your APS password. You can find more information at the Globus and sftp manual. This credential will expire after 7 days.
  6. To reach the NE-CAT files, enter /gdata/dm/NECAT/  in the path bar.
  7. Select your institution, lab (group name) and remote session. 
  8. Select your folder(s) or file(s).
  9. Select your endpoint and start transfer.
  10. APS will check your credentials for data transfer. To transfer data, your name must be on the ESAF which is related to your session. Even if your name is in the ESAF, you may get a “Permission denied” error message at the beginning of the transfer. This should resolve after ~10-20 mins.
AttachmentSize
PDF icon Globus Manual5.93 MB
PDF: