Home
Help Section
User Guide

Programmatic Access

  1. User Guide
  2. Programmatic Access
    1. Overview
      1. Overview
    2. Dcicutils
      1. DCIC utils
    3. REST API
      1. Overview
      2. Connecting to the server
      3. Identifying specific items
      4. Ordering of POST requests
      5. JSON formatting
    4. Downloading Files
Previous
Account Creation
Next
Downloading Files

Overview

There are several ways to access the metadata and data on the 4DN data portal. The most straightforward and intuitive ways to do so are via the search and browse functionality on the web site.

search box search box

browse menu browse menu

The web site also facilitates downloading files individually or in bulk from data sets of interest.

However, as you become more familiar with the data sets available from the portal you may want to programmatically access metadata and data. Additionally, while we provide workbooks and easy to use tools for data submission some advanced users may wish to do submissions programmatically.

The sections below explain some of the options available.

DCIC utils

If you program in Python the 4DN-DCIC has developed and maintains a package of utilities that contains useful functions for accessing 4DN metadata from the portal. You can install it with pip install dcicutils

To get started using this package and for some information on useful functions to access metadata see the documentation here

If you've already installed the package and want to see some examples of how you can use it look here

If you don't use Python expand the REST API section below.

  REST API

Downloading Files

File download from the 4DN data portal now requiresauthentication, even if the file is public. Accounts can becreated by anyone, including those not part of the 4DNNetwork; for more information on account creation, see the[Account Creation](/help/user-guide/account-creation) page.Files can be downloaded from the web portal after loggingin, as usual; for file download from the command line, theprocess is described below.

First you need to create a new access key, if you don't havean access key already. Note that access keys created forJupyterhub can't be used for the rest of the portal.

To create a new access key, first log in to the data portal,then click on your account in the upper right and click onProfile from the dropdown menu. There will be a button nearthe bottom of the page to add an access key. Save the keyand secret; typically this is done by creating a file inyour home directory called keypairs.json with thefollowing contents/format (replacing the x’s with theappropriate key and secret, of course):

{
  "default": {
      "key": "XXXXXXXX",
      "secret": "xxxxxxxxxxxxxxxx",
      "server": "https://data.4dnucleome.org"
  }
}

Once the access key is created and stored, the file can bedownloaded via curl with the following command:

curl -O -L --user <key>:<secret> <download-url>

For more information on file downloads see this page

Previous
Account Creation
Next
Downloading Files