{"lab": {"status": "current", "uuid": "828cd4fe-ebb0-4b36-a94a-d2e3a36cc989", "@id": "/labs/4dn-dcic-lab/", "display_title": "4DN DCIC, HMS", "@type": ["Lab", "Item"], "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "submits_for.828cd4fe-ebb0-4b36-a94a-d2e3a36cc989"]}}, "award": {"@type": ["Award", "Item"], "status": "current", "display_title": "4D NUCLEOME NETWORK DATA COORDINATION AND INTEGRATION CENTER - PHASE I", "uuid": "b0b9c607-f8b4-4f02-93f4-9895b461334b", "@id": "/awards/1U01CA200059-01/", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}}, "title": "Micro-C", "status": "released", "other_tags": ["DNA-DNA", "Pairwise", "3D"], "date_created": "2019-03-28T18:53:02.359079+00:00", "submitted_by": {"error": "no view permissions"}, "last_modified": {"modified_by": {"error": "no view permissions"}, "date_modified": "2020-12-07T17:09:36.025533+00:00"}, "raw_file_types": "Reads (fastq) provided by lab", "reference_pubs": [{"journal": "Cell", "@id": "/publications/e4e88839-f8e5-4263-93c1-d393c5b10625/", "short_attribution": "Hsieh TH et al. (2015)", "display_title": "Hsieh TH et al. (2015) PMID:26119342", "@type": ["Publication", "Item"], "uuid": "e4e88839-f8e5-4263-93c1-d393c5b10625", "date_published": "2015-07-02", "status": "current", "authors": ["Hsieh TH", "Weiner A", "Lajoie B", "Dekker J", "Friedman N", "Rando OJ"], "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}}], "schema_version": "1", "static_content": [{"content": {"options": {"filetype": "md", "collapsible": false, "default_open": true, "convert_ext_links": true}, "filetype": "md", "uuid": "8bf6a722-5841-4a6a-9e4b-947dcf5d7af3", "@id": "/static-sections/8bf6a722-5841-4a6a-9e4b-947dcf5d7af3/", "award": {"status": "current", "@id": "/awards/1U01CA200059-01/", "@type": ["Award", "Item"], "uuid": "b0b9c607-f8b4-4f02-93f4-9895b461334b", "display_title": "4D NUCLEOME NETWORK DATA COORDINATION AND INTEGRATION CENTER - PHASE I", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}}, "content": "The 4DN Hi-C data processing pipeline includes alignment, filtering, and matrix aggregation and normalization steps. Feature calling steps, including insulation scores, compartments, and enriched contacts will be provided in the next version of the pipeline.\n\n
\n\n\n\n
\n\n\n", "name": "resources.data-analysis.hi_c-processing-pipeline.overview", "display_title": "Overview", "@type": ["StaticSection", "UserContent", "Item"], "status": "released", "lab": {"@id": "/labs/4dn-dcic-lab/", "status": "current", "display_title": "4DN DCIC, HMS", "uuid": "828cd4fe-ebb0-4b36-a94a-d2e3a36cc989", "@type": ["Lab", "Item"], "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "submits_for.828cd4fe-ebb0-4b36-a94a-d2e3a36cc989"]}}, "title": "Overview", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "userid.986b362f-4eb6-4a9c-8173-3ab267307e3a"]}}, "location": "tab:data_processing"}, {"content": {"options": {"filetype": "md", "collapsible": false, "default_open": true, "convert_ext_links": true}, "filetype": "md", "uuid": "10000000-0000-0000-0000-ffff00001000", "@id": "/static-sections/10000000-0000-0000-0000-ffff00001000/", "award": {"status": "current", "@id": "/awards/1U01CA200059-01/", "@type": ["Award", "Item"], "uuid": "b0b9c607-f8b4-4f02-93f4-9895b461334b", "display_title": "4D NUCLEOME NETWORK DATA COORDINATION AND INTEGRATION CENTER - PHASE I", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}}, "content": "Hi-C reads are mapped to the [GRCh38](/files-reference/4DNFIZQZ39L9/) (human) or [mm10](/files-reference/4DNFI823LSI8/) (mouse) reference genome using `bwa` version 0.7.17. Specifically, we run:\n\n```bash\nbwa mem -SP5M -t \n```\n\n* The `-SP` option is used to ensure the results are equivalent to that obtained by running `bwa mem` on each mate separately, while retaining the right formatting for paired-end reads. This option skips a step in `bwa mem` that forces alignment of a poorly aligned read given an alignment of its mate with the assumption that the two mates are part of a single genomic segment.\n* The `-5` option is used to report the 5' portion of chimeric alignments as the primary alignment. In Hi-C experiments, when a mate has chimeric alignments, typically, the 5' portion is the position of interest, while the 3' portion represents the same fragment as the mate. For chimeric alignments, `bwa mem` reports two alignments: one of them is annotated as primary and soft-clipped, retaining the full-length of the original sequence. The other end is annotated as hard-clipped and marked as either 'supplementary' or 'secondary'. The `-5` option forces the 5'end to be always annotated as primary.\n* The `-M` option is used to annotate the secondary/supplementary clipped reads as *secondary* rather than *supplementary*, for compatibility with some public software tools such as `picard MarkDuplicates`.\n* The `-t` option is used for multi-threading and should not affect the result.", "name": "resources.data-analysis.hi_c-processing-pipeline.alignment", "display_title": "Alignment", "@type": ["StaticSection", "UserContent", "Item"], "status": "released", "lab": {"@id": "/labs/4dn-dcic-lab/", "status": "current", "display_title": "4DN DCIC, HMS", "uuid": "828cd4fe-ebb0-4b36-a94a-d2e3a36cc989", "@type": ["Lab", "Item"], "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "submits_for.828cd4fe-ebb0-4b36-a94a-d2e3a36cc989"]}}, "title": "Alignment", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "userid.986b362f-4eb6-4a9c-8173-3ab267307e3a"]}}, "location": "tab:data_processing"}, {"content": {"options": {"filetype": "md", "collapsible": false, "default_open": true, "convert_ext_links": true}, "filetype": "md", "uuid": "10000000-0000-0000-0000-ffff00001001", "@id": "/static-sections/10000000-0000-0000-0000-ffff00001001/", "award": {"status": "current", "@id": "/awards/1U01CA200059-01/", "@type": ["Award", "Item"], "uuid": "b0b9c607-f8b4-4f02-93f4-9895b461334b", "display_title": "4D NUCLEOME NETWORK DATA COORDINATION AND INTEGRATION CENTER - PHASE I", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}}, "content": "For filtering valid Hi-C alignments, we use [`pairtools`](https://github.com/mirnylab/pairtools) (previously called `pairsamtools`). Specifically, we use [version `0.2.2`](https://github.com/mirnylab/pairtools/tree/v0.2.2). The filtering workflow outputs a [pairs](https://github.com/4dn-dcic/pairix/blob/master/pairs_format_specification.md) file containing a list of valid contacts. \n\nThis filtering workflow applies the following criteria:\n\n* Reads marked as duplicates are removed.\n* Full-length alignments that are unique are kept.\n * An unmapped portion shorter than 20bp is ignored; and the rest of the alignment is still considered as full-length.\n* In addition, clipped (chimeric) alignments are kept, if they are valid Hi-C contacts. This means one mate is clipped and the other is full-length and the 3'end of the clipped alignment is mapped with 2kb of the full-length alignment in the orientation that the two 3'ends are pointing toward each other.\n * As with full-length alignments, any unmapped portion shorter than 20bp is ignored.\n\nOne of the design choices we have made is to include a lossless bam file as an output of the data processing. This output file, containing all the sequences in the original fastq files, the alignment results, and pairtools-provided flags for read filtering, is provided as a resource. To be able to produce this output file, the contents of the bam file is carried forward in the filtering workflow in intermediate `pairsam` files. Users who are only interested in the valid contact lists may run the same analysis with more light-weight intermediate files.\n\nSpecifically, the filtering workflow consists of the following steps:\n\n* `pairtools parse`\n * A bam file is read in, and a [pairsam](https://pairsamtools.readthedocs.io/en/latest/pairsam.html) file is written out.\n * The pairsam file is a pairs file, listing one read pair per line, with additional columns to track the sam-file lines, and a pairtools read classification.\n * These classifications include information on whether the read aligned to 0, 1, or multiple places in the genome and whether it aligned end-to-end or if it was clipped.\n * This tool also upper-triangularizes the reads, i.e. if the coordinate of second read is higher than the first, the reads are flipped.\n * For more details, see [pairsamtools doc](https://pairsamtools.readthedocs.io/en/latest/parsing.html)\n\n* `pairtools sort`\n * A `pairsam` (or generically `pairs`) file is read in, and a `pairsam` file is written out.\n * The rows are sorted in chr1-chr2-pos1-pos2 order.\n * Note that the flipping order and sort order of chromosomes is not identical. See [the docs](https://pairtools.readthedocs.io/en/latest/sorting.html#chromosomal-order-for-sorting-and-flipping) for more details.\n\n* `pairtools merge`\n * One or more `pairsam` (or generically `pairs`) files are read in, and a `pairsam` file is written out.\n * The files are merged, preserving the sorted order.\n\n* `pairtools dedup --mark-dups`\n * (equivalent to `pairtools markasdup`)\n * Duplicate alignments that share the same pair of 5'end coordinates +/- 3 bps are marked as identified.\n * An arbitrary one is retained with the original classification, while others get a duplicate classification.\n\n* `pairtools select`\n * Only the reads with pairtools classification `UU` and `UC` are retained and output to a `pairs` file.", "name": "resources.data-analysis.hi_c-processing-pipeline.filtering", "display_title": "Filtering", "@type": ["StaticSection", "UserContent", "Item"], "status": "released", "lab": {"@id": "/labs/4dn-dcic-lab/", "status": "current", "display_title": "4DN DCIC, HMS", "uuid": "828cd4fe-ebb0-4b36-a94a-d2e3a36cc989", "@type": ["Lab", "Item"], "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "submits_for.828cd4fe-ebb0-4b36-a94a-d2e3a36cc989"]}}, "title": "Filtering", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "userid.986b362f-4eb6-4a9c-8173-3ab267307e3a"]}}, "location": "tab:data_processing"}, {"content": {"options": {"filetype": "md", "collapsible": false, "default_open": true, "convert_ext_links": true}, "filetype": "md", "uuid": "b32146fd-7b2d-4479-b86f-71b84651234e", "@id": "/static-sections/b32146fd-7b2d-4479-b86f-71b84651234e/", "award": {"status": "current", "@id": "/awards/1U01CA200059-01/", "@type": ["Award", "Item"], "uuid": "b0b9c607-f8b4-4f02-93f4-9895b461334b", "display_title": "4D NUCLEOME NETWORK DATA COORDINATION AND INTEGRATION CENTER - PHASE I", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}}, "content": "Replicates are merged in two steps before producing a matrix.\n\n* Sequencing replicates (replicates within an *experiment*)\n * Sequencing replicates are merged after the alignment but before duplicate removal step (`pairsam dedup`), since reads resulting from a single PCR duplication event may exist in both replicates.\n * Merging is performed on (intermediate) `pairsam` files using `pairsam merge`.\n * DCIC provides a merged output as a `pairs` file.\n* Biological replicates (replicates within an *experiment set*)\n * Biological replicates are merged after duplicate removal step, since PCR duplication event happened independently in these replicates.\n * Merging is performed on `pairs` files using [run-merge-pairs.sh](https://github.com/4dn-dcic/docker-4dn-hic/blob/master/scripts/run-merge-pairs.sh).\n * DCIC provides a merged output as a merged `pairs` file.\n\n", "name": "resources.data-analysis.hi_c-processing-pipeline.merging", "display_title": "Merging replicates", "@type": ["StaticSection", "UserContent", "Item"], "status": "released", "lab": {"@id": "/labs/4dn-dcic-lab/", "status": "current", "display_title": "4DN DCIC, HMS", "uuid": "828cd4fe-ebb0-4b36-a94a-d2e3a36cc989", "@type": ["Lab", "Item"], "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "submits_for.828cd4fe-ebb0-4b36-a94a-d2e3a36cc989"]}}, "title": "Merging replicates", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "userid.986b362f-4eb6-4a9c-8173-3ab267307e3a"]}}, "location": "tab:data_processing"}, {"content": {"options": {"filetype": "md", "collapsible": false, "default_open": true, "convert_ext_links": true}, "filetype": "md", "uuid": "7e577277-525a-4a04-add1-dc7ffa9c462c", "@id": "/static-sections/7e577277-525a-4a04-add1-dc7ffa9c462c/", "award": {"status": "current", "@id": "/awards/1U01CA200059-01/", "@type": ["Award", "Item"], "uuid": "b0b9c607-f8b4-4f02-93f4-9895b461334b", "display_title": "4D NUCLEOME NETWORK DATA COORDINATION AND INTEGRATION CENTER - PHASE I", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}}, "content": "For DNase Hi-C and Micro-C, the Hi-C pipeline is run without \nrestriction enzyme-based intra-fragment contact filtering (as \nthe digestion step does not use restriction enzymes), \nwhile matrix balancing is still performed with the ICE \nalgorithm.\n\n4DN DCIC provides a Hi-C matrix in two different formats: \n`.mcool` format and `.hic` format. The two files are generated\n from the same `pairs` file as input filtered contact list. \nBoth files contain multiple resolutions.\n\n** .hic format**\n\n * A `.hic` file is produced by [Juicertools](https://github.com/theaidenlab/Juicer/wiki/Pre) (version 1.8.9-cuda8) \nand can be visualized using [Juicebox](https://www.aidenlab.org/juicebox/)\n * The matrix is normalized using the VC, VC_SQRT, KR methods.\n\n** .mcool format**\n\n * An `.mcool` file is produced by [Cooler](https://github.com/mirnylab/cooler) (version 0.7.6) and can be visualized using [HiGlass](http://higlass.io/).\n * The matrix is normalized using the *ICE* (iterative \ncorrection and eigenvalue decomposition) matrix balancing \nalgorithm, after the matrix-level removal of the diagonal and \nthe rows/columns with a low value.\n * The `.mcool` file also contains the normalization vectors \ngenerated by *Juicertools* (same as in a `.hic` file generated\n from the same `pairs` file)\n\n
\n\n**Resolutions**: Both `mcool` and `hic` files contain the \nfollowing resolutions.\n\n* 1kb, 2kb, 5kb, 10kb, 25kb, 50kb, 100kb, 250kb, 500kb, 1Mb, \n2.5Mb, 5Mb, 10Mb\n\n", "name": "exptype-dnasehic-matrix-agg", "display_title": "Matrix Aggregation and Normalization", "@type": ["StaticSection", "UserContent", "Item"], "status": "released", "lab": {"@id": "/labs/4dn-dcic-lab/", "status": "current", "display_title": "4DN DCIC, HMS", "uuid": "828cd4fe-ebb0-4b36-a94a-d2e3a36cc989", "@type": ["Lab", "Item"], "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "submits_for.828cd4fe-ebb0-4b36-a94a-d2e3a36cc989"]}}, "title": "Matrix Aggregation and Normalization", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "userid.986b362f-4eb6-4a9c-8173-3ab267307e3a"]}}, "location": "tab:data_processing"}, {"content": {"options": {"filetype": "md", "collapsible": false, "default_open": true, "convert_ext_links": true}, "filetype": "md", "uuid": "0457856c-a342-4781-832e-bb1b95c03290", "@id": "/static-sections/0457856c-a342-4781-832e-bb1b95c03290/", "award": {"status": "current", "@id": "/awards/1U01CA200059-01/", "@type": ["Award", "Item"], "uuid": "b0b9c607-f8b4-4f02-93f4-9895b461334b", "display_title": "4D NUCLEOME NETWORK DATA COORDINATION AND INTEGRATION CENTER - PHASE I", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}}, "content": "The pipeline components are pre-installed in a publicly available Docker image (`duplexa/4dn-hic:v43`) on [Docker Hub](https://hub.docker.com/r/duplexa/4dn-hic/). The source code for the Docker image and pipeline description in Common Workflow Language (CWL) can be found on GitHub.\n\n* Latest runs\n\n Content-wise, `0.2.5`, `0.2.6`, `0.2.7` can be considered (nearly) identical.\n\n * `0.2.7`\n * CWL : https://github.com/4dn-dcic/docker-4dn-hic/tree/v43/cwl\n * Docker : https://github.com/4dn-dcic/docker-4dn-hic/tree/v43\n * `0.2.5`/`0.2.6`\n * CWL : https://github.com/4dn-dcic/docker-4dn-hic/tree/v42.2/cwl\n * Docker : https://github.com/4dn-dcic/docker-4dn-hic/tree/v42.2\n* Old runs\n * `0.2.0`\n * CWL : https://github.com/4dn-dcic/pipelines-cwl/tree/0.2.0/cwl_awsem/\n * Docker : https://github.com/4dn-dcic/docker-4dn-hic/tree/v40\n\nExample set of commands that were actually run as part of the pipeline can be found at https://github.com/4dn-dcic/docker-4dn-hic/blob/v42.2/HiCPipeline.md", "name": "resources.data-analysis.hi_c-processing-pipeline.source-files", "display_title": "Source files", "@type": ["StaticSection", "UserContent", "Item"], "status": "released", "lab": {"@id": "/labs/4dn-dcic-lab/", "status": "current", "display_title": "4DN DCIC, HMS", "uuid": "828cd4fe-ebb0-4b36-a94a-d2e3a36cc989", "@type": ["Lab", "Item"], "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "submits_for.828cd4fe-ebb0-4b36-a94a-d2e3a36cc989"]}}, "title": "Source files", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "userid.986b362f-4eb6-4a9c-8173-3ab267307e3a"]}}, "location": "tab:data_processing"}, {"content": {"options": {"filetype": "html", "collapsible": false, "default_open": true}, "filetype": "html", "uuid": "98dbd1ca-b518-4984-b2eb-4a0bf9f17334", "@id": "/static-sections/98dbd1ca-b518-4984-b2eb-4a0bf9f17334/", "award": {"status": "current", "@id": "/awards/1U01CA200059-01/", "@type": ["Award", "Item"], "uuid": "b0b9c607-f8b4-4f02-93f4-9895b461334b", "display_title": "4D NUCLEOME NETWORK DATA COORDINATION AND INTEGRATION CENTER - PHASE I", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}}, "content": "Below is a description of the file formats available on the portal as outputs of the data processing pipeline. \n\n

\n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
File TypeFile FormatDescription
Alignment.bamFiltered alignments of raw reads to the appropriate reference are available in .bam format. These are generated separately for each replicate experiment in an experiment set.
Contact list.pairsContain a list of pairwise contacts and are generated after alignments are filtered. Pairs are generated separately for each replicate, and an additional pairs file is generated after replicates are merged. Quality metrics generated by PairsQC are also provided for each pairs file (a sample report can be found here).
Contact matrix.mcoolMulti-resolution contact matrices generated in the final step of the processing pipeline are available in .mcool format, generated by Cooler. These can be visualized on the portal with HiGlass. Mcool contact matrices are normalized with the ICE matrix balancing algorithm (iterative correction and eigenvalue decomposition).
Contact matrix.hicContact matrices generated by Juicertools are also available in .hic format, and these can be visualized externally in Juicebox or in the WashU Epigenome Browser. For in situ Hi-C and dilution Hi-C, the .hic matrix is normalized using the VC, VC_SQRT, KR methods.
", "name": "dnasehic-processed-files", "display_title": "dnasehic-processed-files", "@type": ["StaticSection", "UserContent", "Item"], "status": "released", "lab": {"@id": "/labs/4dn-dcic-lab/", "status": "current", "display_title": "4DN DCIC, HMS", "uuid": "828cd4fe-ebb0-4b36-a94a-d2e3a36cc989", "@type": ["Lab", "Item"], "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "submits_for.828cd4fe-ebb0-4b36-a94a-d2e3a36cc989"]}}, "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "userid.986b362f-4eb6-4a9c-8173-3ab267307e3a"]}}, "location": "tab:processed_files"}], "controlled_term": {"@type": ["OntologyTerm", "Item"], "@id": "/ontology-terms/EFO:0008808/", "term_name": "Micro-C XL", "display_title": "Micro-C XL", "term_id": "EFO:0008808", "term_url": "http://www.ebi.ac.uk/efo/EFO_0008808", "uuid": "65e3f12b-656b-4c4e-b419-31691c75c106", "preferred_name": "Micro-C XL", "status": "released", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}}, "experiment_name": "micro-c", "valid_item_types": ["ExperimentHiC"], "experiment_category": "Sequencing", "assay_classification": "3C via Ligation", "assay_subclass_short": "Hi-C", "assay_subclassification": "DNA-DNA Pairwise Interactions", "@id": "/experiment-types/micro-c/", "@type": ["ExperimentType", "Item"], "uuid": "bc7899cf-8352-4360-b9be-efc3c7e3a574", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}, "display_title": "Micro-C", "external_references": [], "@context": "/terms/", "aggregated-items": {}, "validation-errors": []}