ISMB-Tutorial-Part-2-Soo-Nils-Peter.html

<!DOCTYPE html>
<html>
  <head>
    <title>3D Genome Data Procesing, Analysis, and Visualization</title>
    <meta charset="utf-8">
    <style>
      @import url(https://fonts.googleapis.com/css?family=Yanone+Kaffeesatz);
      @import url(https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic);
      @import url(https://fonts.googleapis.com/css?family=Ubuntu+Mono:400,700,400italic);

      body { font-family: 'Droid Serif'; }
      h1, h2, h3 {
        font-family: 'Yanone Kaffeesatz';
        font-weight: normal;
      }

      .column:first-of-type {float:left}
        .column:last-of-type {float:right}

        .split-40 .column:first-of-type {width: 40%}
        .split-40 .column:last-of-type {width: 60%}

      .remark-code, .remark-inline-code { font-family: 'Ubuntu Mono'; }
    </style>
  </head>
  <body>
    <textarea id="source">

class: center, middle

# 3D Genome Data Processing, Analysis, and Visualization Tutorial

## ISMB 2017, Prague, Czech Republic

#### Nezar Abdennur, Soo Lee, Nils Gehlenborg, Peter Kerpedjiev, Jian Ma

---

class: center, middle

# WiFi

**Username:** ISMBECCB

**Password:** PRAGUE17

---

# Preparation

### Connect to a Linux VM instance on Amazon Cloud
```
ssh <username>@<aws_public_ip>
```

  * username: prague
  * passwd: pipelinepipeline

---

# Practice!
Hi-C data processing pipeline

1. Mapping
  * Tools : `bwa`, `samtools`
  * output: `bam` file

2. Filtering / sorting / Creating a list of contacts
  * Tools: `pairsamtools`, `pairix`
  * outputs: `pairs`, `bam` files

3. Binning
  * Tools: `cooler`
  * outputs: `cool` file

4. Normalization
  * Tools: `cooler`
  * outputs: `cool` file


---

# Run docker


* create a data directory
```
mkdir -p data
```
&nbsp;

* pull docker image and create a container with bash entry point
```
docker run -it -v ~/data:/d/:rw duplexa/4dn-hic:v27 bash
```
&nbsp;

* cd into the data directory
```
cd /d/
```

---

# Download Data

* FASTQ input data : 
```
  wget -O input_R1.fastq.gz https://goo.gl/VYdHX9
  wget -O input_R2.fastq.gz https://goo.gl/wPDXV3
  gunzip input_R1.fastq.gz
  gunzip input_R2.fastq.gz
```

* bwaIndex
```
wget -O hg38.bwaIndex.tgz https://goo.gl/SU61DB 
tar -xzf hg38.bwaIndex.tgz
```

* chrom size
```
wget -O hg38.mainonly.chrom.size https://goo.gl/WfSQEV 
```

---

# Output files can also be downloaded

  * wget -O output.bam http://goo.gl/SZAbFS
  * wget -O output.pairs.gz http://goo.gl/Dmh5gn
  * wget -O output.cool http://goo.gl/z5YNrd

---

# Alignment!

  * Tools : 
    * `bwa` (http://bio-bwa.sourceforge.net/) : alignment
    * `samtools` (https://samtools.github.io/) : format conversion (sam->bam)

  * Command
    ```bash
    bwa mem -SP5M <index> <fastq1> <fastq2> | samtools view -bhS - > output.bam
    ```

  * input: `fastq` files, bwa genome index files

  * output: `bam` file

  * Add `-t <ncores>` to `bwa mem` to use multiple cores (e.g. `-t 4`)

---

# How to read a `bam` file

  * A `bam` file is a binary file that stores alignment information for sequenced reads

  * Tools : 
    * `samtools` (https://samtools.github.io/) : viewing a bam file

  * Command for viewing
    ```bash
    samtools view <bamfile> | less
    ```

  * SAM/BAM format: https://samtools.github.io/hts-specs/SAMv1.pdf

  * Example lines
  ```bash
  SRR1658581.203938	81	chr5	76758064	0	51M50S	=	76757789	-326	GGGGTTTTACCTTGTTGGCCAGGCTGGTCTTGAACTCCTGACCTCATGATCGATCATCACCATCTTGGTTTTGGTGGGTTTTGGCCGGCTTCCTTACTGCA	.BB@DCCC@DCDDBBBDDDDCDDDDDDDDEDCC>DECCEEEFFHGHHHJIGIJJJJHHFIIIJIIJHIJJJJJJJJJJJJJJJIJJJJHHHHHFFFFFCCC	NM:i:0	MD:Z:51	AS:i:51	XS:i:48	SA:Z:chr5,76780991,+,51M50S,0,0;
  SRR1658581.203938	2113	chr5	76780991	0	51M50H	=	76757789	-23203	TGCAGTAAGGAAGCCGGCCAAAACCCACCAAAACCAAGATGGTGATGATCG	CCCFFFFFHHHHHJJJJIJJJJJJJJJJJJJJJIHJIIJIIIFHHJJJJIG	NM:i:0	MD:Z:51	AS:i:51	XS:i:43	SA:Z:chr5,76758064,-,51M50S,0,0;
  SRR1658581.203938	161	chr5	76757789	60	101M	=	76758064	326	TTGATTTTTCCTTCTGGACAGGCTCCAATATGGCTCTGTGCAGCACTGTTATTGGTCCTGTCGTTACTGAAAATTCTGATTTTTTTTTTTTGGAAACGGAG	CCCFFFFFHHHHHJJJJJJJJJJEHJJJJIIIJFEIJIHIIJIIIGIIHHEIIEGHGHGIFHIHIGIGIIJJHGHHHH;?CEEDDDDDDDD<-4>AC?B85	NM:i:0	MD:Z:101	AS:i:101	XS:i:0

  ```

---


# Filtering!

  * Tools : 
    * `pairsamtools` (https://github.com/mirnylab/pairsamtools) : filtering

  * Command
    ```bash
    samtools view -h <input_bam_file> | {
        pairsamtools parse -c <chromsize_file> 
    } | { 
        pairsamtools sort --nproc 4
    } | { 
        pairsamtools select '(pair_type == "CX") or (pair_type == "LL")'
    } | { 
        pairsamtools dedup --output \
            >( pairsamtools split \
                --output-pairs <output.pairs.gz> )
    }
    ```

  * input: `bam` file

  * output: `bam`, `pairs` file


---


# Pairs file
 
  * A `pairs` file is a block-gzipped text file that stores list of contact loci

  * Tools:
    * `pairix` (https://github.com/4dn-dcic/pairix) : indexing & querying `pairs` files

  * Command
    ```bash
    pairix <pairsfile>  # indexing 
    pairix <pairsfile> 'chr2:1-60000000|chr4:5000000-6000000' # querying
    ```

  * Pairs format: https://github.com/4dn-dcic/pairix/blob/master/pairs_format_specification.md

  * Example lines
  ```bash
  #columns: readID chr1 pos1 chr2 pos2 strand1 strand2
  EAS139:136:FC706VJ:2:2104:23462:197393 chr1 10000 chr1 20000 + +
  EAS139:136:FC706VJ:2:8762:23765:128766 chr1 50000 chr1 70000 + +
  ```

---

# Binning!

  * Tools:
    * `cooler` (https://github.com/mirnylab/cooler)

  * Command
    ```bash
    cooler cload pairix <chrsize_file>:<bin_size> <pairs_file> <out_prefix>.cool
    ```

  * input: `pairs` file, `chromsize` file

  * output : `cool` file

  * binsize : e.g.) 5000 (high resolution), 500000 (lower resolution) 

---

# How to read a `cool` file

  * A `cool` file is an HDF5-format file that stores contact matrices and normailzation vectors.

  * Tools:
    * `cooler` (https://github.com/mirnylab/cooler)

  * Command
    ```bash
    cooler dump -t pixels --header --join -r chr19 out.cool 
    ```

  * Cool format : http://cooler.readthedocs.io/en/latest/datamodel.html


---

# Normalization! (Matrix balancing)

  * Tools:
    * `cooler` (https://github.com/mirnylab/cooler)

  * Command
    ```bash
    cooler balance <cool_file>
    ```

  * input : `cool` file
  
  * output : `cool` file

  * To read a 'normalized' matrix, add `-b` to the `cooler dump` command.

---

# Distiller 

  * A modular Hi-C mapping pipeline for reproducible data analysis.
  * Nextflow/Docker-based pipeline
  * https://github.com/mirnylab/distiller-nf


---

# Visualization

To explore the features of a matrix, it helps to be able to see the data. Example with HiGlass:

<p style="text-align: center">
    <img src="img/sox2.png" width="600" />
</p>

---

# Existing tools

<p style="text-align: center">
    <table style="margin: 0px auto">
        <tr>
            <td> Juicebox </td>
            <td> <img style="margin-left: 20px" src="img/juicebox-rectangle.png" height="120" /> </td>
        </tr>
        <tr>
            <td>WashU Epigenome Browser</td>
            <td><img style="margin-left: 20px" src="img/washu-epigenome-browser.png" height="120" /></td>
        </tr>
        <tr> 
            <td> 3D Genome Browser </td>
            <td><img style="margin-left: 20px" src="img/3d-genome-browser.png" height="120" /></td>
        </tr>
    </table>
</p>

---

# HiGlass 


<p style="text-align: center">
    <img src="img/higlass-overview.png" width=400 />
</p>

HiGlass is a viewer for HiC and other genomic data. It can be used
online (<a href="http://higlass.io/" >http://higlass.io</a>), or run locally with private data.

<h3>
</h3>

---

# Capabilities

HiGlass is designed to facilitate the comparison of multiple genomic datasets through
"tracks" and "views".


<p style="text-align: center">
    <img src="img/g12878_vs_K257.png" width=600/>
</p>

---

# Anatomy of HiGlass

<table>
    <tr>
        <td>
            <p style="text-align: center">
                <img src="img/higlass-anatomy-11.png" width=400/>
            </p>
        </td>
        <td>
            <p style="margin-left: 15px">
            HiGlass is composed of a collection of views and
            tracks.
            </p>
        </td>
    </tr>
</table>

---

# Anatomy of HiGlass

<table>
    <tr>
        <td>
            <p style="text-align: center">
                <img src="img/higlass-anatomy-10.png" width=400/>
            </p>
        </td>
        <td>
            <p style="margin-left: 15px">
            Each view is composed of a set of tracks which share
            common axes. 
            </p>
        </td>
    </tr>
</table>

---

# Anatomy of HiGlass

<table>
    <tr>
        <td>
            <p style="text-align: center">
                <img src="img/higlass-anatomy-9.png" width=400/>
            </p>
        </td>
        <td>
            <p style="margin-left: 15px">
            Settings related to the view can be accessed via the 
            "view settings" menu. 
            </p>
        </td>
    </tr>
</table>

---

# Anatomy of HiGlass

<table>
    <tr>
        <td>
            <p style="text-align: center">
                <img src="img/higlass-anatomy-8.png" width=400/>
            </p>
        </td>
        <td>
            <p style="margin-left: 15px">
            New views can be created by cloning an existing view.
            </p>
        </td>
    </tr>
</table>

---

# Anatomy of HiGlass

<table>
    <tr>
        <td>
            <p style="text-align: center">
                <img src="img/higlass-anatomy-7.png" width=400/>
            </p>
        </td>
        <td>
            <p style="margin-left: 15px">
            Views can be closed by clicking on the "close view" button ("X").
            </p>
        </td>
    </tr>
</table>

---

# Anatomy of HiGlass

<table>
    <tr>
        <td>
            <p style="text-align: center">
                <img src="img/higlass-anatomy-6.png" width=400/>
            </p>
        </td>
        <td>
            <p style="margin-left: 15px">
            The contents of a view can be modified by adding tracks.
            </p>
        </td>
    </tr>
</table>

---

# Anatomy of HiGlass

<table>
    <tr>
        <td>
            <p style="text-align: center">
                <img src="img/higlass-anatomy-5.png" width=400/>
            </p>
        </td>
        <td>
            <p style="margin-left: 15px">
            Tracks can be layed out horizontally where they track
            the horizontal axis.
            </p>
        </td>
    </tr>
</table>

---

# Anatomy of HiGlass

<table>
    <tr>
        <td>
            <p style="text-align: center">
                <img src="img/higlass-anatomy-4.png" width=400/>
            </p>
        </td>
        <td>
            <p style="margin-left: 15px">
            Tracks can also be layed out vertically where they 
            track the vertical axis.
            </p>
        </td>
    </tr>
</table>

---

# Anatomy of HiGlass

<table>
    <tr>
        <td>
            <p style="text-align: center">
                <img src="img/higlass-anatomy-3.png" width=400/>
            </p>
        </td>
        <td>
            <p style="margin-left: 15px">
            And tracks can be placed in the center, where they move
            along both axes.
            </p>
        </td>
    </tr>
</table>

---

# Anatomy of HiGlass

<table>
    <tr>
        <td>
            <p style="text-align: center">
                <img src="img/higlass-anatomy-2.png" width=400/>
            </p>
        </td>
        <td>
            <p style="margin-left: 15px">
            Track settings can be adjusted via the track settings
            menu.
            </p>
        </td>
    </tr>
</table>

---

# Anatomy of HiGlass

<table>
    <tr>
        <td>
            <p style="text-align: center">
                <img src="img/higlass-anatomy-1.png" width=400/>
            </p>
        </td>
        <td>
            <p style="margin-left: 15px">
                Overlays can be added on top of existing tracks by clicking
                on the '+' icon of the track toolbar.
            </p>
        </td>
    </tr>
</table>

---

# Anatomy of HiGlass

<table>
    <tr>
        <td>
            <p style="text-align: center">
                <img src="img/higlass-anatomy-0.png" width=400/>
            </p>
        </td>
        <td>
            <p style="margin-left: 15px">
                Tracks can be closed by clicking on the 'X' of the
                track toolbar.
            </p>
        </td>
    </tr>
</table>

---

# Practical exercise

Re-create this figure:

<p style="text-align: center">
    <img src="https://cloud.githubusercontent.com/assets/2143629/24535688/a6043f1a-15a3-11e7-8197-d5d3ce227bc6.png" height=400/>
</p>

---

class: center, middle

### Navigate to HiGlass website

# [http://higlass.io/app](http://higlass.io/app)

---

# Remove genome position search bar

<p style="text-align: center">
    <video width="600" loop="loop" autoplay="autoplay">
        <source src="img/remove-genome-position-search-bar.mp4" type="video/mp4">
    </video>
</p>

The genome position search bar serves to show the visible extent of the data.
It can also be used to navigate to a particular locus.

---

# Remove all existing tracks

<p style="text-align: center">
    <video width="580" loop="loop" autoplay="autoplay">
        <source src="img/remove-tracks.mp4" type="video/mp4">
    </video>
</p>

We don't need any of the default tracks, so we'll remove them.

---

# Add new dataset

<p style="text-align: center">
    <video width="620" loop="loop" autoplay="autoplay">
        <source src="img/add-schwarzer-wt.mp4" type="video/mp4">
    </video>
</p>

---

# Resize the current view and replicate it

<p style="text-align: center">
    <video width="620" loop="loop" autoplay="autoplay">
        <source src="img/resize-and-triple.mp4" type="video/mp4">
    </video>
</p>

---

# Replace existing dataset and copy it

<p style="text-align: center">
    <video width="560" loop="loop" autoplay="autoplay">
        <source src="img/replace-and-double.mp4" type="video/mp4">
    </video>
</p>

---

# Lock zoom and location

<p style="text-align: center">
    <video width="560" loop="loop" autoplay="autoplay">
        <source src="img/lock-zoom.mp4" type="video/mp4">
    </video>
</p>

http://higlass.io/app/?config=FQWaiKoATPSMVYyOu11rBg

---

# Zoom top to: 

```
chr14:50,143,364-88,897,098 & chr14:70,582,768-97,882,196 [offset 0,0:0,0]
```

<p style="text-align: center">
    <video width="500" loop="loop" autoplay="autoplay">
        <source src="img/higlass-zoom-top.mp4" type="video/mp4">
    </video>
</p>

By providing a set of genomic coordinates, we can zoom to a defined region
in the heatmap.

---

# Zoom bottom to:

```
chr14:66,384,132-76,643,845 & chr14:67,964,160-79,229,728 [offset 0,0:0,0]
```

<p style="text-align: center">
    <video width="500" loop="loop" autoplay="autoplay">
        <source src="img/higlass-zoom-bottom.mp4" type="video/mp4">
    </video>
</p>

http://higlass.io/app/?config=JvtlSiVnSi-SSraD0zmsjA


---

# Project viewport

<p style="text-align: center">
    <video width="600" loop="loop" autoplay="autoplay">
        <source src="img/project-viewport.mp4" type="video/mp4">
    </video>
</p>

Projecting a viewport shows the extent of one view on another. In this case,
we're showing the extent of the bottom views on the top.

---

# Share a link


<p style="text-align: center">
    <video width="600" loop="loop" autoplay="autoplay">
        <source src="img/higlass-export-link.mp4" type="video/mp4">
    </video>
</p>

The current view configuration can be shared by exporting it as a link (http://higlass.io/app/?config=XVJAsgUbRva2diamiW9mfg)

---

# Add 1D track

<p style="text-align: center">
    <video width="580" loop="loop" autoplay="autoplay">
        <source src="img/higlass-add-rnaseq.mp4" type="video/mp4">
    </video>
</p>

We can add any sort of 1D data as a horizontal or vertical track. In this
case we add RNAseq data as horizontal track on top.

---

# Add gene annotations

<p style="text-align: center">
    <video width="580" loop="loop" autoplay="autoplay">
        <source src="img/higlass-add-annotations.mp4" type="video/mp4">
    </video>
</p>

Adding gene annotations lets us verify that the RNAseq profile matches
the locations of the genes (http://higlass.io/app/?config=YKBKin5SRhWIjuWwwnwpYw).

---

# Running HiGlass locally

To look at private data, you have to run HiGlass locally. This is easily
accomplished using a docker container.

The following commands can be found on the higlass wiki (google: higlass wiki) or go to: https://github.com/hms-dbmi/higlass/wiki

```bash
docker stop higlass-container; 
docker rm higlass-container;

docker pull gehlenborglab/higlass:v0.2.6 


docker run --detach \
           --publish 8989:80 \
           --volume ~/hg-data:/data \
           --volume ~/tmp:/tmp \
           --name higlass-container \
           gehlenborglab/higlass:v0.2.6
```

---

# Importing data

We can import multi-res data into local instances of HiGlass. First, let's
get a multires file:

```bash
wget https://s3.amazonaws.com/pkerp/public/matrix.multi.cool
```

---

# Importing data

We can import multi-res data into local instances of HiGlass. First, let's
get a multires file:

```bash
wget https://s3.amazonaws.com/pkerp/public/matrix.multi.cool
```

To load it into HiGlass move it to `~/tmp`: 

```bash
cp matrix.multi.cool ~/tmp
```

---

# Importing data

We can import multi-res data into local instances of HiGlass. First, let's
get a multires file:

```bash
wget https://s3.amazonaws.com/pkerp/public/matrix.multi.cool
```

To load it into HiGlass move it to `~/tmp`: 

```bash
cp matrix.multi.cool ~/tmp
```

and run the following command:

```bash
docker exec higlass-container python higlass-server/manage.py \
  ingest_tileset \
  --filename /tmp/matrix.multi.cool \
  --datatype matrix \
  --filetype cooler 
```

This makes the file accessible to `higlass` and registers it with the list of
available data. 

---

# Displaying local data

<table>
    <tr>
        <td>
            <ul>
                <li style="margin-bottom: 20px">Navigate to <a href="http://localhost:8989">http://localhost:8989</a></li>
                <li style="margin-bottom: 20px">Click on the "Add track" button</li>
                <li style="margin-bottom: 20px">Search for "matrix"</li>
                <li style="margin-bottom: 20px">Double click on "matrix.multi.cool</li>
            </ul>
        </td>
        <td>
            <p style="text-align: center">
                <video width="500" loop="loop" autoplay="autoplay">
                    <source src="img/higlass-load-local.mp4" type="video/mp4">
                </video>
            </p>
        </td>
    </tr>
</table>

---


# Documentation:

* HiGlass Documentation
    * <a href="https://github.com/hms-dbmi/higlass/wiki">https://github.com/hms-dbmi/higlass/wiki</a>


* Examples
    * <a href="http://higlass.io/examples">http://higlass.io/examples</a>

---


    </textarea>
    <script src="https://remarkjs.com/downloads/remark-latest.min.js">
    </script>
    <script>
      var slideshow = remark.create();
    </script>
  </body>
</html>