Preprocessing read files from NCBI SRA

SRA (for Sequence Read Archive) is a NCBI binary format for short reads.

It's thoroughly described in the ​SRA Handbook

Processing SRA files requires the ​NCBI SRA Toolkit, which is installed on our systems.

The main command is fastq-dump <SRA archive file>, like

fastq-dump SRR060751.sra

If your reads are paired, by default the #1 and #2 reads will end up concatenated together in the same file. To get them into separate files, instead use a command like

fastq-dump --split-files SRR060751.sra

You can ask for gzipped output instead of typical fastq:

fastq-dump --gzip SRR060751.sra

See ​Converting SRA format data into FASTQ for all program options.

Note that a fastq file is about 4-5x larger than its corresponding SRA file.

fastq-dump can be used to download/fetch the SRA file, or you can download (eg. using wget) the SRA file directly and then run fastq-dump to get the fastq file. Downloading SRA file directly will avoid changing home dir path for large file (see below).

As of fastq-dump version 2.8.1, running fastq-dump will require the vdb-config to be set up correctly. By default, downloaded/cache file is copied to the user's home directory, which is likely to run out of space. Run vdb-config -i and use the GUI to enter a different location. Manually editing the file, $HOME/.ncbi/user-settings.mkfg, doesn't seem to work. See ​ NCBI SRA Installation/Config. Other alternatives: i) simply symlink the ncbi directory in your home directory to somewhere else with larger storage, or ii) download the SRA file directly (eg. using wget) before using fastq-dump.

#download SRR4090409.sra (e.g. use wget) from SRA and convert to fastq
fastq-dump SRR4090409.sra

#download SRA file via fastq-dump (important: home directory or vdb-config file must be set up correctly), and convert to fastq
fastq-dump SRR4090409

Mapping short reads

Regular mappers

These mapping tools are useful for reads of DNA origin that should map to a continuous stretch of genomic DNA. Some of these tools can tolerate short indels but they're not designed for reads that span a splice junction

One may choose between bowtie version 1 (faster but ignores indels) and bowtie version 2 (slower but performs gapped alignment (i.e., indels)). For a feature comparision, see ​How is Bowtie 2 different from Bowtie 1?

​bowtie version 1

Sample command:

bsub bowtie  -k 1 -n 2 -l 70 --best --sam --solexa1.3-quals /nfs/genomes/mouse_gp_jul_07_no_random/bowtie/mm9 s_7.txt s7_mm9.k1.n2.l36.best.sam

Parameters included in the sample command:

• -l/--seedlen <int> seed length for -n (default: 28) -- Set to longest possible length of high-quality bases. Use the FastQC output to determine length of high-quality positions.
• -n/--seedmms <int> max mismatches in seed (can be 0-3, default: -n 2)
• -k <int> report up to <int> good alignments per read (default: 1) -- If you want only uniquely mapped reads, however, also use '-m 1' to ignore multi-mapped reads
• --best (in the case of multi-mapped reads, keep only the best hit(s))
• --sam to get SAM output format (which is the best format for downstream analysis)
• --all report all alignments (much slower, ie. turn-off -k option)

Choices for fastq encoding (which is listed as "Encoding" in the top "Basic Statistics" table of the FastQC output file). See the ​FASTQ format page for more details.

• --solexa-quals (for input quality scores from Illumina versions 1.2 and earlier)
• --solexa1.3-quals or --phred64-quals (for input quality scores from Illumina versions 1.3-1.7)
• --phred33-quals (default "Sanger format"; for input quality scores from Illumina versions 1.8 and later)

To see other parameters log into tak and type bowtie

​bowtie version 2

Bowtie 2 was designed as an improvement to bowtie 1, specifically, it supports gapped alignment. See the first ​bowtie2 FAQ for how they differ. Early versions of bowtie 2 had some issues, but these seem to have been fixed. Bowtie 2 uses a different set of genome index files (*.bt2) than bowtie 1 (*.ebwt). Bowtie 2 works with indels.

Sample command:

bsub bowtie2 --phred64 -L 22 -N 1 -x /nfs/genomes/mouse_gp_jul_07_no_random/bowtie/mm9 -U s_7.txt -S s7_mm9.L22.N1.sam

The parameters included in the sample command:

• -L <int> length of seed substrings; must be >3 and <32 (default=22)
• -N <int> max # mismatches in seed alignment; can be 0 or 1 (default=0)
• -S name of SAM output file

Choices for fastq encoding (which is listed as "Encoding" in the top "Basic Statistics" table of the FastQC output file). See the ​FASTQ format page for more details.

• --solexa-quals (for input quality scores from Illumina versions 1.2 and earlier)
• --phred64 (for input quality scores from Illumina versions 1.3-1.7)
• --phred33 (default "Sanger format"; for input quality scores from Illumina versions 1.8 and later)

bowtie2 can also perform local alignments where the unaligned end(s) of a read are clipped (so, for example, remaining adapter won't prevent alignment) by adding the argument --local.

The bowtie2 command can be modified to output mapped reads as BAM, such as

bsub "bowtie2 -x /nfs/genomes/mouse_gp_jul_07_no_random/bowtie/mm9 -U s_7.txt | samtools view -bS - > s7_mm9.bam"

​bwa - Burrows-Wheeler Alignment Tool

Bwa is a software package containing several related algorithms using the Burrows-Wheeler Transform. It works well even with indels, but not with spliced (RNA) reads.

Sample commands for short (upto 100 bp) reads:

# Align single-end reads
bsub "bwa aln /nfs/genomes/mouse_gp_jul_07_no_random/bwa/mm9.fa Reads.fq > Mapped_reads.sai"
bsub "bwa samse /nfs/genomes/mouse_gp_jul_07_no_random/bwa/mm9.fa Mapped_reads.sai Reads.fq > Reads.bwa.sam"
# Align paired-end reads
bsub "bwa aln /nfs/genomes/mouse_gp_jul_07_no_random/bwa/mm9.fa Reads.1.fq > Mapped_reads.1.sai"
bsub "bwa aln /nfs/genomes/mouse_gp_jul_07_no_random/bwa/mm9.fa Reads.2.fq > Mapped_reads.2.sai"
bsub "bwa sampe /nfs/genomes/mouse_gp_jul_07_no_random/bwa/mm9.fa Mapped_reads.1.sai Mapped_reads.2.sai Reads.1.fq Reads.2.fq > Reads.bwa.sam"

Use "-n" option to change the maximum number of mismatches in the alignment.

Sample commands for long (70bp or longer) reads:

# Align single-end reads
bsub "bwa mem /nfs/genomes/sgd_2010/bwa/sacCer3.fa Reads.fq > Mapped_reads.bwa.sam"
# Align paired-end reads
bsub "bwa mem /nfs/genomes/sgd_2010/bwa/sacCer3.fa Reads.1.fq Reads.2.fq > Mapped_reads.bwa.sam"

For aligning long reads using the bwa mem option, there's no maximum number of mismatches, this is analogous to a local alignment using blat/blast.

Other tools Many other regular mapping tools are also available, although they generally require a tool-specific indexed version of the genome.

Splice-aware mappers

These mappers permit the beginning and end of a read to map to (originate from) different places in the genome, which is common for spliced RNA.

​tophat version 1

Running TopHat version 1 requires a change to a user's environment on tak and only applies to the specific tak session. First run this command:

export PATH="/usr/local/share/tophat1:$PATH"

and then check that your terminal will use the correct TopHat version:

tophat --version

Sample command:

bsub tophat -o s_7_tophat_out -p 6 --phred64-quals --segment-length 20 -I 200000 -G /nfs/genomes/mouse_gp_jul_07_no_random/gtf/Mus_musculus.NCBIM37.67_noNT.gtf --no-novel-juncs /nfs/genomes/mouse_gp_jul_07_no_random/bowtie/mm9 s_7.txt

The parameters included in the sample command are:

• -o/--output-dir <word> All output files will be created in this directory (default = tophat_out)
• --segment-length <int> Shortest length of a spliced read that can map to one side of the junction. For reads shorter than ~45 nt, set this to half the read length (so set '--segment-length 20' for 40-nt reads). For longer reads, the default length (25) can be used.
• -I <int> Maximum intron length. If your genome has introns that are all shorter (or many that are longer) than the default value (500000), set this to a more appropriate value.
• -G <GTF file> Supply bowtie with a GTF file of transcript models. This can help bowtie identify functions that may otherwise be missed.
• --no-novel-juncs Only look for spliced reads across junctions in the supplied GTF file. Not used if looking for novel isoforms.
• -p/--num-threads Use this many threads to align reads (default is 1)

Choices for fastq encoding (which is listed as "Encoding" in the top "Basic Statistics" table of the FastQC output file). See the ​FASTQ format page for more details.

• --solexa-quals (for input quality scores from Illumina versions 1.2 and earlier)
• --solexa1.3-quals or --phred64-quals (for input quality scores from Illumina versions 1.3-1.7)

​tophat version 2

TopHat version 2 uses bowtie2, rather than bowtie, for its mapping. As a result, TopHat 2 uses a different set of genome index files (*.bt2) than TopHat 1 (*.ebwt).

Sample command:

# Single-end reads
bsub tophat -o s_7_tophat_out --phred64-quals --library-type fr-firststrand --segment-length 20 -I 200000 -G /nfs/genomes/mouse_gp_jul_07_no_random/gtf/Mus_musculus.NCBIM37.67_noNT.gtf --no-novel-juncs /nfs/genomes/mouse_gp_jul_07_no_random/bowtie/mm9 s_7.txt
# Paired-end reads
# For PE reads, specifiy expected (mean) inner distance using -r option (default is 50bp)
bsub tophat -o s_7_tophat_out --phred64-quals --library-type fr-firststrand --segment-length 20 -I 200000 -G /nfs/genomes/mouse_gp_jul_07_no_random/gtf/Mus_musculus.NCBIM37.67_noNT.gtf --no-novel-juncs /nfs/genomes/mouse_gp_jul_07_no_random/bowtie/mm9 s_7.1.txt s_7.2.txt

The parameters included in the sample command are:

• -o/--output-dir <word> All output files will be created in this directory (default = tophat_out)
• --segment-length <int> Shortest length of a spliced read that can map to one side of the junction. For reads shorter than ~45 nt, set this to half the read length (so set '--segment-length 20' for 40-nt reads). For longer reads, the default length (25) can be used.
• -I <int> Maximum intron length. If your genome has introns that are all shorter (or many that are longer) than the default value (500000), set this to a more appropriate value.
• -G <GTF file> Supply bowtie with a GTF file of transcript models. This can help bowtie identify functions that may otherwise be missed.
• --no-novel-juncs Only look for spliced reads across junctions in the supplied GTF file. Not used if looking for novel isoforms.
• --library type Take advantage of strandedness of library for mapping (especially across splice junctions); can be fr-unstranded, fr-firststrand, or fr-secondstrand

Choices for fastq encoding (which is listed as "Encoding" in the top "Basic Statistics" table of the FastQC output file). See the ​FASTQ format page for more details.

• --solexa-quals (for input quality scores from Illumina versions 1.2 and earlier)
• --solexa1.3-quals or --phred64-quals (for input quality scores from Illumina versions 1.3-1.7)
• For "Sanger / Illumina 1.8" or "Sanger / Illumina 1.9", bowtie can use the default "phred33" encoding

Choices for controlling alignment (eg. mismatches)

• --read-mismatches/-N Final read alignments having more than these many mismatches are discarded (default is 2).
• --read-gap-length Final read alignments having more than these many total length of gaps are discarded (default is 2).
• --read-edit-dist Final read alignments having more than these many edit distance (ie. mismatches+indels) are discarded (default is 2).
• --segment-mismatches Read segments are mapped independently, allowing up to this many mismatches in each segment alignment (default is 2).

​STAR

STAR is an ultrafast universal RNA-seq aligner. It maps >60 times faster than Tophat2. To use STAR, a genome directory specific for the STAR mapper needs to be generated first. STAR tends to align more reads to pseudogenes compared to Tophat2. However, the pseudogene problem can be significantly minimized by providing an annotation file containing known splice junctions. If no annotation is available for a genome of interest, a 2-pass mapping procedure is recommended. The first pass generates a splice junctions file, which is then used as the annotation file to run the second pass mapping.

Sample command:

To generate genome index files for STAR:

bsub STAR --runMode genomeGenerate --genomeDir /path/to/GenomeDir --genomeFastaFiles /path/to/genome/fasta1 /path/to/genome/fasta2 --sjdbGTFfile /path/to/GTF/FileName.gtf --sjdbOverhang 39 --runThreadN 8 

To map:

bsub STAR --genomeDir /path/to/GenomeDir --readFilesIn /path/to/read1.fastq /path/to/read2.fastq --sjdbScore 2 --outFileNamePrefix whateverPrefix --runThreadN 8

The parameters included in the above sample commands are:

• --runMode <alignReads, genomeGenerate> "alignReads" does the actual mapping. "genomeGenerate" generates the genomeDir required for mapping (default = alignReads).
• --genomeDir </path/to/GenomeDir> Specifies the path to the directory used for storing the genome information created in the genomeGenerate step.
• --genomeFastaFiles <genome FASTA files> Specifies genome FASTA files to be used.
• --sjdbGTFfile <GTF_file.gtf> Supplies STAR with a GTF file during the genomeGenerate step. Combined with the --sjdbScore <n> option during mapping, this will bias the alignment toward annotated junctions, and reduces alignment to pseudogenes.
• --readFilesIn <read1.fastq read2.fastq> Specifies the fastq files containing the reads, can be single-end or paired-end.
• --sjdbScore <n> Provides extra alignment score for alignments that cross database junctions (default = 2). If this score is positive, it will bias the alignment toward annotated junctions. This is only used if during the genomeGenerate step a splice junction annotation file is used.
• --runThreadN <n> Specifies the number of threads to use.
• --sjdbOverhang Specifies the length of the genomic sequence around the annotated junction to be used in constructing the splice junctions database. For short reads (<50) use readLength - 1, otherwise a generic value of 100 will work as well (see manual for more info).