Changes between Version 2 and Version 3 of SOPs/qc_SRA

Timestamp:

08/25/20 11:36:47 (4 years ago)

Author:

gbell

Comment:

--

SOPs/qc_SRA

@@ -9 +9 @@
 Processing SRA files requires the [[https://ncbi.github.io/sra-tools/|NCBI SRA Toolkit]], which is installed on our systems.
 
-The main command is **fastq-dump <SRA archive file>**, like
+=== Downloading and processing one NCBI SRA sample at a time ===
 
-''**fastq-dump SRR060751.sra**''
+NCBI short-read files can be downloaded
+  * in SRA format for subsequent conversion, or 
+  * and converted to fastq.gz format at the same time (recommended)
 
-If your reads are paired, by default the #1 and #2 reads will end up concatenated together in the same file.  
-To check if the SRA sample has paired reads or not, go to the [https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=run_browser SRA Run browser], enter the sample ID, and look in the table under "Layout".
-
-To get matched paired reads into separate files, use a command like
-
-''**fastq-dump --split-3 SRR060751.sra**''
-
-This works the same as using the "--split-files", but "--split-3" puts unpaired reads (if any) into a third file.
-
-You can ask also for gzipped output instead of typical fastq:
-
-''**fastq-dump --split-3 --gzip SRR060751.sra**''
-
-See [[https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=toolkit_doc&f=fastq-dump|Converting SRA format data into FASTQ]] for all program options.
-
-Note that a fastq file is about 4-5x larger than its corresponding SRA file.
-
-fastq-dump can be used to download/fetch the SRA file, or you can download (eg. using wget) the SRA file directly and then run fastq-dump to get the fastq file.  Downloading SRA file directly will avoid changing home dir path for large file (see below).
-
-'''Note:''' As of fastq-dump version 2.8.1, running fastq-dump will require the vdb-config to be set up correctly.  By default, downloaded/cache file is copied to the user's home directory, which is likely to run out of space.  Run,
+To download one SRR ID at a time to get fastq.gz format, use the command fastq-dump, like
 
 {{{
-vdb-config --restore-defaults
-vdb-config -i #use the GUI to enter a different location.  
+fastq-dump --split-3 --gzip SRR123456
 }}}
 
-Manually editing the file, $HOME/.ncbi/user-settings.mkfg, doesn't seem to work.  See [[https://ncbi.github.io/sra-tools/install_config.html | NCBI SRA Installation/Config]].  Other alternatives: i) simply symlink the NCBI directory in your home directory to somewhere else with larger storage, or ii) download the SRA file directly (eg. using wget) before using fastq-dump.
+With the option "--split-3",
+  * single-end reads will end up in a single file, named SRR123456.fastq.gz
+  * paired-end reads will produce two files (named SRR123456_1.fastq.gz and SRR123456_2.fastq.g)
+  * unpaired reads (if any) will be placed into a third file.
+
+See the [[https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=toolkit_doc&f=fastq-dump|fastq-dump documentation]] for all program options.
+
+We recommend always gzipping fastq files because
+  * fastq.gz files are much smaller than fastq files
+  * our typically-used analysis programs all permit fastq.gz input
+
+
+'''Note:''' Running fastq-dump places downloaded or cache files into the user's home directory, which is likely to run out of space.  To prevent this, you have at least 3 options:
+
+Option 1: symlink the NCBI directory in your home directory to somewhere else with larger storage, such as with a command like
 
 {{{
-#download SRR4090409.sra (e.g. use wget) from SRA and convert to fastq
-fastq-dump SRR4090409.sra
-
-#download SRA file via fastq-dump (important: home directory or vdb-config file must be set up correctly), and convert to fastq
-fastq-dump SRR4090409
+ln -s /path/to/large/storage ~/ncbi
 }}}
 
-In order to '''download a list of SRA files''' from NCBI, it is convenient to use prefetch. 
+Option 2: edit your ~/.ncbi/user-settings.mkfg file to include the following line:
+{{{
+/repository/user/default-path = "/path/to/large/storage"
+}}}
 
-As mentioned in [[https://www.ncbi.nlm.nih.gov/sra/docs/sradownload/| SRA website ]], you can download list of Run accessions from search results page ([[https://www.ncbi.nlm.nih.gov/sra/?term=cancer |- Example offsite image]]) - select Runs of interest by clicking on the checkboxes, click on "Send To", "file", and select "Accession List" in the drop-down menu.
+Option 3: Modify your environment with vdb-config
+{{{
+vdb-config --restore-defaults # To restore your settings
+vdb-config -i # To use the GUI to enter a different location with "Set Default Import Path".  
+}}}
+
+=== Downloading and processing multiple NCBI SRA samples ===
+
+To '''download a list of SRR files''' (such as for all of the samples of a data series) from NCBI, use prefetch.
 
 Given a set of SRA files listed in a single column in the text file "SraAccList.txt" (e.g. SRR7623010, SRR7623011, etc.), the following command will download the entire set:
 
 {{{
-prefetch --option-file sraAccList.txt
+prefetch -O output_directory sraAccList.txt
 }}}
 
-This is current as of prefetch v. 2.9.3 (2.9.3-1).  Note that the default location for downloaded files is in your home directory under ~/ncbi/ncbi_public/sra.  With this default, one can quickly run out of space.  One solution to address this problem is to edit your ~/.ncbi/user-settings.mkfg file to include the following line:
-{{{
-/repository/user/main/public/root = "/destination/for/big/storage/here"
-}}}
+If you don't specify an output directory, the SRR files will be downloaded to ~/ncbi/ncbi_public/sra (or your configured "Import Path" as described above).  
 
-or point to the destination folder with -O 
+To get this list of SRR IDs, go the [[https://www.ncbi.nlm.nih.gov/Traces/study/|SRA Run Selector]] and enter a project accession.  Once on a [[https://www.ncbi.nlm.nih.gov/Traces/study/?acc=SRP000002|project page, go to the "Select" section and click on "Accession List" (or if you want a subset of these, click on "Metadata" in the "Select" section to get a comma separated file, SraRunTable.txt)
 
-
-'''Download Metadata:'''
-After searching in [[ https://www.ncbi.nlm.nih.gov/Traces/study/?| SRA Run Selector ]] -> click on "Metadata" in the middle of the page in "Select" section -> This will generate a comma separated file SraRunTable.txt.