| 1 | | == Performing and reporting statistical tests == |
| 2 | | |
| 3 | | Doing a statistical analysis often involves several steps. The steps often depend on whether one is doing one test (comparing two vectors) or multiple tests (like differential expression analysis). Much of this is ideally discussed before the experiment is performed. In brief: |
| 4 | | |
| 5 | | 1. Understand the details of the experiment |
| 6 | | 2. Look at the raw data |
| 7 | | 3. Produce a draft figure showing the desired comparison |
| 8 | | 4. Determine the statistical test(s) |
| 9 | | 5. Organize the data |
| 10 | | 6. Perform the statistical test(s) |
| 11 | | 7. Do multiple hypothesis correction (if needed) |
| 12 | | 8. Interpret the statistical test |
| 13 | | |
| 14 | | In more detail: |
| 15 | | |
| 16 | | - '''Understand the details of the experiment''' |
| 17 | | 1. What is the big picture; what is begin studied? |
| 18 | | 2. What is the experimental design? Is there biological and/or technical replication? |
| 19 | | 3. What measurements were taken? |
| 20 | | 1. Is there any missing data? How should that be handled? |
| 21 | | 1. What specific question is the statistics hoping to address? This can be tricky, as slightly different questions may require completely different tests. |
| 22 | | |
| 23 | | - '''Look at the raw data''' |
| 24 | | 1. Do some kind of scatterplot to show all the data. |
| 25 | | 1. What does a global look at the data tell you? |
| 26 | | 1. Did the experiment work? |
| 27 | | 1. How are the values distributed? |
| 28 | | 1. Are there any obvious problems? |
| 29 | | 1. Are there obvious batch effects? |
| 30 | | 1. Are there any obvious outliers due to some technical reason? |
| 31 | | 1. Could the data need some form of normalization or scaling to make factors more comparable? |
| 32 | | |
| 33 | | - '''Produce a draft figure showing the desired comparison''' |
| 34 | | 1. What comparisons should be made? |
| 35 | | 1. Does it look to the eye that there's something going on? It may not be worth going any further is summary measures show nothing interesting. |
| 36 | | |
| 37 | | - '''Determine the statistical test(s)''' |
| 38 | | 1. What test best addresses the question the scientist is asking? |
| 39 | | 1. Is there a standard test for this method? |
| 40 | | 1. Does you data meet all assumptions required of the test? |
| 41 | | 1. Given the answers to the above question, the best test may be the simplest one. |
| 42 | | |
| 43 | | - '''Organize the data''' |
| 44 | | 1. Is the data in a format that can be used by the statistical software? |
| 45 | | 1. Does it make sense to scale, normalize, or transform (such as with logs) the data? |
| 46 | | |
| 47 | | - '''Perform the statistical test(s)''' |
| 48 | | 1. Run the analysis with a tool like R, Prism, or Excel |
| 49 | | 1. Capture all of the appropriate results and details of the test, often more than just the p-value |
| 50 | | 1. Does the software produce any warnings? If so, how should you react? |
| 51 | | |
| 52 | | - ''' Do multiple hypothesis correction''' |
| 53 | | 1. If more than one test is performed, you may need multiple hypothesis correction |
| 54 | | 1. Which is better: Bonferroni, FDR, or another test? |
| 55 | | 1. Be prepared to explain why this is necessary |
| 56 | | |
| 57 | | - ''' Interpret the statistical test(s)''' |
| 58 | | 1. What does the p-value and/or the confidence interval tell you? |
| 59 | | 1. Does the statistical model of the data effectively describe the data? How valid is the p-value? |
| 60 | | 1. Does the statistics imply that the result is biologically meaningful (Perhaps the hardest question)? |
| 61 | | |
| 62 | | Through all of this, resist any temptation to bias the analysis to produce the result one wants or expects. |
| 63 | | |
| 64 | | === Some journals explicitly state statistics guidelines === |
| 65 | | * Nature [http://www.nature.com/authors/policies/checklist.pdf Reporting Checklist], [http://www.nature.com/nature/authors/submissions/final/3d_MS_checklist.pdf Manuscript checklist], and [http://www.nature.com/authors/policies/reporting.pdf Reporting Life Sciences Research] |
| 66 | | * Journals of the American Society of Microbiology [http://iai.asm.org/site/misc/2014AprilIAIITA.pdf Infection and Immunity Instructions to Authors] (See "Statistics" on page 19) |
| 67 | | * Science Translational Medicine [http://stm.sciencemag.org/site/misc/auth_inst_ra.xhtml Instructions for Authors] (see "Experimental Design and Statistics Guidelines") |
| | 1 | Moved to http://barcwiki.wi.mit.edu/wiki/SOPs/StatisticalSteps |
