Work on getting Mutect algorithm up and running #167

jstjohn · 2015-05-18T23:51:24Z

This PR is to track progress and collaborate on an implementation of the MuTect algorithm referenced in #114.

…at filter reads to have a consistent interface. Using this interface, I added filters to eliminate reads with high levels of soft clipping, and reads that are marked as duplicates.

…er string is set. Further, refactored depth filter to provide generalized attribute based genotype filter. Using generalized filter, added strand bias filter and multiallelic site filter.

…_mutect_fnothaft

…mutect_fnothaft

…version of adam/avocado

…ect_work

…tests which will be filled in

AmplabJenkins · 2015-05-18T23:55:01Z

Can one of the admins verify this patch?

… normal reads

fnothaft · 2015-05-20T18:55:41Z

Jenkins, add to whitelist and test this please.

AmplabJenkins · 2015-05-20T19:01:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/avocado-prb/99/
Test PASSed.

AmplabJenkins · 2015-05-20T19:06:30Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/avocado-prb/100/
Test PASSed.

fnothaft · 2015-05-25T23:28:29Z

avocado-core/src/main/scala/org/bdgenomics/avocado/algorithms/mutect/LikelihoodModel.scala

+ */
+object MfmModel extends LikelihoodModel {
+
+  def e(q: Int): Double = pow(10.0, -0.1 * q.toDouble)


I'd prefer to use https://github.com/bigdatagenomics/adam/blob/master/adam-core/src/main/scala/org/bdgenomics/adam/util/PhredUtils.scala#L32 which precomputes the phred-to-log conversion.

Weird, for some reason when I switch from val ei = e(obs.phred) to val ei = phredToSuccessProbability(obs.phred) my tests start failing! not sure what is going on there, your code looks correct at first glance. Leaving it as is for now.

Ah, it should have been phredToErrorProbability(obs.phred) just switched code to that.

…erly tested, and fixed bug in filters so they work now

…Observation object and parsed from read MD tags by ReadExplorer

…ct_work

Adding known-sites variant filter.

jstjohn · 2015-08-01T22:15:38Z

Woot! Super close. Ok so Frank, the only thing I need to change is the known sites filter. The thing is that this is not a binary keep/throw away kind of situation. Basically if the site is a known variant, then you have a different prior probability of the mutation being germline than if it is a site that had never been seen before.E See lines 167-169 of MutectGenotyper.scala for how this is used.

As you can see, right now I have the placeholder in my code val dbSNPsite = false. Ideally this would be the real value, or I could just do this filter later -- during the postprocessing step as you have implemented in your latest PR for example. So you don't necessarily throw out all sites that overlap dbSNP, you just have to have more evidence in the normal that the site is actually not just a heterozygous site -- the burden of proof is higher for the site being unique to the tumor.

One way to get this is to pass this information through with the variant, and decide which cutoff to use for the log odds score during the post processing step. If we go in this direction, what is the best place to stick this arbitrary bit of data, the normal log odds of being germline? Thanks for your time and thoughts on this @fnothaft!

fnothaft · 2015-08-02T01:09:05Z

@jstjohn oh, hah! Goofy mistake on my side. I think that is an easy change to make, actually! I will have a PR against this branch with the fix tomorrow.

jstjohn · 2015-08-02T02:44:56Z

Thanks Frank!!

On Sat, Aug 1, 2015 at 6:09 PM, Frank Austin Nothaft <
[email protected]> wrote:

@jstjohn https://github.com/jstjohn oh, hah! Goofy mistake on my side.
I think that is an easy change to make, actually! I will have a PR against
this branch with the fix tomorrow.

—
Reply to this email directly or view it on GitHub
#167 (comment)
.

…genotyper use broadcast.

Add dbsnp to mutect caller

…ring as discussed in the MuTect paper

…ct_work

AmplabJenkins · 2015-08-30T17:09:43Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/avocado-prb/133/
Test PASSed.

Updating from bigdatagenomics/avocado

Update ADAM to 0.18.2

…label reads. Runs with MuTect, but there are bugs within the program

AmplabJenkins · 2016-03-10T23:51:31Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/avocado-prb/137/
Test PASSed.

AmplabJenkins · 2016-03-11T00:21:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/avocado-prb/138/

Build result: FAILURE

GitHub pull request #167 of commit f7ad48b automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-02 (centos spark-test spark-compile) in workspace /home/jenkins/workspace/avocado-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/avocado.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/avocado.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/avocado.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/167/merge^{commit} # timeout=10 > git branch -a --contains 239bf46 # timeout=10 > git rev-parse remotes/origin/pr/167/merge^{commit} # timeout=10Checking out Revision 239bf46 (origin/pr/167/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 239bf462ff06fe881648bce0eb0c26fbe37e8618First time build. Skipping changelog.Triggering avocado-prb ? 2.3.0,centosTriggering avocado-prb ? 1.0.4,centosTriggering avocado-prb ? 2.2.0,centosavocado-prb ? 2.3.0,centos completed with result FAILUREavocado-prb ? 1.0.4,centos completed with result FAILUREavocado-prb ? 2.2.0,centos completed with result SUCCESS
Test FAILed.

tdanford and others added 17 commits April 25, 2015 10:07

Added base quality score based read filters.

c34d30c

Miscellaneous read filter cleanup. Refactored preprocessing stages th…

aa88c53

…at filter reads to have a consistent interface. Using this interface, I added filters to eliminate reads with high levels of soft clipping, and reads that are marked as duplicates.

Updates to depth filter. Genotypes are not discarded, rather the filt…

16da202

…er string is set. Further, refactored depth filter to provide generalized attribute based genotype filter. Using generalized filter, added strand bias filter and multiallelic site filter.

adding in @tdanford's mutect work, merging into current avocado

65c68dd

Merge branch 'master' into jsj_mutect_hack

cc840b9

Merge branch 'common-filters' of github.com:fnothaft/avocado into jsj…

10181ec

…_mutect_fnothaft

Merge branch 'master' of github.com:bigdatagenomics/avocado into jsj_…

3fc75b2

…mutect_fnothaft

restoring successful compile with in-progress mutect against lastest …

265b142

…version of adam/avocado

Merge branch 'common-filters' of github.com:fnothaft/avocado into mut…

fe6772c

…ect_work

pulling over MuTect LikelihoodModel, fixing some bugs, and making tests

310c924

initial pull over of Mutect.scala base function, and a base suite of …

571bb2a

…tests which will be filled in

more fixes and clarifications on the mutect likelihood model

c0503f6

adding in initial tests of likelihood given an actual mutation

5d96d8b

added in tests for mutant site

c8cb648

reorganizing tests slightly

5506148

added in a final test covering the normal somatic classification

f9fe9c2

Adding mutect tests and TODO comments to main code

aac84f9

jstjohn mentioned this pull request May 18, 2015

Add an implementation of Mutect #114

Open

jstjohn added 3 commits May 18, 2015 17:20

adding somatic classification code/tests into MuTect

cde7a00

making test reads slightly more realistic, assigned to both strands now

c32ef7f

adding a small mutect test for sites with no tumor coverage, but some…

fe85550

… normal reads

Merge branch 'master' into mutect_work

1a92281

fnothaft reviewed May 25, 2015
View reviewed changes

jstjohn and others added 12 commits June 2, 2015 11:44

passing mate rescue status through to AlleleObservation

31d1f00

adding in custom mutect strand bias method

b034e7b

fixing bug in MutectGenotyper tests where filters were not being prop…

b0ee22d

…erly tested, and fixed bug in filters so they work now

refactoring variables to camelCase

03aaaf8

faster O(log(n)) recursive search for minimal passing k value

607ebab

cleaning up code a bit

0f9931c

cleaning up variable names

ab9b0ca

Implementation of the sum of mismatching quality scores in the Allele…

cc088c9

…Observation object and parsed from read MD tags by ReadExplorer

Merge branch 'master' of github.com:bigdatagenomics/avocado into mute…

fb18208

…ct_work

Merge branch 'master' of github.com:bigdatagenomics/avocado into mute…

8d0c286

…ct_work

Adding known-sites variant filter.

88a5cba

Merge pull request #3 from fnothaft/add-dbsnp

d5de796

Adding known-sites variant filter.

fnothaft and others added 5 commits August 3, 2015 00:39

Added site detection to mutect genotyper. Also, made site filter and …

4ed4e93

…genotyper use broadcast.

Fixed MD tag bug caused by upstream change to MD tag code in ADAM.

00a89c1

Merge pull request #4 from fnothaft/add-dbsnp

7b5da97

Add dbsnp to mutect caller

adding in the required settings to enable Panel of Normal (PON) filte…

b501595

…ring as discussed in the MuTect paper

Merge branch 'master' of github.com:bigdatagenomics/avocado into mute…

ba1b236

…ct_work

stewSquared and others added 6 commits January 7, 2016 09:49

Merge pull request #2 from bigdatagenomics/master

5b313c8

Updating from bigdatagenomics/avocado

Merge branch 'master' into mutect_work

d3b16d1

Merge pull request #3 from bigdatagenomics/master

2444c9a

Update ADAM to 0.18.2

Merge branch 'master' into mutect_work

4819062

Added support for having 2 input ADAM reads file, with the option to …

9bb2980

…label reads. Runs with MuTect, but there are bugs within the program

implementing isabels changes

0b2955e

adding isabels changes

f7ad48b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Work on getting Mutect algorithm up and running #167

Work on getting Mutect algorithm up and running #167

jstjohn commented May 18, 2015

AmplabJenkins commented May 18, 2015

fnothaft commented May 20, 2015

AmplabJenkins commented May 20, 2015

AmplabJenkins commented May 20, 2015

fnothaft May 25, 2015

jstjohn May 26, 2015

jstjohn May 26, 2015

jstjohn commented Aug 1, 2015

fnothaft commented Aug 2, 2015

jstjohn commented Aug 2, 2015

AmplabJenkins commented Aug 30, 2015

AmplabJenkins commented Mar 10, 2016

AmplabJenkins commented Mar 11, 2016

Work on getting Mutect algorithm up and running #167

Are you sure you want to change the base?

Work on getting Mutect algorithm up and running #167

Conversation

jstjohn commented May 18, 2015

AmplabJenkins commented May 18, 2015

fnothaft commented May 20, 2015

AmplabJenkins commented May 20, 2015

AmplabJenkins commented May 20, 2015

fnothaft May 25, 2015

Choose a reason for hiding this comment

jstjohn May 26, 2015

Choose a reason for hiding this comment

jstjohn May 26, 2015

Choose a reason for hiding this comment

jstjohn commented Aug 1, 2015

fnothaft commented Aug 2, 2015

jstjohn commented Aug 2, 2015

AmplabJenkins commented Aug 30, 2015

AmplabJenkins commented Mar 10, 2016

AmplabJenkins commented Mar 11, 2016

Build result: FAILURE