tag:blogger.com,1999:blog-1871542942842750523.post8449379335447335164..comments2023-07-24T10:40:57.739-04:00Comments on dechronization: When MrBayes Fails...Glorhttp://www.blogger.com/profile/17707197225963721646noreply@blogger.comBlogger15125tag:blogger.com,1999:blog-1871542942842750523.post-68497278271540448562009-04-25T14:39:00.000-04:002009-04-25T14:39:00.000-04:00For those who remain interested in this thread, Jo...For those who remain interested in this thread, John Harshman (one of the authors of the Hackett et al. paper) has <A HREF="https://www.blogger.com/comment.g?blogID=1871542942842750523&postID=7201858613022811049" REL="nofollow"> weighed in </A> with some thoughts in the comments to another post that involved this paper.Glorhttps://www.blogger.com/profile/17707197225963721646noreply@blogger.comtag:blogger.com,1999:blog-1871542942842750523.post-87070096589137692612009-04-10T15:34:00.000-04:002009-04-10T15:34:00.000-04:00First, I sympathize with the suspicion about wheth...First, I sympathize with the suspicion about whether MrBayes is converging. Current Bayesian tree proposals are somewhat "stupid" in the sense that they are data-agnostic, and also may not update branch lengths to match proposed topologies. This is easier to do in an ML setting.<BR/><BR/>That said, the obvious question here is "If MrBayes fails, how do we know if the other methods succeed." Does anyone know if the ML or MP searches that the authors used succeeded? (e.g. did multiple independent runs give the same answer?)<BR/><BR/>Clearly, it is not great practice to say "We checked this method, and it failed, so we used another method which we didn't check." :-)<BR/><BR/>-BenRIBenRIhttps://www.blogger.com/profile/00143159769881278026noreply@blogger.comtag:blogger.com,1999:blog-1871542942842750523.post-7169827139447997982009-04-10T10:54:00.000-04:002009-04-10T10:54:00.000-04:00I agree with with Stephen (blackrim) comment, most...I agree with with Stephen (blackrim) comment, most people change the method because their speed instead of their philosophy of each approach (beyond the "archaic" nature of the approach, :P)Salvahttps://www.blogger.com/profile/01062764779798191688noreply@blogger.comtag:blogger.com,1999:blog-1871542942842750523.post-90907601815426047942009-04-09T00:58:00.000-04:002009-04-09T00:58:00.000-04:00I've worked a little bit with similar datasets...I've worked a little bit with similar datasets, and seen similar problems. It seems that MrBayes often doesn't find good topologies in reasonable time for large (>100?) numbers of taxa, even with lots of chains etc.<BR/><BR/>One way around this is to start the run with a ML topology (e.g. from RaxML or Garli) with a small number of perturbations (e.g. nperts=5 in MrBayes). As long as the MCMC is behaving, this should still be a valid way to sample the posterior. If combining data from multiple runs, you'll need to leave enough burnin to make sure that the start points are as independent as possible (you can see this happening by tracking the stdev of split frequencies, which starts off rising as runs diverge, then starts dropping again).<BR/><BR/>Any thoughts?Rob Lanfearhttps://www.blogger.com/profile/16032743102416057563noreply@blogger.comtag:blogger.com,1999:blog-1871542942842750523.post-22619854537790118072009-04-07T21:46:00.000-04:002009-04-07T21:46:00.000-04:00Good stuff, y'all. Beware that Dr. Moore is prepa...Good stuff, y'all. Beware that Dr. Moore is preparing a response that is tentatively titled "Attention Phylo-whores"!Glorhttps://www.blogger.com/profile/17707197225963721646noreply@blogger.comtag:blogger.com,1999:blog-1871542942842750523.post-18709553400935438042009-04-06T17:58:00.000-04:002009-04-06T17:58:00.000-04:00It may be common practice now to do multiple indep...It may be common practice now to do multiple independent runs, but it sure wasn't the norm back then.Dan Warrenhttps://www.blogger.com/profile/07528161395964087899noreply@blogger.comtag:blogger.com,1999:blog-1871542942842750523.post-32246723092081207922009-04-06T14:21:00.000-04:002009-04-06T14:21:00.000-04:00Shouldn't we expect to see this problem with large...Shouldn't we expect to see this problem with large, multigene (=more paramters) datasets with star-trees?<BR/><BR/>Isn't it possible that the figure is showing different parameter estimates for a similar set of trees? Put another way - is it possible their analyses found the same topologies, but the large number of parameters complicates the analysis?<BR/><BR/>As far as the scary story goes - I thought it was widely recommended to do at least 4 independent runs for any dataset?DanShttps://www.blogger.com/profile/16293290766231344611noreply@blogger.comtag:blogger.com,1999:blog-1871542942842750523.post-85426567614140052232009-04-06T09:59:00.000-04:002009-04-06T09:59:00.000-04:00Nice to see this discussion. Another horror story...Nice to see this discussion. Another horror story, the perpetrators will remain anonymous. For some time a ~14 gene analysis for several hundred taxa has resulted in 100% posterior probability of a (very) novel placement for a taxon of note. This result was reported at various meetings etc. etc. Turns out for the taxon in question a single gene (1/14) was in fact a contaminant that wasn't caught. Now this might not have been a problem specific to Bayesian analyses, but it's definitely scary (suggesting a tiny amount of error can swamp analysis).<BR/><BR/>blackrim- How are you measuring performance? <BR/><BR/>barb- Easy now, parsimony definitely has it's place. Among other things you're doing nothing fast without it for the fast ML methods on large datasets (these generate a parsimony tree as a starting point). It's also the only method that can handle truly massive datasets, witness the 70k terminal analysis done by Goloboff's team presented at the last Cladistics meeting.Mattnoreply@blogger.comtag:blogger.com,1999:blog-1871542942842750523.post-76098755228542316752009-04-06T01:23:00.000-04:002009-04-06T01:23:00.000-04:00This comment has been removed by the author.Dan Warrenhttps://www.blogger.com/profile/07528161395964087899noreply@blogger.comtag:blogger.com,1999:blog-1871542942842750523.post-35182092172705620312009-04-06T00:52:00.000-04:002009-04-06T00:52:00.000-04:00Back when we were first testing AWTY's predecessor...Back when we were first testing AWTY's predecessor (Converge) on a bunch of different data sets, we saw this sort of thing happen with posteriors in some of the larger data sets. The most egregious offender was only ~85 taxa, and we found things that looked like this going out to 40 million generations or more. Worse yet, even after the posteriors for the separate runs flattened out, we were left with substantial disagreement between runs on the PPs for many clades. We did ten separate runs of over 50 million generations each, and the results split perfectly into two sets of five runs, each of which was internally consistent but had substantial disagreement with all of the runs from the other set. That's about the worst news you could get out of MCMC, because it means that if you just did two runs there's a 50% chance you'd get results that no longer showed radically changing posteriors and that were consistent between runs. Most people would call that a good result, and yet the truth was that the chains were nowhere near convergence.<BR/><BR/>That was our poster child for misbehaving data, though, and we had other data sets with more species that were nowhere near that bad. Just to check out what was going on we did a bunch of random addition sequences followed by likelihood searches and found that there were just a buttload of local optima, and that the top two were not that different in likelihood. Clearly we had chains getting stuck on each of those optima for a long time.<BR/><BR/>I feel like I should be telling this around a campfire with a flashlight under my chin because it's basically the equivalent of a spoooooky story for Bayesian phylogenetics people.Dan Warrenhttps://www.blogger.com/profile/07528161395964087899noreply@blogger.comtag:blogger.com,1999:blog-1871542942842750523.post-59748819414322405092009-04-05T12:25:00.000-04:002009-04-05T12:25:00.000-04:00I think people have shifted to fast ML methods for...I think people have shifted to fast ML methods for large datasets because of practical reasons (simply impossible to run extremely large datasets on MrBayes). In fact, I think many people switched to Bayesian methods years ago for practical reasons as well (getting PP's was much faster than PAUP bootstraps), ignoring or adjusting to the philosophical differences in the approaches. <BR/><BR/>In my experience, RAxML especially, is ready for wide use. I have found very good performance. But the larger question users should be asking is whether they are looking for the ML tree with attempts at the confidence interval (using some technique) or are users looking for samples from the posterior distribution of trees. Important questions and consequences that require users to not treat these as blackboxes.<BR/><BR/>Interesting stuff.<BR/><BR/><A HREF="http://blackrim.net/semaphoront" REL="nofollow">blog link</A>Anonymoushttps://www.blogger.com/profile/05680952247968724544noreply@blogger.comtag:blogger.com,1999:blog-1871542942842750523.post-74100315481530309012009-04-05T09:05:00.000-04:002009-04-05T09:05:00.000-04:00Oh my goodness - I think I might need to print out...Oh my goodness - I think I might need to print out Barb's comment and post it to my door.<BR/><BR/>Good post, Rich - I feel like I'm constantly making comments about the number of generations run when I'm reviewing manuscripts and clearly the issue of how long runs need to be and what the potential for reaching true stationarity is has not been adequately explored yet.Susan Perkinshttps://www.blogger.com/profile/05944116263349266952noreply@blogger.comtag:blogger.com,1999:blog-1871542942842750523.post-57017092010616043622009-04-04T18:31:00.000-04:002009-04-04T18:31:00.000-04:00Hackett et al.'s solution was to revert back to pa...Hackett et al.'s solution was to revert back to parsimony??!?! That seems a bit archaic.Unknownhttps://www.blogger.com/profile/15880630848584404638noreply@blogger.comtag:blogger.com,1999:blog-1871542942842750523.post-48758156650140764832009-04-04T08:04:00.000-04:002009-04-04T08:04:00.000-04:00Interesting post, Rich; and Todd, thanks for point...Interesting post, Rich; and Todd, thanks for pointing out the Yang paper - I missed that. <BR/><BR/>The issue with stationarity of likelihoods just scratches the surface of the "convergence" issue. I've analyzed datasets that very quickly reached apparent stationarity, and for which independent runs reached the same stable plateau. But in fact, within individual runs, shifts between islands in tree space occurred approximately every 30-50 million generations. This suggests that a valid sample from the posterior would probably require billions of generations.<BR/>This would be very difficult to detect without tools like <A HREF="http://king2.scs.fsu.edu/CEBProjects/awty/awty_start.php" REL="nofollow">AWTY<BR/></A>, of which I am a huge fan. <BR/><BR/>As an aside, I haven't found TRACERs diagnostics as useful as AWTY, because AWTY explictly looks at the one parameter we care about - topology - whereas TRACER looks at all the molecular evolutionary parameters, and a valid sample of those does not necessarily imply a valid sample of trees.Dan Raboskyhttps://www.blogger.com/profile/10771030521328260748noreply@blogger.comtag:blogger.com,1999:blog-1871542942842750523.post-67369538901155604392009-04-03T22:12:00.000-04:002009-04-03T22:12:00.000-04:00Also troubling for me with respect to Bayesian ana...Also troubling for me with respect to Bayesian analyses is Ziheng Yang's 2008 paper:<BR/>Phil. Trans. R. Soc. B (2008) 363, 4031–4039<BR/>Radically different trees with PP of 1 on almost every node result depending on the model selected for large data sets in whales.<BR/>Although he doesn't see the problems as general for Bayesian analysis, if really big data sets drastically differ in trees produced depending on assumed models and the only way to fix it is to arbitrarily fiddle with priors, then likelihood trees seem like a better alternative.<BR/><BR/>From the discussion:<BR/>"In both the small and large datasets, the data size-<BR/>dependent prior almost always led to reduced PPs for<BR/>the MAP trees. It thus appears useful in reducing the<BR/>apparent conflicts in Bayesian phylogenetic analysis.<BR/>The implementation here involves some arbitrariness."Todd Jackmanhttps://www.blogger.com/profile/02075021843436668482noreply@blogger.com