HCOL 196, March 25, 2011

Today we discussed problem #2 of the last problem set. The calculations you did were mostly fine, but a lot can be learned by looking at this problem from an algebraic rather than a numerical point of view. So for the fair die, the numerical spreadsheet is above in the picture, and the algebraic one is below. Since there is only one value of p, the marginal P(D|F) is equal to the only joint entry, P(p,D|F).

For the biased die, the corresponding spreadsheet is as below:

For this problem, it is essential that the priors add up to 1; that is, you cannot use the shortcut of setting each prior probability to 1, knowing that you’ll divide out any constant when you compute the posterior. The reason is that for this problem, we stop with the marginal distributions and never carry out that last division. That means, that to get the correct marginal P(D|B), you must have the P(pi|B) for i=1, 2,…, 10 add up to 1.

Once you have the two marginals, a third spreadsheet combines them to compute the posterior probabilities:

The formula for the posterior probability of “F” is shown below (you can get this directly from the spreadsheet above):

We also have the relationship between the posterior odds and the posterior probability (above). In the case where there are just two states of nature, you can directly compute the posterior odds as the product of the prior odds and the “Bayes factor,” which is the ratio of the two likelihoods. This is sometimes convenient.

I pointed out that P(D|B) is an approximation to an integral, and the approximation gets better and better, the more rows you have in your spreadsheet. The integral is shown below.

I then turned to a parapsychology experiment that I once analyzed. Some researchers had published a paper in which they claimed that people could control a random event generator just by thinking about it. I found this very implausible. The experimental setup is shown in the figure:

The box has a red and a green lamp, and one or the other will light up at random, driven by electronics that measure random radioactive decays in a sample, with a 50% chance of red and a 50% chance of green. The subject (usually a student) tries to make the lamp come up one color (say red) more times than the other. A computer records the results automatically.

The experiment was run for many years with many subjects. Over 100 million random events were counted, with a slight excess in the intended direction. The p-value (two-sided) was very, very small, and would be considered highly statistically significant in the standard statistics world.

However, my Bayesian analysis is completely at odds with this result. The same data, using a method similar to the dice experiment we discussed in the problem, yields a Bayes factor of 12, which supports the null hypothesis that there is nothing funny going on. The data make me more confident that the subjects cannot affect the random event generating device than before I looked at these data.

The British statistician, Dennis Lindley, pointed out many years ago that this sort of thing can happen in theory. That is, a significance test can strongly reject the null hypothesis at the same time that the same data would support the null hypothesis when looked at from a Bayesian point of view. From my point of view, since I trust the Bayesian methodology, this is reason to be very wary of some aspects of standard statistics, in particular, p-values and hypothesis testing.

Finally, I mentioned that the prior probability still needs to be estimated. In a case like this, one way of assigning a prior probability is to imagine how much data you would require to change a skeptical point of view into one that was at least neutral towards the alternative hypothesis (that the student can indeed control the equipment). In my case, I would require that the student start the device with the intention to have 100 successive “red” flashes, for example, and actually obtain that outcome. That would be pretty impressive. The probability of that happening is 1/2100=1/1030, which means that my prior odds in favor of the null hypothesis would be approximately 1030, and my posterior odds about 12×1030.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: