21 January 2015

Predictions for 2015

One of the most pernicious problems in trying to understand the world is that human beings are really bad at judging how confident they should be in any given opinion.  One trick that's been going the rounds is a little game called self-calibration... make predictions about what you think will happen and how confident you are in each prediction.  Then you can graph the results.  With perfect knowledge, your answers should be clustered in a single point (100% accuracy, 100% confidence).  All fallible humans, though, should have answers in a rough line from the lower-left to the upper-right.  The more perfect the line, the better-calibrated you are: you don't have perfect knowledge, but you're pretty accurate in knowing the limits of your own knowledge.

In other words, any predictions you make with 10% confidence should come true 10% of the time, while any predictions you make with 90% confidence should come true nine out of ten times.  If half of the predictions you make with 50% confidence come true, you're well-calibrated.

Gwern points us to a Kant quote on the matter:
The usual touchstone of whether what someone asserts is mere persuasion or at least a subjective conviction, i.e., firm belief, is betting. Often someone pronounces his propositions with such confident and inflexible defiance that he seems to have entirely laid aside all concern for error. A bet disconcerts him. Sometimes he reveals that he is persuaded enough for one ducat but not for ten. For he would happily bet one, but at ten he suddenly becomes aware of what he had not previously noticed, namely that it is quite possible that he has erred.

I've been doing this privately for a while, in a less mathy way.  For example, I wrote this the week before the last government shutdown:
After the House passes a CR tied to the Vitter thing (which is basically fuck you, staffers) then the Senate will again reject it, going into shutdown (high confidence). There will not be a clean CR or a temporary extension before this time (high confidence). The shutdown will continue for an uncertain number of days, based on the degree to which the public loudly blames the GOP; with low confidence, I would suggest it would be reasonable to see a four-day shutdown and a ten-point drop in generic GOP polls (along with a five-point drop in Obama). However, I would not be surprised to see a number of possibilities within a week's span. A shutdown longer than a week would be possible, but mildly surprising.
Two scenarios seem likely to end the shutdown: a clean CR as the GOP caves completely in the face of public scorn, and some sort of mild concession that's served up as face-saving. I would be surprised if something else ended the shutdown (low-medium confidence), although there might be some sort of legislative trick that I don't know, since my knowledge of the intricacies of both congressional procedure and governmental budgeting is very limited, decreasing my confidence in this prediction.
Verdict: I did fairly well. The government shut down (easy to predict) and the GOP polling fell off a cliff[1], although Obama's approval held steady, rather than dropping. The end happened pretty much exactly as I thought, as well: a huge GOP collapse and surrender, with perhaps the mildest of fig leaves to hide behind in the form of the slightly stricter scrutiny of income for Obamacare recipients, although no one is under any illusions about who just massively lost here.
This post will be similar, but a little more empirical.  I'm going to rip off the whole graphing practice from the rationalist blogsphere (like current favorite Slate Star Codex), and be more precise than "high" or "low."  There's a whole site for this called PredictionBook, actually.  Surprisingly, overconfidence when wrong is not as much of a problem as underconfidence when correct, so that's something I'll be watching for.  I'll be thinking at all times: if I was betting money on this, what odds would I accept?  As Alex Tabarrok has said, "A bet is a tax on bullshit."

Italics have been fulfilled, red is wrong, and unaltered is inconclusive as yet.

One year from now, on 1/21/16...

President Obama will wear a cornflower blue tie to his SOTU.  10%
Obama's proposed tax plan/community college plan will not pass into law in any recognizable form.  90%
Comprehensive immigration reform will not pass into law.  80%
There will be a terrorist attack in America that kills more than 5 people. 95%
There will be a terrorist attack in America that kills more than 20 people.  90%
There will be a terrorist attack in America that kills more than 100 people.  5%
Ebola will not infect and kill more than 10 people in America.  95%
There will be an opening on the Supreme Court through retirement, disability, or death.  50%
Gitmo will still be open and have at least 100 prisoners still in residence.  40%
Gitmo will still be open and have at least one prisoner in residence.  95%
Bitcoin (specifically the current implementation of cryptocurrency known as Bitcoin) will be under $100.  65%
Bitcoin will be under $50.  35%
I will still be in my current job.  95%
My team will have gotten quiz champion at a pub quiz.  90%