I was thinking about this when Scott's article came out. I used to be a lifeguard. Most of my job consisted of watching a pool of people where no one drowned.
In some ways it gets far worse odds than the volcanologist problem. How often you need to check makes a difference. As a lifeguard those checking intervals are every 15-30 seconds. How often do the volcano need to be checked? Once a day? Once a week?
To some extent I was definitely serving the same purpose as a security guard, to make people feel safe, to lower insurance costs, etc. But people did occasionally start drowning, and I did have to jump in and save them.
I did some back of the envelope math based on how often I saved people and how often I had to check the pool. The odds were much lower than a 1 in a 1,000 chance that someone was drowning. It was more like a 1 in a 100,000 chance that someone was drowning while I was checking the pool. I was a teenager while I was a lifeguard, many other lifeguards I worked with were also teenagers. No one drowned at any of the pools I worked at, and we made a few saves each summer.
I can't help but think that being right 99.9% of the time when being wrong is catastrophic is actually a really crappy record. If I had been a lifeguard that was only right 99.999% of the time, there would have been at least one dead kid.
Football Outsiders has long has the problem of making the best case that their pre-season predictions are useful. A lot of simpleminded analysis says that they are less accurate than just naively predicting 8-8 for every team. But of course, while 8-8 across the board will have a lower value on certain metrics of error... its useless. Its way more useful as they use it- to headline a chapter in their seasonal prospective and then go on to explain *why* their metrics made a prediction. Counter-intuitive predictions especially are useful, because it lets the point out that, say, a good defense with great 3rd down stop rates and fumble recovery #'s is going to regress to the mean more than a bad offense with good completion percentages, bad red-zone numbers and a lot of close losses because offense and completion percentage are very stable year-to-year while the other metrics are very unstable but very invluential on wins and losses.
I was thinking about this when Scott's article came out. I used to be a lifeguard. Most of my job consisted of watching a pool of people where no one drowned.
In some ways it gets far worse odds than the volcanologist problem. How often you need to check makes a difference. As a lifeguard those checking intervals are every 15-30 seconds. How often do the volcano need to be checked? Once a day? Once a week?
To some extent I was definitely serving the same purpose as a security guard, to make people feel safe, to lower insurance costs, etc. But people did occasionally start drowning, and I did have to jump in and save them.
I did some back of the envelope math based on how often I saved people and how often I had to check the pool. The odds were much lower than a 1 in a 1,000 chance that someone was drowning. It was more like a 1 in a 100,000 chance that someone was drowning while I was checking the pool. I was a teenager while I was a lifeguard, many other lifeguards I worked with were also teenagers. No one drowned at any of the pools I worked at, and we made a few saves each summer.
I can't help but think that being right 99.9% of the time when being wrong is catastrophic is actually a really crappy record. If I had been a lifeguard that was only right 99.999% of the time, there would have been at least one dead kid.
Football Outsiders has long has the problem of making the best case that their pre-season predictions are useful. A lot of simpleminded analysis says that they are less accurate than just naively predicting 8-8 for every team. But of course, while 8-8 across the board will have a lower value on certain metrics of error... its useless. Its way more useful as they use it- to headline a chapter in their seasonal prospective and then go on to explain *why* their metrics made a prediction. Counter-intuitive predictions especially are useful, because it lets the point out that, say, a good defense with great 3rd down stop rates and fumble recovery #'s is going to regress to the mean more than a bad offense with good completion percentages, bad red-zone numbers and a lot of close losses because offense and completion percentage are very stable year-to-year while the other metrics are very unstable but very invluential on wins and losses.
Thanks for mentioning The Phantom Tollbooth.
This is reminding me quite a bit of the Euthyphro Dilemma