The Department of Homeland Security announced that it was discontinuing its color-coded security advisory system today. In the software security world we often try to have big dashboards with red, yellow, and green indicating important things about our software. This is a great example of where such dashboards fail.
If you look at the DHS chronology of changes to the advisory system, you can see that it came in at yellow and was mostly yellow for its first 4 years. In response to the threat posed by liquids on planes, the threat level was briefly raised to red, then it settled down at orange and DHS seems to have forgotten it. For more than 4 years it has just sat at orange—unloved, untended, unimportant.
This is typical of security dashboards. A zillion complexities are somehow squeezed into four ordinal color values. Complex qualities like geography, industry, mode of transportation, political affiliation, population density, and dozens of other factors contribute to how likely or unlikely any given target is on any given day. And somehow, all these zillions of factors were funneled into one big color that is wrong for most people.
It’s arguable that the threat level has been green, for example, for huge swaths of the American heartland. Or yellow for certain industries while orange—or even red—for others. It is not clear how anyone anywhere benefited from this dashboard during its 9-year tenure. The fact that it sat at one color for 4 years is just testimony to how impossible it is to decide which of the zillion things justify a bump up or down on so coarse a measure.
As we create security dashboards in software, or in business risk, or in functional testing, or anything else that is hard to measure but begging for metrics, let this be a cautionary tale about dialing the resolution down so far that it becomes meaningless.