The annual scoring exercise. The 2007 predictions get explicit review; new predictions for 2008 are made.
This is a longer post because the calibration discipline is structurally important and the cumulative archive across years is now substantial.
2007 predictions, scored
The explicit predictions made for 2007:
1. Continue weekly cadence. 95%. Resolved AFFIRMATIVE.
2. Complete and publish the DDoS book. 60%. Resolved AFFIRMATIVE.
3. Continue the CISO role at Gala Coral productively. 90%. Resolved AFFIRMATIVE.
4. Attend at least four conferences. 80%. Resolved AFFIRMATIVE.
5. Speak at at least one conference. 70%. Resolved AFFIRMATIVE.
6. Substantial piece on internal segmentation. 55%. Resolved PARTIAL.
7. Major Vista-targeting malware family. 60%. Resolved PARTIAL — specific Vista-aware malware emerged; nothing dramatically targeting.
8. Mobile-platform malware incident. 55%. Resolved PARTIAL — specific iPhone research; no operational incident.
9. Continued DDoS-for-hire growth. 90%. Resolved AFFIRMATIVE.
10. UK consumer-impact data breach. 80%. Resolved AFFIRMATIVE — TJX (TK Maxx in UK) and others.
11. Major UK gambling-operator attack. 85%. Resolved AFFIRMATIVE.
12. Continued web-application worm activity. 80%. Resolved AFFIRMATIVE.
13. Vista SP1 ships. 80%. Resolved PARTIAL — beta cycle through 2007; expected early 2008.
The cumulative score: 9 affirmatives, 4 partials, 0 misses. Calibration is reasonable.
Calibration assessment
Specific patterns from 2007:
Direction calls were reliable. All predicted directions resolved as predicted.
Magnitude calls were generally appropriate.
Timing was approximately right. Vista SP1 expected early 2008 rather than late 2007 — bounded slip.
The cumulative calibration archive across multiple years suggests I am reasonably calibrated.
2008 predictions
For the year ahead, with explicit probabilities and deadlines.
Threat-side
1. Continued mass-mailing at sustained volume. 95%, 31 December 2008. No structural change visible.
2. At least one major Vista-targeting malware family. 70%, 31 December 2008. Vista deployment grows; targeting follows.
3. A meaningful mobile-platform malware incident. 60%, 31 December 2008. iPhone exposure grows; specific research will operationalise.
4. Continued DDoS-for-hire growth and at least one substantial public incident. 85%, 31 December 2008. The trajectory continues.
5. A specific high-profile data breach involving substantial UK consumer impact. 85%, 31 December 2008. The disclosure trajectory continues; specific incidents follow.
6. A specific major DNS-protocol vulnerability disclosed. 70%, 31 December 2008. Kaminsky's preliminary work suggests substantial subsequent disclosure.
7. Continued web-application worm activity. 80%, 31 December 2008. The category trajectory continues.
8. Specific politically-motivated DDoS following Estonia pattern. 75%, 31 December 2008. The Estonia precedent will be repeated.
Defensive-side
9. Vista SP1 ships. 90%, 30 June 2008. The cumulative trajectory is consistent.
10. Continued Microsoft Trustworthy Computing progress. 90%, 31 December 2008. The trajectory is established.
11. Specific PCI compliance enforcement actions. 85%, 31 December 2008. The cumulative regulatory pressure continues.
12. Cumulative two-factor authentication deployment at major UK retail banks. 90%, 31 December 2008. The trajectory continues.
Structural
13. Continued data-breach disclosure expansion. 85%, 31 December 2008.
14. Specific UK regulatory tightening on data protection. 65%, 31 December 2008. The political trajectory continues.
15. Specific industry-level conversations about software-vendor liability. 55%, 31 December 2008. The cumulative conversation continues.
Personal
16. Continue the CISO role at Gala Coral productively. 85%, 31 December 2008.
17. Continue weekly cadence on the notebook. 95%, 31 December 2008.
18. Attend at least four conferences. 80%, 31 December 2008.
19. Speak at at least one conference. 80%, 31 December 2008. Specific Infosec Europe opportunity is emerging.
20. Specific follow-up to the DDoS book. 60%, 31 December 2008. Cumulative reception will inform; specific updates or further writing may follow.
A meta-prediction
21. By end of 2008 I will have ten full years of prediction-scoring data. 95%.
The cumulative meta-analysis from ten years will be substantively interesting. Specific systematic biases visible across the cumulative archive will inform subsequent prediction discipline.
A specific note on the Infosec Europe prediction
For 2008, specific Infosec Europe opportunity is emerging. The cumulative conference engagement, the DDoS book, the cumulative practitioner network — all support specific speaking opportunities. Specific 2008 will likely include a substantive Infosec Europe presentation.
The 80% probability reflects realistic confidence; specific scheduling decisions are bounded. The cumulative trajectory toward more substantive conference presence continues.
A closing reflection
The calibration discipline produces cumulative learning across years. Specific predictions inform; specific reviews inform; specific cumulative patterns inform.
For my continued practice: the discipline continues. The cumulative archive continues growing.
More in 2008.