Following last week's prediction-scoring exercise, I committed to a more rigorous prediction discipline — specific predictions, with probabilities and deadlines, scored explicitly at the deadline.
This post is the application of that discipline to 2001. The predictions are specific enough to be testable, with probabilities reflecting my actual uncertainty. The discipline is uncomfortable — putting a probability on an uncertain prediction forces honesty in a way that prose does not — and is worth doing for exactly that reason.
The threat-landscape predictions
1. An automatically-propagating Windows-targeted worm appears in 2001. The vulnerable substrate is large; the exploitation infrastructure is mature; the propagation arithmetic is favourable. Probability: 85%. Deadline: 31 December 2001.
2. The worm in (1) targets either IIS Unicode-style HTTP exploits or SMB/NetBIOS exploits. These are the two largest available substrate populations. Probability: 75%. Deadline: 31 December 2001.
3. A significant DDoS attack against a household-name commercial site exceeds Mafiaboy's scale. The toolkits are evolving; the bandwidth available to attackers is growing. Probability: 70%. Deadline: 31 December 2001.
4. A practical, public WEP-key recovery tool is released. The cryptographic weaknesses are sufficient; the engineering work is converging. Probability: 75%. Deadline: 31 December 2001.
5. ILOVEYOU-class mass-mailing worms continue to appear, at a rate of at least one per quarter. The category is permanent. Probability: 80%. Deadline: 31 December 2001.
6. A specific Linux-targeted worm or exploit kit emerges, beyond the script-kiddie category. Linux deployment is large enough to be worth attacker attention. Probability: 55%. Deadline: 31 December 2001.
The defensive-response predictions
7. Microsoft ships default executable-attachment blocking in Outlook. The pressure is sufficient; the product cycle should permit it. Probability: 65%. Deadline: 31 December 2001.
8. Microsoft launches a substantial security initiative analogous to Trustworthy Computing. Public commitment to development-process change. Probability: 50%. Deadline: 31 December 2001.
9. BCP 38 / source-address-validation deployment becomes a peering norm at major US carriers. Some specific public commitment from at least one of the major operators. Probability: 60%. Deadline: 31 December 2001.
10. The Honeynet Project publishes a major cross-operator analysis paper. Cumulative data from multiple honeynets, published with statistical findings. Probability: 80%. Deadline: 31 December 2001.
11. The Linux 2.4 kernel reaches mainstream production deployment. Major distributions ship 2.4-based releases as default. Probability: 90%. Deadline: 30 June 2001.
12. Snort 2.0 development begins publicly. Architectural redesign discussions appear on the mailing list. Probability: 70%. Deadline: 31 December 2001.
The structural-shift predictions
13. A specific high-profile data-breach incident produces regulatory response. ICO action in the UK, FTC action in the US, EU directive proposed, or similar. Probability: 60%. Deadline: 31 December 2001.
14. "Cloud"-style hosting starts being deployed by smaller organisations in measurable numbers. Outsourced computing for non-technical organisations begins to be normal. Probability: 50%. Deadline: 31 December 2001.
15. Network segmentation becomes mainstream advice for non-trivial corporate deployments. Default architectural recommendation in industry guidance. Probability: 70%. Deadline: 31 December 2001.
16. A major closed-source vendor (Microsoft, Cisco, or similar) ships a substantial open-source-style transparency initiative. Some kind of source review programme, audit transparency, or open-development effort. Probability: 30%. Deadline: 31 December 2001.
The personal predictions
17. I attend at least four conferences in 2001. The Manchester gathering was valuable; I committed to quarterly attendance. Probability: 80%. Deadline: 31 December 2001.
18. I speak at at least one conference. I have been thinking about it; commitment forces action. Probability: 65%. Deadline: 31 December 2001.
19. The honeypot is expanded to a small range of IPs (at least 8 distinct addresses). The Honeynet tooling makes this feasible. Probability: 70%. Deadline: 30 June 2001.
20. I write the small-business-oriented piece I have been thinking about. The gap is real; the commitment is reasonable. Probability: 75%. Deadline: 30 June 2001.
21. The notebook continues at weekly cadence through 2001. This is the easiest to commit to since the discipline is well-established. Probability: 95%. Deadline: 31 December 2001.
What this exercise has felt like
Writing this list, with explicit probabilities, has been more uncomfortable than I expected. A few specific frictions:
Some predictions are obviously embarrassing. Number 16 — major closed-source vendors shipping transparency initiatives — at 30% probability feels too low for a prediction I am even bothering to write down. I should be willing to say either it is at least 50% likely, in which case it is a serious prediction, or it is below 30%, in which case it is not worth committing to. Calibrating between these is hard.
The deadline discipline produces clarity. "By 31 December 2001" is more specific than "during 2001". The specificity exposes ambiguities. "Major commercial site DDoS" — what counts as commercial? as major? as DDoS? The deadline forces me to think about the scoring rules in advance.
The probability discipline produces honesty. When I have to say "75% chance", I cannot be vaguely confident. I have to specifically identify why I think it is 75% and not 60% or 90%. The number is more accountable than the prose.
The cumulative scoring will be informative. If I am consistently over-confident at 75% (i.e., predictions that fail at higher rates), I learn something specific. If I am consistently under-confident at 50%, similarly. The numerical record gives me feedback that prose alone does not.
What I am going to do differently with this list
Three small things:
Review quarterly. Pull this list up every three months, mark predictions that have resolved, note any that should have already resolved, update probabilities on ones still in flight. This is a meaningful change from "score at the end of the year" — the more frequent review forces engagement.
Adjust when I learn something. If new information arrives that changes my probability estimate, I will update it explicitly with the date and the reason. The history of my estimates is more informative than the final number.
Score honestly even when uncomfortable. If a prediction was 80% confident and turned out wrong, I will write that down clearly. The temptation to retro-fit explanations ("of course it was wrong because of X") is the failure mode this exercise is meant to resist.
A small reflection
The exercise of writing down 21 specific predictions with probabilities is, I am increasingly convinced, the kind of discipline that will pay back over years. The cost is modest — an evening to compose the list, periodic review afterwards. The benefit is a structured record of how my mental models are or are not matching reality.
For any reader who finds this kind of exercise interesting: I would recommend trying it. The first list is the hardest; the practice of producing them gets easier; the cumulative record is its own form of self-knowledge.
For my own writing: this kind of post will recur annually. The end-of-year review and beginning-of-year predictions exercise is now part of the cadence. By 2003 or 2004 I should have enough scored predictions to do meta-analysis on my own forecasting accuracy. That meta-analysis is the eventual payoff.
More as the year wraps. End-of-year notebook post next.