An off-cadence post, this one. I have been writing for eighteen months now, with reasonable confidence, about a discipline I have been actively learning for two and a half years. The mismatch between confidence-on-paper and depth-of-understanding is something I have been worrying about lately, and I want to write the worry down before it accumulates into a problem.
What I have written that I am less sure about now
A few examples, in increasing order of how much they bother me.
The Bugtraq post from last February. I described the experience of reading the list as "humbling". It was, but my framing of what was humbling was wrong. I described it as "there are people on this list with decades of experience", which is true but not the point. The point I have come to is that the discipline itself is older and deeper than any individual reader; the humility comes from the realisation that even the senior people on the list are constantly being surprised by things, that the field is genuinely expanding in ways nobody fully tracks, and that any given expert is partial in their coverage. I had it as a static hierarchy. It is more like an ocean.
The chroot post from May. I wrote that chroot was "the oldest containment mechanism in Unix". Strictly true. But my treatment of it as "the primary containment mechanism in Unix" implies a centrality that chroot does not actually have in the modern Unix security model. The real primary mechanisms are user-id separation, file permissions, and process credentials. Chroot is an additional mechanism, and a relatively weak one. I think the post still gets the technical detail right, but the framing implies that chroot is more important than it is.
The Snort rule-writing post from January. I gave specific advice about rule structure, including the "write rules in pairs" pattern. After six months of running rules I now know much more, and I think some of my advice was the kind of advice you give when you have been doing something for two weeks and are still excited by it. The pattern of paired alert+pass rules works for some things and is overkill for many. I would write that post differently now. I am not going to redact it; it is the right kind of mistake to leave visible.
The predictions post. Of the five things I predicted, two are clearly correct (Snort ruleset maturing, distributed DoS becoming a thing), one is unclear (honeypots becoming a category — they are forming, slowly), one I underestimated (Y2K's security implications are turning out to be more about opportunity — the rushed remediation is producing real bugs — than I had thought), and one is unresolved (the disclosure conversation has moved a little but not as much as I implied). The win rate is OK. The confidence in the predictions, looking back at them, is higher than my actual hit rate justifies.
Why I am writing this
Not for performative humility. I am writing it because I have noticed a pattern in my own work, and the pattern bothers me.
The pattern is: when I write about something I have been doing for a while, the writing is calibrated and useful. When I write about something I have been doing for only a few weeks, the writing is more confident than it should be. The difference is not visible to a reader; it would not be visible to me either, if I had not gone back and reread the older posts after more experience.
The risk of this is that I get a reputation as a writer of confident, plausible, partially-wrong material. Several of the people whose writing has shaped my own thinking — names on Bugtraq, in Phrack, on the firewall-wizards list — write with markedly more uncertainty than I do. They use phrases like "I think" and "in my experience" and "this might be wrong because". I do not, mostly. I should.
Where this comes from, in part
A chunk of this is, frankly, my own self-image. I started writing partly to be the person who knows about defensive computing. The performance of expertise is a powerful drug. It is much more comfortable to write "the right way to do X is Y" than "I think the right way might be Y, but I have not been doing this long enough to be sure, and here are the cases I am most worried about".
The latter is worse to read for someone looking for an answer. It is much more accurate. It also produces, over time, a body of writing whose claims you can trust, because the calibration is honest.
The former is easier to read and easier to write. It also produces, over time, a body of writing that ages badly, because the confident claims turn out to be partial and the writer has nowhere to retreat to without contradicting themselves.
I would rather age well.
What I am going to change
Small things.
When I write about something I have been doing for less than three months, I am going to label it explicitly. "Two weeks of trying X has taught me" rather than "the right way to do X is". The second is, in nearly every case, an over-reach.
When I have a strong opinion about an emerging area, I am going to write it down with the reasons I might be wrong. Not as a rhetorical hedge but as a real list. If I cannot list reasons I might be wrong, I do not understand the subject well enough to write about it confidently.
When I write predictions, I am going to score them later. The 1999 predictions were the first I made publicly. I am going to score them honestly at year-end. If most are wrong, I will write that down in the same plain way I wrote the predictions.
When I am asked for advice about something I do not know well — and this happens a few times a week, by email, from people who have read the blog — I am going to actually say "I do not know". I have been saying "in my experience" when in fact I do not have experience. This is dishonest, even if not in any major way.
A reading I keep returning to
A few weeks ago I reread part of The Cuckoo's Egg, Cliff Stoll's account of running down a network intrusion in 1986. The thing that struck me on the second reading was not the technical detail but the tone. Stoll writes as someone who is not sure what is going on, follows leads carefully, second-guesses his own conclusions, and is frequently surprised. The book is, in its texture, an exercise in calibrated uncertainty. It is a much better book for that.
The public security writers I find most useful, I now realise, all do this. The ones I find least useful — and there are many — write with confident certainty about things they know less well than I do. The difference is the texture of the writing, more than its content.
The goal, for me, is to develop the texture. The content will follow. Or, more honestly, the content will be more useful if the texture is calibrated.
Back to regular cadence next week. There is a Trinoo follow-up I want to write, and BIND 8.2.2 has produced enough field experience that there is something useful to say. None of those will require I revisit this post. Most of them will, in some quiet way, be shaped by it.