Tales from AI Development Hell

For some years now I have been having the same two conversations, and the gap between them is why this book exists.

The first happens in a room with technical people. It is precise, sceptical and productive. Somebody explains what a system does and, more usefully, what it does not. Limitations are stated plainly. Somebody says that the evaluation was run on a held-out set and that live behaviour may differ. Somebody else says that the failure cases cluster in a particular segment. Nobody is selling anything, and the register of the discussion is closer to a seminar than to a pitch.

The second happens two floors up, some weeks later, and it concerns the same system. It is confident, strategic and rich in nouns. The limitations have gone. The evaluation figure has survived, detached from its conditions. There is a date, a benefit and a slide with a piggy bank on it. And the person presenting is not misrepresenting anything, because they are faithfully reporting what they were told by somebody who was faithfully reporting what they were told.

I have watched perfectly good information degrade across three floors of a building more often than I have watched anyone lie, and the degradation is more dangerous, because nobody is looking for it.

This book is an attempt to describe how that happens, using cases where the consequences were serious enough that somebody with subpoena powers went and found out.

Where I am standing

I write as a practitioner rather than as a researcher, though I hold an academic position and take the scholarship seriously. That vantage has an obvious limitation, which is that I have not conducted original empirical research into any of the organisations in this book, and I have relied on the work of investigators, regulators, auditors and journalists who did.

It also has an advantage that I want to state, because it shapes what follows. I have sat in the meetings. I know what it is like to be the person who thinks the threshold is wrong six weeks before a launch date that has been announced to a board. I know what happens to a caveat in the fourth redraft of a deck. I know why the appendix gets detached, and it is not conspiracy; it is that the deck was sixty slides and the executive had forty minutes.

Books about organisational failure written from outside organisations tend to be baffled by the ordinariness of what they find, and they reach for explanations involving greed, stupidity or hubris. Those exist. They are not, in my experience, the common case. The common case is a competent person, under time pressure, doing something defensible, inside a structure that will convert the aggregate of many defensible decisions into an outcome that nobody chose.

If this book has a single argument, it is that the structure is the thing to work on, and that the structure is management's, not the technology's.

What I have tried to avoid

Three temptations, and I have not resisted all of them equally well.

The first is the villain. It is much easier to write a compelling chapter if somebody in it is contemptible, and every case here offered a candidate. I have declined, not out of charity but because the villain version is analytically useless. If the cause was a bad person, the lesson is to avoid hiring bad people, and your organisation already believes it has done that.

The second is the unearned fact. There is a version of this book in which the Amazon recruiting story is stated flatly, in which an algorithm takes children from their parents, and in which every figure is the most dramatic one in circulation. That version reads better. It also forfeits the right to complain about anybody else's evidentiary standards, which is most of what the book is for.

The third is the framework. Consultancy has trained a generation of executives to expect a two-by-two, a maturity model and a set of levers, and there is real commercial pressure on a book like this to supply them. The tools here are deliberately thin, mostly one page, and several of them are simply questions. That is not modesty. It is that the elaborate versions do not get used, and an unused control is indistinguishable from an absent one.

What I want from you

Not agreement. Several of the judgements in this book are contestable and I have tried to mark them as such, particularly where I disagree with people whose work I respect.

What I want is narrower and more practical. I would like you to finish the book able to ask five questions about any system your organisation runs, and to be uncomfortable when nobody can answer them.

If you get that far, the rest tends to follow, because the discomfort is the mechanism. Every organisation in these pages contained the information required to prevent what happened to it. What it lacked was somebody senior enough to be uncomfortable in the right meeting.

That is a job you can do on Monday, and it does not require a budget.

Tales from AI Development Hell

Moonshots, Misfires and Management: How to Avoid an AI Boondoggle