Here is my argument: Capturing expert judgment is its own kind of work
One, because it takes a specific kind of knowledge capture skill that ‘requirements gathering’ expertise does not supply.1 This is because when you talked to the experts who ‘do the thing’ every day, you have to recognize that with practice the steps drop out of their own awareness. As a result, experts who describe what they do leave out nearly 70% of the decision steps.1 On the other hand, an interview specifically built to find those steps recovers more of them: Descriptions run about 70% more complete and accurate.1
To put knowledge into words, the interview has to draw out the decisions and how they were made. Relying on free recall will leave these out. Because humans are humans, knowledge rarely gets put into words unless the questions aim directly at articulating that tacit knowledge.2
Business analysts and product managers run free recall: the expert describes the procedure with no help.1 That is not cognitive task analysis. Cognitive task analysis asks what the conditions were, what the expert did, what else they considered, and how they decided. It then checks a write-up of those answers with the expert. Capture needs that method, and it needs time and a place in the plan.
Where does this judgment live today? Consulting answer, “it depends.” Some judgment is in the data itself, born through models, forecasts and analytics. Other judgment sits on what Hayek called knowledge of the particular circumstances of time and place, which never reaches the data.6 Merchandising judgment on pricing, promo, placement and product is almost always backed by data, so capturing it is an analysis exercise to find out what is knowable. At a large US grocer, $2.17M in gross profit was approved after bulk margin drift was found in knowable merchant data. Store leader judgment only comes out in context, through quality decisions. A store operator on hot humid days, with strawberries on the floor 3 to 4 days, visually inspects the bottom ones for mold; that judgment is not in the data, and no rule tells a new operator when, where, or how. Ask a store manager what they did and you get the what. What you really need is the why, but you have to know how to ask for it.7
In a European supermarket chain, store managers changed the system’s order advice.
- In all five stores, managers ordered early, before busy days, so stock arrived when the store had spare hands. The authors inferred this from order and sales data.
- Shifting orders cut the swing in daily shelf-stocking work by 42% on average. The cost was half a day more inventory.
- The two factors the system left out are in-store handling costs and sales potential from better in-stock.
- The hypothesized attributes together explain about 45% of the variation in order advancement, leaving about 55% unexplained. No sales lift was measured.3
In another study’s example, a History of Arts expert didn’t tell the interviewer that pictures in photo libraries were often in paper format, not digital; the interviewer learned this by asking about file format. That withheld knowledge was a fact (vs judgment).4 Across 34 customer-analyst interviews, customers didn’t mention key facts until the interviewer intentionally asked about it.4 Hidden knowledge was uncovered when the analyst noticed logical disconnects and followed up. However, that follow-up was ad hoc, not repeatable and not scalable. Cognitive task analysis is.
The paper is a vision paper, and one analyst ran all the interviews, which were simulated. A survey of software-requirements work covered 228 companies in 10 countries. In it, 48% of respondents named incomplete or hidden requirements among their five most critical problems, more than any other problem.5 The survey counts incomplete or hidden requirements in general. It gives the scale of the gap and says nothing about judgment. In AI projects, a RAND study found the most common cause of failure was misunderstanding what the project was for. Data scientists needed subject-matter experts to explain why things happened, not just what happened.8
This job goes to product managers and business analysts because historically and canonically requirements gathering has fallen on these two roles.
In the past it’s usually been the province of a business analyst or a product manager to capture the judgment as part of the overall requirements. It didn’t get its own time and budget. It got folded into requirements and discovery.
I estimate that cognitive task analysis adds 10 to 20% to the budget. The range comes from my project experience. I have not measured it.
What I really want to talk about is the difficulty in encoding the knowledge because the product managers don’t really know how to do it. It’s a new skill set, a new set of frameworks, and a new way of thinking about it.
These people aren’t trained in interviewing for decisions, vs features or workflows. Cognitive task analysis is the tool for the job in that it treats the expert’s judgment as the product, vs as a requirement for the product.
The gaps cost, I would say, rework later because it was part and parcel to understanding how we could best support the stores.
Three gaps caused that rework:
- not understanding the emergent nature of a store operator’s day to day
- incomplete digital signaling to build a picture of operations, bottlenecks and potential fixes
- the gap between requirements gathering and true knowledge capture in the form of cognitive task analysis
If you fund AI or transformation work, stop folding judgment capture into ordinary requirements and discovery. Give it its own time and budget, and hire interviewers skilled in cognitive task analysis. AI raises the stakes. It makes it easier to build the wrong thing faster, and easier to keep adding features to a tool that doesn’t work than to step back and re-examine the gaps. AI lets you spend more resources faster, and those are wasted if you don’t understand what you’re trying to accomplish.
1 Clark et al., Journal of Surgical Research, 2012 · 2 Clark, Feldon, van Merrienboer, Yates, Early, Handbook of Research on Educational Communications and Technology, 2008 · 3 van Donselaar et al., Management Science, 2010 · 4 Ferrari et al., Requirements Engineering, 2016 · 5 Méndez Fernández et al., Empirical Software Engineering, 2017 · 6 Hayek, American Economic Review, 1945 · 7 Hoffman, Crandall, Shadbolt, Human Factors, 1998 · 8 Ryseff, De Bruhl, Newberry, RAND, 2024