Inter-annotator agreement is measured by having two or more domain experts label the same cases independently and counting how often they match. It is the cheapest and most under-used measurement in enterprise AI, because it converts an argument about accuracy into a number everyone can see.
Worked example: if two senior claims adjusters agree on 72% of cases, no model is going to be judged “95% accurate” on the same task, and the disagreements they had are precisely the rules nobody had written down. Gotcha: measure it before an accuracy target goes into a contract; afterwards it looks like an excuse rather than a finding.