Machine studying (ML) and synthetic intelligence (AI) programs rely closely on human-annotated information for coaching and analysis. A significant problem on this context is the prevalence of annotation errors, as their results can degrade mannequin efficiency. This paper presents a predictive error mannequin skilled to detect potential errors in search relevance annotation duties for 3 industry-scale ML purposes (music streaming, video streaming, and cell apps). Drawing on real-world information from an intensive search relevance annotation program, we show that errors may be predicted with average mannequin efficiency (AUC=0.65-0.75) and that mannequin efficiency generalizes nicely throughout purposes (i.e., a world, task-agnostic mannequin performs on par with task-specific fashions). In distinction to previous analysis, which has typically centered on predicting annotation labels from task-specific options, our mannequin is skilled to foretell errors instantly from a mix of job options and behavioral options derived from the annotation course of, with a view to obtain a excessive diploma of generalizability. We show the usefulness of the mannequin within the context of auditing, the place prioritizing duties with excessive predicted error possibilities significantly will increase the quantity of corrected annotation errors (e.g., 40% effectivity beneficial properties for the music streaming utility). These outcomes spotlight that behavioral error detection fashions can yield appreciable enhancements within the effectivity and high quality of information annotation processes. Our findings reveal important insights into efficient error administration within the information annotation course of, thereby contributing to the broader area of human-in-the-loop ML.
