The AI Quality Analyst, AI & Innovation safeguards the quality, reliability, and measurable performance of the team’s AI data extraction processes, which convert commercial risk insurance documents — such as policies, endorsements, submissions, schedules of values, and loss runs — into structured data. Rather than building the extraction systems, this role exists to support the organization’s Enterprise AI Evaluation Strategies — the standards, methods, and processes used to determine whether AI systems are accurate, reliable, and ready for use. The role spans both the foundation and the practice of evaluation: building and curating the ground-truth (“golden”) datasets against which AI performance is measured, and contributing directly to the evaluation processes those datasets power. Reporting to the Director, AI & Innovation, the AI Quality Analyst creates, curates, and maintains ground-truth files, helps design and run evaluations, reviews AI extraction outputs against ground truth, and delivers clear, data-driven feedback that the development team uses to advance the extraction process. A central part of the role is engaging directly with business users: training them on the ground-truth application, guiding them through the process of producing and reviewing ground-truth data, and collecting the associated source files and validated outputs. The AI Quality Analyst maintains a comprehensive, working understanding of the extraction processes to define what “correct” looks like and to safeguard extraction quality end to end. The role is well suited to someone who brings a working knowledge of commercial risk insurance, is familiar with AI/ML and large language model concepts, and is skilled at quality review, data stewardship, and stakeholder engagement — without requiring hands-on AI/ML development experience. Key Responsibilities • Own, create, and maintain the curated ground-truth (“golden”) datasets and ground-truth labels that developers rely on to measure, benchmark, and improve AI extraction quality — the foundation of the team’s evaluation work. • Contribute to the design, execution, and continuous improvement of the organization’s Enterprise AI Evaluation Strategies, making sure ground-truth data is built to serve the evaluations it supports rather than simply collected and stored. • Help design evaluation plans and test sets for AI extraction use cases, selecting representative and edge-case samples across commercial risk insurance document types, lines of business, and levels of complexity. • Participate in evaluation cycles — baseline, regression, and pre-release testing — whenever models, prompts, or processes change, and help define the acceptance thresholds that determine whether changes are ready for production. • Help calibrate automated evaluation methods, such as LLM-as-judge scoring, against human-reviewed ground truth to confirm those methods can be trusted at scale. • Contribute to enterprise evaluation standards, templates, and playbooks so evaluation practices are consistent and reusable across AI initiatives beyond data extraction. • Define, document, and uphold clear standards and annotation guidelines for what constitutes a correct extraction, so quality can be measured consistently and reproducibly. • Apply commercial risk insurance knowledge to define what a correct extraction looks like for key policy and risk data — such as named insureds, lines of coverage, limits, deductibles and retentions, premiums, exclusions, and loss history — and to resolve ambiguous or complex cases. • Engage with and train business users on the ground-truth application, guide them through creating and reviewing ground-truth data, and collect the associated source files and validated outputs needed to build and expand the ground-truth library. • Review AI extraction outputs against ground truth, identify and categorize discrepancies and error types, and deliver actionable diagnostics and feedback to the team advancing the extraction process. • Coordinate human-in-the-loop review and annotation workflows, including inter-annotator agreement, to keep ground-truth data accurate, consistent, and trustworthy. • Manage the versioning and lifecycle of ground-truth datasets so quality results remain reproducible and comparable as models, prompts, and processes change. • Partner with developers to define and interpret quality metrics and acceptance criteria — accuracy, precision, recall, F1, completeness, groundedness/faithfulness, hallucination rate, and schema conformance — and help translate results into clear findings. • Monitor quality trends across extraction runs to surface drift, degradation, and anomalies before they reach downstream products, escalating issues to the team. • Maintain a comprehensive, working understanding of the existing AI data extraction processes the team leads, partnering closely with the Manager and team in support of their continuous improvement. • Support responsible-AI review of AI outputs, including robustness, bias, safety, and compliance considerations at the output level. • Define and maintain quality metrics, dashboards, and reporting that give the team and stakeholders clear visibility into extraction performance and ground-truth coverage. • Document ground-truth methodology, metric definitions, annotation standards, and quality practices to establish and mature the team’s AI quality practice. • Collaborate with the Analytics Engineering function, which consumes validated AI-extracted data into Gold-layer assets, and the Applied AI Analyst, which evaluates AI from a business and product perspective, to align quality standards with downstream needs. • 2–5+ years of experience in data quality, quality assurance, data analysis, annotation/labeling, or a related analytical or business-facing role, ideally with exposure to AI/ML, LLM, or data-extraction systems. Hands-on AI/ML or software development experience is not required. • Experience contributing to structured testing, validation, or evaluation processes, such as QA test plans, user acceptance testing, or data or model validation. • Working knowledge of commercial risk insurance — including policy structure, coverage terms, limits, deductibles, exclusions, endorsements, submissions, and loss runs — gained through underwriting, brokerage, claims, risk management, or insurance operations, sufficient to judge whether extracted data is accurate and complete. • General familiarity with large language models and generative AI — what they do well, where they fall short, and why they can produce inaccurate or fabricated outputs — sufficient to understand, review, and reason about AI outputs. • Strong analytical rigor and exceptional attention to detail, with the ability to assess extraction accuracy and translate findings into clear, actionable recommendations. • Excellent communication and stakeholder-engagement skills, with the ability to train and support business users and to build effective ground-truth workflows with non-technical partners. • Comfort working with data and data tools. Preferred Qualifications • Bachelor’s degree in Computer Science, Information Systems, Mathematics, or a related field. • Familiarity with technical generative-AI concepts such as prompt engineering, retrieval-augmented generation (RAG), and LLM-as-judge evaluation. • Understanding of evaluation methodology and metrics for AI/ML and information extraction, such as precision, recall, F1 score, accuracy, and hallucination detection. • Experience building, curating, or maintaining labeled or ground-truth datasets and annotation workflows, including inter-annotator agreement. • Insurance industry designations such as CPCU, AINS, or ARM. • Basic proficiency in SQL for querying evaluation and production data. • Basic scripting or Python familiarity; the role is centered on quality review and data stewardship rather than software development. • Familiarity with AI/LLM observability and evaluation tooling such as LangSmith. • Familiarity with the Databricks platform and tools such as MLflow and Unity Catalog. Success Measures • Coverage, accuracy, and freshness of the ground-truth datasets, and how well they support the evaluations they are built for. • Meaningful contribution to Enterprise AI Evaluation Strategies, including reliable evaluation cycles, clear acceptance criteria, and adoption of evaluation standards across AI initiatives. • The degree to which developers can quickly and confidently measure and validate extraction quality using the ground-truth files provided. • Adoption and proficiency of business users on the ground-truth application, and the volume and quality of source files and validated outputs collected. • Early detection of quality drift, degradation, and anomalies before they impact downstream products. • Clarity and usefulness of the quality metrics, dashboards, diagnostics, and feedback provided to the team and stakeholders. #LI-JM