We are sharing a specialised full-time opportunity for experienced technical professionals to operate at the intersection of AI research, machine-learning data systems, evaluation, and real-world model performance. Selected professionals will take ownership of research and evaluation initiatives designed to generate high-quality, defensible research signal and translate that signal into measurable improvements in AI systems. The role combines evaluation design, ML-oriented data development, failure analysis, quality calibration, and close collaboration with researchers, domain experts, and operational teams. Key Responsibilities Research Evaluation & Signal Quality • Own research and evaluation initiatives from problem framing through data design, quality calibration, and signal validation • Define rigorous approaches for determining whether experimental results provide reliable and defensible research signal • Analyse model and system failures to identify root causes, edge cases, and opportunities for improvement • Evaluate whether datasets, experiments, and conclusions meet appropriate quality thresholds • Act as a quality gate when signal strength, data integrity, or supporting evidence is insufficient ML-Oriented Data & Evaluation Design • Design ML-oriented data systems including task definitions, annotation schemas, rubrics, incentives, and supporting pipelines • Structure data and evaluation workflows around downstream model-performance objectives • Translate ambiguous real-world behaviour into measurable evaluation frameworks and new data categories • Identify gaps in evaluation or dataset coverage and recommend where additional investment or iteration is needed • Develop quality-assurance processes that maintain strong and consistent research standards Failure Analysis & Iterative Model Improvement • Investigate model and system behaviour to identify recurring weaknesses and performance limitations • Iterate rapidly on evaluations, datasets, feedback loops, and quality standards • Use experimental findings to guide improvements in model or agent performance • Determine when research directions should be expanded, revised, paused, or discontinued based on evidence • Maintain a systems-level perspective focused on end-to-end AI performance rather than isolated components Research Collaboration & Technical Communication • Work closely with researchers, domain experts, operators, and cross-functional teams throughout project kickoff, calibration, and iteration • Communicate research findings, trade-offs, limitations, and signal strength clearly to technical and non-technical stakeholders • Translate research progress into credible narratives grounded in evidence • Support alignment between experimental work and real-world system requirements • Contribute strong technical judgement in ambiguous, high-impact research environments Ideal Profile • Strong professional judgement regarding research signal quality and whether findings are ready to support broader conclusions • Experience designing ML-oriented datasets, evaluation frameworks, annotation systems, rubrics, or QA processes • Ability to translate complex and ambiguous real-world system behaviour into structured research and evaluation opportunities • Strong ownership mindset and comfort making decisions in uncertain or rapidly evolving environments • Excellent written and verbal communication skills • Ability to explain technical trade-offs, limitations, evidence quality, and research findings clearly • Proven experience working directly with researchers, technical experts, or domain specialists during project calibration and iteration • Systems-level understanding of model, agent, or AI-system performance • Experience with reinforcement-learning environments, simulators, or feedback-driven training systems is advantageous • Experience improving agentic systems or AI systems operating within real-world workflows is beneficial • Prior work within applied research or production environments with direct impact on deployed systems is advantageous • Experience designing evaluations for complex or real-world tasks is strongly valued • Familiarity with expert incentive design or high-stakes technical research programmes is beneficial Engagement Details • Full-time engagement • Fully remote • Compensation: $600,000–$2,000,000/year • Work will span research evaluation, ML-oriented data design, failure analysis, quality calibration, and iterative AI-system improvement • Responsibilities may include acting as a quality gate for research claims, datasets, and evaluation results • Collaboration will involve researchers, domain experts, operational teams, and other technical stakeholders • Project priorities, evaluation frameworks, and research directions may evolve based on experimental findings and system performance • Work must be completed without using confidential or proprietary information belonging to any employer, client, institution, or other third party About the Platform This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams. By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy