AI Software Engineering Domain Expert — Model Evaluation & Prompt Refinement

About the role

We're looking for experienced software engineers to join our network of AI Domain Experts, helping shape the next generation of AI coding and reasoning systems. In this role, you'll apply your technical judgment to evaluate, critique, and refine AI-generated outputs, ensuring they meet rigorous standards of correctness, clarity, and professional quality. This is a remote, flexible engagement ideal for engineers who enjoy critical analysis, technical writing, and working at the intersection of software engineering and AI training.

What you'll do

• Evaluate AI-generated code, technical explanations, and written outputs for correctness, quality, and coherence
• Author, test, and refine prompts to elicit better model behavior across a range of software engineering tasks
• Apply structured rubrics to score and annotate AI outputs consistently and objectively
• Conduct fact-checking, technical editing, and quality assurance passes on AI-produced documentation and reports
• Write clear, professional feedback and evaluation reports that inform model training and improvement
• Perform independent research to verify technical claims and identify edge cases or failure modes
• Collaborate with cross-functional teams to surface patterns in model performance and suggest refinements
• Contribute to RLHF-style feedback loops by ranking and annotating model responses

Requirements

• Strong background in software engineering, computer science, or a related technical discipline
• Excellent critical thinking, analytical reasoning, and attention to detail
• Demonstrated ability to write clear, professional technical documentation and reports
• Comfort evaluating and critiquing technical content against defined rubrics
• Strong written and business communication skills
• Ability to work independently, manage time effectively, and meet deadlines in a remote setting
• Experience with prompt engineering, AI output evaluation, or data annotation is a strong plus

Compensation

This engagement offers competitive hourly compensation of $100–$200/hour, commensurate with experience and role complexity. Work is remote and flexible, with opportunities for ongoing, recurring assignments based on performance.