Kategória: Munka, állás
Honlap:
Részletek...At Instructure , we believe in the power of people to grow and succeed throughout their lives. Our goal is to amplify that power by creating intuitive products that simplify learning and personal development, facilitate meaningful relationships, and inspire people to go further in their education and careers.
We do this by giving smart, creative, passionate people opportunities to create awesome. And that's where you come in:
We are looking for an AI Engineer to join the team behind the Canvas Agent. As we build out agentic workflows that can "act" upon Canvas and assist educators and students, ensuring the quality, safety, and accuracy of these interactions is our top priority. In this role, you will design and implement robust LLM-as-a-judge evaluation pipelines that automatically assess the Canvas Agent's multi-step reasoning, tool usage, and conversational helpfulness.
What You'll Do
Design the Evaluation Framework: Build and maintain scalable LLM-as-a-judge pipelines to automatically score the Canvas Agent's actions, responses, and tool usage across a variety of complex educational workflows.
Develop Rubrics & Datasets: Create comprehensive grading rubrics and curate high-quality "golden" datasets (both real and synthetically generated) to baseline and test the agent's performance.
Optimize Judge Prompts: Engineer and iterate on prompts for the judges, ensuring automated scoring aligns with high quality evaluations.
Full-Stack Contribution: Step beyond evaluation pipelines to participate in full-stack product development as needed, collaborating with the team to build and refine the core AI features, UI components, and application architecture.
Accelerate Iteration: Integrate your automated evaluations directly into our CI/CD pipelines, creating a "paved path" that allows our AI product teams to ship updates with high velocity and total confidence.
Analyze & Report: Monitor evaluation metrics to identify failure modes, hallucinati...
