Senior AI developer
- Toronto, ON
- Permanent
- Temps-plein
- Identify, design, and prototype GenAI agents to support PathWise model design, build, testing, and documentation activities.
- Design and implement RAG-based solutions (document ingestion, chunking, embeddings, retrieval strategies, re-ranking, prompt construction) to make PathWise knowledge and model documentation easily discoverable and actionable.
- Develop and iterate on prompts, tools, and workflows within prototypes so that they can be properly evaluated, refined, and specified for production implementation.
- Define and track evaluation metrics (e.g., answer quality, latency, robustness, safety) for GenAI agents, and use these to guide iteration and recommendations to the core development team.
- Run structured experiments, gather feedback from actuaries and other users, and translate findings into clear technical specifications for production teams.
- Implement and maintain appropriate controls, guardrails, and governance to ensure compliance with Aon’s AI requirements, model governance standards, and relevant regulatory expectations.
- Embed observability, monitoring, and evaluation into GenAI agents to improve reliability, transparency, and performance.
- Champion Aon’s AI governance requirements, including safety, responsible use of data, and clear documentation of model and agent behavior.
- Collaborate closely with:
- Actuaries and PathWise users to understand their workflows, gather requirements, and co-design solutions.
- Product management to prioritize features, define success metrics, and shape the AI agent roadmap.
- PathWise Development & Infrastructure teams to prepare prototypes for hardened implementation, including deployment, security, and monitoring considerations.
- Provide technical leadership by:
- Identifying new and creative ways to apply GenAI and automation to actuarial and financial modeling.
- Mentoring colleagues and sharing knowledge on GenAI, RAG, and agentic system design.
- Bachelor’s degree in Computer Science, Computer Engineering, or a related technical field (or equivalent practical experience).
- 7+ years of experience in Python software engineering, including 2–3 years of demonstrable experience building LLM / GenAI / agentic systems using modern Python frameworks.
- Hands-on experience designing and implementing RAG systems, including document ingestion and preprocessing, chunking and embedding strategies, retrieval and re-ranking approaches, and prompt construction.
- Proven experience building tool-using agents (e.g., orchestrating external tools/APIs, validation steps, and reasoning chains).
- Demonstrated capability in prompt engineering and prompt optimization (e.g., system prompts, few-shot examples, evaluation and refinement).
- Experience defining, implementing, and using evaluation frameworks and metrics for LLM/GenAI systems (e.g., offline benchmarks, success criteria for specific workflows, human-in-the-loop evaluation or A/B tests).
- Familiarity with vector databases, orchestration frameworks, and MLOps / LLMOps tooling.
- Strong understanding of Agile methodologies and the software development life cycle (SDLC).
- Experience with collaboration and delivery tools such as Azure DevOps, Jira, and Confluence.
- Prior experience in actuarial, risk, or financial modeling domains is strongly preferred; alternatively, a proven ability to quickly learn and work with complex financial or quantitative systems is essential.