General AI/ML Engineering Questions
Start with questions that apply to any AI engineering role. Ask candidates to explain the difference between training, validation, and test sets — and why this matters in production. Ask how they would detect and handle data drift in a deployed model. Ask them to describe a time they had to choose between model accuracy and inference speed, and how they made the decision.
Good answers demonstrate practical experience, not textbook knowledge. Listen for mentions of specific tools, real-world constraints, and lessons learned from failures.
LLM and Prompt Engineering Questions
For LLM-focused roles, ask: How would you design a RAG system for a company with 10,000 internal documents? Walk through the chunking, embedding, retrieval, and generation steps. What's the difference between fine-tuning and few-shot prompting, and when would you use each? How do you evaluate prompt quality at scale? Describe your approach to prompt versioning and testing.
Advanced LLM Questions
For senior candidates: How would you reduce hallucinations in a production LLM system? What's your approach to handling multi-turn conversations with context windows? How do you optimize LLM costs when processing thousands of requests per day?
MLOps Questions
For MLOps roles: Describe how you'd set up a CI/CD pipeline for ML models. How does it differ from traditional software CI/CD? How would you implement model monitoring in production — what metrics would you track and what would trigger an alert? Walk through how you'd handle a model rollback if a newly deployed version shows degraded performance.
System Design Questions
For senior roles, present system design scenarios: Design a real-time recommendation system that serves 10,000 requests per second. Design an AI-powered document processing pipeline that handles 50 different document types. Design a multi-model serving infrastructure that can A/B test different models in production.
Behavioral Questions That Actually Matter
Skip generic behavioral questions. Instead ask: Tell me about a model that performed well in testing but failed in production. What happened and what did you learn? Describe a situation where you had to push back on a stakeholder's AI request because it wasn't technically feasible. How do you stay current with AI developments? What's the most significant thing you've learned in the last 6 months?
Using This Question Bank
Select 4-6 questions relevant to your specific role. Mix foundational questions with specialization-specific ones. Leave time for the candidate to ask you questions — the quality of their questions often reveals more than their answers. South's pre-screening process uses similar questions to ensure candidates meet your technical bar before they reach your interview pipeline.
