AI and LLM Model Selection
Overview
Navigate the crowded and complex landscape of artificial intelligence in AI and Model Selection, a two-day course designed to forge expert decision-making skills for AI. This intensive program equips you with a principled framework for model selection, starting with classical machine learning and extending to the frontiers of generative AI and LLMs. You will learn to apply rigorous, objective metrics to dissect model tradeoffs, mitigate common failure points like concept drift and hallucinations, and master advanced enhancement techniques from retrieval-augmented generation (RAG) to parameter-efficient fine-tuning (PEFT). By benchmarking and assessing models in practical, real-world scenarios, you will gain the critical skills to choose the optimal model for high-performing AI systems that deliver tangible results.
Duration
2 Days
Who Should Take This Course
Audience
Data Scientists, AI/ML staff, Developers, Architects, Data Professionals and Technical Managers.
Prerequisites
Participants should have some familiarity with AI/ML. A laptop with internet access is required to complete the lab work.
Why You Should Take This Course
Upon completing this course, participants will be able to:
- Understand and apply objective methods to measure and compare LLMs
- Compare classical ML models, deep learning architectures, and LLMs to determine the right model for
the job - Evaluate tradeoffs between LLMs
- Differentiate between methods of enhancing LLM performance
- Benchmark and assess AI models in real-world scenarios
Course Outline
AI and LLM Model Selection
-
Day 1 – AI Foundations and Classical-to-Modern Model Selection
1. Introduction to Model Selection in AI Systems
- Development Processes with AI
- Role of Model Selection
- Traditional ML vs. Deep Learning vs. Generative AI Approaches
- Model Selection Criteria
2. Model Families in AI
- Classical Models
- Deep Learning Models
- Generative AI Models
- Tradeoffs
3. Evaluation Metrics for AI and LLM Applications
- Classification
- Regression Metrics
- NLP-Specific Metrics
- Special-Purpose Metrics
4. Overfitting and Model Robustness in AI
- Bias-Variance Tradeoff
- Concept Drift
- LLM Hallucinations
- Robust AI
Day 2 – Practical Selection of LLMs and Fine-Tuning Approaches
5. Choosing an LLM
- LLM Training Variants
- LLM Attributes and Tradeoffs
- Open-Source vs. Proprietary
- Benchmarks
6. LLM Enhancement Methods
- Prompt Engineering, Retrieval-Augmented Generation (RAG), and Fine-Tuning
- Prompt Engineering and Prompt Routines
- RAG, Variants, and Embedding Models
- Fine-Tuning and Parameter-Efficient Fine-Tuning (PEFT)
7. Evaluating LLMs and LLM Systems
- Model Performance
- Inference Performance
- System Performance
- Evaluation Guidelines and Strategies
8. Model Selection for Real-World AI Use Cases
- Model Selection Strategy
- Document Q&A
- Log File Analysis
- Research Automation