AI
AI and LLM Model Selection
Navigate the crowded and complex landscape of artificial intelligence in AI and Model Selection, a two-day course designed to forge expert decision-making skills for AI. This intensive program equips you with a principled framework for model selection, starting with classical machine learning and extending to the frontiers of generative AI and LLMs. You will learn to apply rigorous, objective metrics to dissect model tradeoffs, mitigate common failure points like concept drift and hallucinations, and master advanced enhancement techniques from retrieval-augmented generation (RAG) to parameter-efficient fine-tuning (PEFT). By benchmarking and assessing models in practical, real-world scenarios, you will gain the critical skills to choose the optimal model for high-performing AI systems that deliver tangible results.
Duration
2 days
Audience
Data Scientists, AI/ML staff, Developers, Architects, Data Professionals and Technical Managers.
prerequisites
Participants should have some familiarity with AI/ML. A laptop with internet access is required to complete the lab work.
Upon completing this course, participants will be able to:
- Understand and apply objective methods to measure and compare LLMs
- Compare classical ML models, deep learning architectures, and LLMs to determine the right model for
the job - Evaluate tradeoffs between LLMs
- Differentiate between methods of enhancing LLM performance
- Benchmark and assess AI models in real-world scenarios
Day 1 – AI Foundations and Classical-to-Modern Model Selection
1. Introduction to Model Selection in AI Systems
- Development Processes with AI
- Role of Model Selection
- Traditional ML vs. Deep Learning vs. Generative AI Approaches
- Model Selection Criteria
2. Model Families in AI
- Classical Models
- Deep Learning Models
- Generative AI Models
- Tradeoffs
3. Evaluation Metrics for AI and LLM Applications
- Classification
- Regression Metrics
- NLP-Specific Metrics
- Special-Purpose Metrics
4. Overfitting and Model Robustness in AI
- Bias-Variance Tradeoff
- Concept Drift
- LLM Hallucinations
- Robust AI
Day 2 – Practical Selection of LLMs and Fine-Tuning Approaches
5. Choosing an LLM
- LLM Training Variants
- LLM Attributes and Tradeoffs
- Open-Source vs. Proprietary
- Benchmarks
6. LLM Enhancement Methods
- Prompt Engineering, Retrieval-Augmented Generation (RAG), and Fine-Tuning
- Prompt Engineering and Prompt Routines
- RAG, Variants, and Embedding Models
- Fine-Tuning and Parameter-Efficient Fine-Tuning (PEFT)
7. Evaluating LLMs and LLM Systems
- Model Performance
- Inference Performance
- System Performance
- Evaluation Guidelines and Strategies
8. Model Selection for Real-World AI Use Cases
- Model Selection Strategy
- Document Q&A
- Log File Analysis
- Research Automation
Is there a discount available for current students?
UMBC students and alumni, as well as students who have previously taken a public training course with UMBC Training Centers are eligible for a 10% discount, capped at $250. Please provide a copy of your UMBC student ID or an unofficial transcript or the name of the UMBC Training Centers course you have completed. Asynchronous courses are excluded from this offer.
What is the cancellation and refund policy?
Student will receive a refund of paid registration fees only if UMBC Training Centers receives a notice of cancellation at least 10 business days prior to the class start date for classes or the exam date for exams.