LLMOps Workshop

Welcome to the LLMOps workshop! This course will guide you through building, evaluating, monitoring, and deploying Large Language Model solutions efficiently using Azure AI, Azure Machine Learning Prompt Flow, Content Safety, and Azure OpenAI. Let’s master LLMOps together!

Table of Contents

The workshop is composed of the lessons below. They are sorted by coding levels: no-code for beginners, low-code for intermediates, and full-code for advanced learners, indicating the coding effort needed.

Each lesson includes an estimated time that covers both the conceptual part and the execution of the labs. Please note that this is an estimate to assist with time planning and may vary slightly depending on the case.

Lesson 1: Intro to LLMs and Azure AI Services

no-code. 1h duration

• Introduction to LLMs: GPTs and other models.
• LLMOps: applying MLOps principles to LLM Solutions.
• Azure AI Services Overview:
  • Azure OpenAI
  • Azure AI Studio
  • Azure AI PromptFlow
  • Azure Content Safety

Lesson 2: Building LLMs Orchestration Flows

low-code. 1.5h duration

• LLM App Orchestration.
• Prompt Flow Standard and Chat flows.

Lesson 3: Evaluating and Deploying LLMs

low-code. 1.5h duration

• Prompt flow Evaluation flows to evaluate LLMs Solutions.
• Generated content metrics: groundedness, relevance, etc.
• Deploying LLMs Flows.

Lesson 4: Monitoring and Responsible AI

low-code. 1h duration

• Monitoring LLMs orchestration flows.
• Generated content performance metrics.
• Operational metrics.
• Content safety to protect your solution.

Lesson 5: Automating Everything

full-code. 2h duration

• Github and Github Actions.
• Evaluation and Deployment Automation.

Extras

LLM Performance Testing

low-code. 2h duration

• Benchmarking Azure OpenAI Models
• Load Testing LLM Apps