Challenge 02 - Build the Context Engine

< Previous Challenge - Home - Next Challenge >

Pre-requisites

Introduction

Agent Mole may need to discover your indices, mappings, and joins before answering a query, and may repeat that work across conversations. Elastic’s Context Engine, available as Context in Kibana, moves that recurring interpretation out of the agent’s question-answering loop. It lets you create AI Indices, attach source indices, and run automations using Kibana Workflows to generate reusable context.

An AI Index is a special index (named ai-index-idx-<name>) that holds Knowledge Indicators (KIs) for agents to consult before querying source data. Each KI has a type, title, description, and content. In this challenge, you will create index_metadata KIs describing each source’s purpose, key fields, join keys, known gaps, and verified ES|QL access patterns. An AI Index does not merge or replace source documents.

Without KIs, an agent may repeatedly call tools such as list_indices and get_index_mapping. With KIs, it can retrieve routing facts and move to targeted ES|QL queries. KIs describe structure and meaning, not live values: current counts, statuses, and dates still require source queries.

In this challenge you will build an AI Index and KIs for Precision Labs data that Agent Optimus will use in the next challenge.

Description

Use Kibana and the workflow files in the workflows/ folder in your GitHub Codespace, or in the Resources.zip file provided by your coach.

NOTE: As of October 2026, Context Engine is an experimental feature. You will need to enable it in Kibana before you can use it.

Enable Context Engine

Create the AI Index

Routing context for Precision Labs customer-success data in three Elasticsearch indices: precision-crm-accounts (500 customer accounts with tier, arr = annual recurring revenue in USD, churn risk score, renewal date, and account owner), precision-support-tickets (3,000 support tickets with priority critical, high, medium or low, category, and status; customer_id in accounts joins to account_id in tickets), and precision-kb-articles (150 knowledge base articles by category). Knowledge Indicators describe what each index holds, what its fields mean, how the indices join, and ES|QL query patterns that work. Example questions: Which Enterprise accounts have the highest churn risk, and what are they contacting support about? Which Enterprise accounts renewing in the next 60 days have open critical tickets, and which knowledge base articles cover their issues? Known gaps: there is no product usage, billing, or contract data, and Knowledge Indicators describe structure and meaning, not live values, so query the source indices for current counts, statuses, and dates.

Add the Source Indices

On the precision-corpus AI Index’s Overview tab, find Sources, click Edit, and under the Elasticsearch data tab add each index:

Click Save. Sources should show 3.

NOTE: The Connectors tab lists technical-preview sources such as GitHub, Azure Blob Storage, SharePoint, and OneDrive. The same AI Index can be fed from those.

Generate the Knowledge Indicators with a Workflow

The workflow runs once per source index. It fetches the mapping, samples 20 random documents, runs grounding ES|QL aggregations (counts by category), renders that into a KI, and writes it to the AI Index with context-engine.createKi. Three indices finish in about a minute.

Inspect and Edit the KIs

KIs are ordinary Elasticsearch documents. In Discover try these queries:

NOTE: ES|QL mode, set the time range wide enough to include when you ran the workflow.

FROM ai-index-idx-precision-corpus
| KEEP type, title, content
| SORT type
| LIMIT 20

Semantic-ish match:

FROM ai-index-idx-precision-corpus
| WHERE MATCH(content, "which accounts are likely to churn")
| KEEP title, content
| LIMIT 3

Generated KIs may not say that arr means annual recurring revenue in USD. If so, add it with a plain document update in Dev Tools:

POST ai-index-idx-precision-corpus/_update/precision-crm-accounts
{
  "script": {
    "source": "ctx._source.content += '\n\nNote: arr field = Annual Recurring Revenue in USD (not an array).'",
    "lang": "painless"
  }
}

Re-run the workflow whenever source schemas or data distributions change. It uses stable KI IDs to target the same logical KIs, but regenerated content can differ and replace manual corrections. The supplied workflow runs manually; scheduling requires separate configuration.

Success Criteria

Learning Resources

Tips

Advanced Challenges