Global Search

In [1]:

# Copyright (c) 2024 Microsoft Corporation.
# Licensed under the MIT License.

# Copyright (c) 2024 Microsoft Corporation. # Licensed under the MIT License.

In [2]:

import os

import pandas as pd
from graphrag.query.indexer_adapters import (
    read_indexer_communities,
    read_indexer_entities,
    read_indexer_reports,
)
from graphrag.query.structured_search.global_search.community_context import (
    GlobalCommunityContext,
)
from graphrag.query.structured_search.global_search.search import GlobalSearch
from graphrag.tokenizer.get_tokenizer import get_tokenizer
from graphrag_llm.completion import create_completion
from graphrag_llm.config import ModelConfig

import os import pandas as pd from graphrag.query.indexer_adapters import ( read_indexer_communities, read_indexer_entities, read_indexer_reports, ) from graphrag.query.structured_search.global_search.community_context import ( GlobalCommunityContext, ) from graphrag.query.structured_search.global_search.search import GlobalSearch from graphrag.tokenizer.get_tokenizer import get_tokenizer from graphrag_llm.completion import create_completion from graphrag_llm.config import ModelConfig

Global Search example¶

Global search method generates answers by searching over all AI-generated community reports in a map-reduce fashion. This is a resource-intensive method, but often gives good responses for questions that require an understanding of the dataset as a whole (e.g. What are the most significant values of the herbs mentioned in this notebook?).

LLM setup¶

In [3]:

api_key = os.environ["GRAPHRAG_API_KEY"]

config = ModelConfig(
    type="litellm",
    model_provider="openai",
    model="gpt-4.1",
    api_key=api_key,
)
model = create_completion(config)

tokenizer = get_tokenizer(config)

api_key = os.environ["GRAPHRAG_API_KEY"] config = ModelConfig( type="litellm", model_provider="openai", model="gpt-4.1", api_key=api_key, ) model = create_completion(config) tokenizer = get_tokenizer(config)

Load community reports as context for global search¶

• Load all community reports in the community_reports table from GraphRAG, to be used as context data for global search.
• Load entities from the entities tables from GraphRAG, to be used for calculating community weights for context ranking. Note that this is optional (if no entities are provided, we will not calculate community weights and only use the rank attribute in the community reports table for context ranking)
• Load all communities in the communities table from the GraphRAG, to be used to reconstruct the community graph hierarchy for dynamic community selection.

In [4]:

# parquet files generated from indexing pipeline
INPUT_DIR = "./inputs/operation dulce"
COMMUNITY_TABLE = "communities"
COMMUNITY_REPORT_TABLE = "community_reports"
ENTITY_TABLE = "entities"

# community level in the Leiden community hierarchy from which we will load the community reports
# higher value means we use reports from more fine-grained communities (at the cost of higher computation cost)
COMMUNITY_LEVEL = 2

# parquet files generated from indexing pipeline INPUT_DIR = "./inputs/operation dulce" COMMUNITY_TABLE = "communities" COMMUNITY_REPORT_TABLE = "community_reports" ENTITY_TABLE = "entities" # community level in the Leiden community hierarchy from which we will load the community reports # higher value means we use reports from more fine-grained communities (at the cost of higher computation cost) COMMUNITY_LEVEL = 2

In [5]:

community_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet")
entity_df = pd.read_parquet(f"{INPUT_DIR}/{ENTITY_TABLE}.parquet")
report_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet")

communities = read_indexer_communities(community_df, report_df)
reports = read_indexer_reports(report_df, community_df, COMMUNITY_LEVEL)
entities = read_indexer_entities(entity_df, community_df, COMMUNITY_LEVEL)

print(f"Total report count: {len(report_df)}")
print(
    f"Report count after filtering by community level {COMMUNITY_LEVEL}: {len(reports)}"
)

report_df.head()

community_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet") entity_df = pd.read_parquet(f"{INPUT_DIR}/{ENTITY_TABLE}.parquet") report_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet") communities = read_indexer_communities(community_df, report_df) reports = read_indexer_reports(report_df, community_df, COMMUNITY_LEVEL) entities = read_indexer_entities(entity_df, community_df, COMMUNITY_LEVEL) print(f"Total report count: {len(report_df)}") print( f"Report count after filtering by community level {COMMUNITY_LEVEL}: {len(reports)}" ) report_df.head()

Total report count: 10
Report count after filtering by community level 2: 10

Out[5]:

	id	human_readable_id	community	level	parent	children	title	summary	full_content	rank	rating_explanation	findings	full_content_json	period	size
0	9785e64b429dacd70cce8238051f3438b387ebcd6920ef...	7	7	1	0	[]	Paranormal Military Squad and Operation Dulce	This community centers on the Paranormal Milit...	# Paranormal Military Squad and Operation Dulc...	8.5	The impact severity rating is high due to the ...	[{'explanation': 'The Paranormal Military Squa...	{\n    "title": "Paranormal Military Squad and...	2026-01-13	4
1	212c8ff1796bff731abe3d54f3216511ab15d5fb0d97eb...	8	8	1	0	[]	Paranormal Military Squad: Leadership and Scie...	This community centers on the Paranormal Milit...	# Paranormal Military Squad: Leadership and Sc...	8.0	The community poses a high impact due to its i...	[{'explanation': 'Taylor Cruz, often referred ...	{\n    "title": "Paranormal Military Squad: Le...	2026-01-13	3
2	dbff99eaacade8ac4337a9bde0f533313445f0643ab3d0...	9	9	1	0	[]	Paranormal Military Squad and Operation: Dulce...	This community centers around the Paranormal M...	# Paranormal Military Squad and Operation: Dul...	8.5	The community poses a high impact severity due...	[{'explanation': 'The military complex is the ...	{\n    "title": "Paranormal Military Squad and...	2026-01-13	3
3	b6e8ab1ac8ecc605bbec09ad78819b95ac08d62c5a9991...	0	0	0	-1	[7, 8, 9]	Paranormal Military Squad and Operation: Dulce	This community centers on the Paranormal Milit...	# Paranormal Military Squad and Operation: Dul...	8.5	The community poses a high impact due to its e...	[{'explanation': 'Taylor Cruz is the central a...	{\n    "title": "Paranormal Military Squad and...	2026-01-13	10
4	97038bee58306d13b709b615fb2b816cae0c11273a9e73...	1	1	0	-1	[]	Team of Agents Investigating Dulce Base	This community centers on a specialized team o...	# Team of Agents Investigating Dulce Base\n\nT...	7.5	The impact severity rating is high due to the ...	[{'explanation': 'The core of this community i...	{\n    "title": "Team of Agents Investigating ...	2026-01-13	5

Build global context based on community reports¶

In [6]:

context_builder = GlobalCommunityContext(
    community_reports=reports,
    communities=communities,
    entities=entities,  # default to None if you don't want to use community weights for ranking
    tokenizer=tokenizer,
)

context_builder = GlobalCommunityContext( community_reports=reports, communities=communities, entities=entities, # default to None if you don't want to use community weights for ranking tokenizer=tokenizer, )

Perform global search¶

In [7]:

context_builder_params = {
    "use_community_summary": False,  # False means using full community reports. True means using community short summaries.
    "shuffle_data": True,
    "include_community_rank": True,
    "min_community_rank": 0,
    "community_rank_name": "rank",
    "include_community_weight": True,
    "community_weight_name": "occurrence weight",
    "normalize_community_weight": True,
    "max_tokens": 12_000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)
    "context_name": "Reports",
}

map_llm_params = {
    "max_tokens": 1000,
    "temperature": 0.0,
}

reduce_llm_params = {
    "max_tokens": 2000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 1000-1500)
    "temperature": 0.0,
}

context_builder_params = { "use_community_summary": False, # False means using full community reports. True means using community short summaries. "shuffle_data": True, "include_community_rank": True, "min_community_rank": 0, "community_rank_name": "rank", "include_community_weight": True, "community_weight_name": "occurrence weight", "normalize_community_weight": True, "max_tokens": 12_000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000) "context_name": "Reports", } map_llm_params = { "max_tokens": 1000, "temperature": 0.0, } reduce_llm_params = { "max_tokens": 2000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 1000-1500) "temperature": 0.0, }

In [8]:

search_engine = GlobalSearch(
    model=model,
    context_builder=context_builder,
    tokenizer=tokenizer,
    max_data_tokens=12_000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)
    map_llm_params=dict(map_llm_params),
    reduce_llm_params=dict(reduce_llm_params),
    allow_general_knowledge=False,  # set this to True will add instruction to encourage the LLM to incorporate general knowledge in the response, which may increase hallucinations, but could be useful in some use cases.
    json_mode=False,
    context_builder_params=context_builder_params,
    concurrent_coroutines=32,
    response_type="multiple paragraphs",  # free form text describing the response type and format, can be anything, e.g. prioritized list, single paragraph, multiple paragraphs, multiple-page report
)

search_engine = GlobalSearch( model=model, context_builder=context_builder, tokenizer=tokenizer, max_data_tokens=12_000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000) map_llm_params=dict(map_llm_params), reduce_llm_params=dict(reduce_llm_params), allow_general_knowledge=False, # set this to True will add instruction to encourage the LLM to incorporate general knowledge in the response, which may increase hallucinations, but could be useful in some use cases. json_mode=False, context_builder_params=context_builder_params, concurrent_coroutines=32, response_type="multiple paragraphs", # free form text describing the response type and format, can be anything, e.g. prioritized list, single paragraph, multiple paragraphs, multiple-page report )

In [9]:

result = await search_engine.search("What is operation dulce?")

print(result.response)

result = await search_engine.search("What is operation dulce?") print(result.response)

## Overview of Operation: Dulce

Operation: Dulce is a classified, high-stakes mission led by the Paranormal Military Squad, an elite unit specializing in the investigation and management of anomalous phenomena and existential threats. The primary objective of the operation is to investigate the enigmatic Dulce Base, a secretive and technologically advanced underground facility. The mission is marked by its complexity, uncertainty, and the necessity for strict adherence to protocols, as it seeks to uncover hidden technologies, existential threats, and anomalous phenomena within the base. The operation is considered career-defining for those involved and may have profound implications for humanity's future [Data: Reports (4, 7)].

---

## Mission Objectives and Structure

The core focus of Operation: Dulce is the descent and exploration of the Dulce base. The mission involves several key objectives:

- **Investigation of Anomalous Phenomena:** The squad is tasked with identifying and managing unexplained phenomena, particularly those associated with advanced alien technology and existential threats [Data: Reports (0, 7)].
- **Rescue and Reconnaissance:** The mission structure includes rescue operations, reconnaissance, and the confrontation of unknown dangers within the base. The team must be prepared to face both psychological and physical threats [Data: Reports (3)].
- **Retrieval and Analysis of Alien Technology:** A central aspect of the operation is the retrieval and analysis of enigmatic alien technology discovered at a crash site and subsequently studied at Dulce base. This technology may fundamentally alter human understanding of physics and reality, with significant implications for scientific advancement and security [Data: Reports (2)].

The operation leverages the technical and analytical skills of the squad, emphasizing preparedness, adaptability, and the integration of specialized equipment and secure infrastructure [Data: Reports (9)].

---

## Operational Context and Environment

Operation: Dulce is conducted from a secure military complex, with the operational hub being a high-security briefing room within Dulce base. This facility supports mission planning and execution, emphasizing security, technological capability, and operational readiness [Data: Reports (5)]. The squad operates with advanced communication infrastructure, including encrypted radio transmitters, and relies on robust technical support to maintain operational discipline and security [Data: Reports (9)].

The mission context is defined by uncertainty and the need for trust and expertise among team members. The investigation of existential threats and cosmic secrets may reshape understandings of security, technology, and reality itself [Data: Reports (4)].

---

## Organizational Structure

Operation: Dulce is hierarchical and mission-centric, with the Paranormal Military Squad at its core. The squad is supported by specialized agents, advanced communications systems, and secure infrastructure. The operation is recognized for its elite status and critical role in national security, particularly in managing unconventional threats [Data: Reports (7, 5)].

---

## Implications

The success of Operation: Dulce may have far-reaching consequences, potentially altering the trajectory of scientific knowledge and national security policy. The mission's focus on alien technology and existential threats underscores its significance, and the strict protocols and elite expertise required highlight the gravity of the operation [Data: Reports (4, 2, 0)].

---

## Summary

In summary, Operation: Dulce is a top-secret mission centered on the investigation of the Dulce Base, involving the Paranormal Military Squad's expertise in handling anomalous phenomena, alien technology, and existential threats. The operation is supported by advanced infrastructure and strict protocols, with objectives that may have profound implications for humanity's understanding of technology, security, and reality [Data: Reports (4, 7, 0, 3, 2, 5, 9)].

In [10]:

# inspect the data used to build the context for the LLM responses
result.context_data["reports"]

# inspect the data used to build the context for the LLM responses result.context_data["reports"]

Out[10]:

	id	title	occurrence weight	content	rank
0	4	Dulce Base and Operation: Dulce Community	1.0	# Dulce Base and Operation: Dulce Community\n\...	9.0
1	0	Paranormal Military Squad and Operation: Dulce	1.0	# Paranormal Military Squad and Operation: Dul...	8.5
2	3	Paranormal Military Squad and Operation: Dulce	1.0	# Paranormal Military Squad and Operation: Dul...	8.5
3	8	Paranormal Military Squad: Leadership and Scie...	1.0	# Paranormal Military Squad: Leadership and Sc...	8.0
4	9	Paranormal Military Squad and Operation: Dulce...	0.2	# Paranormal Military Squad and Operation: Dul...	8.5
5	1	Team of Agents Investigating Dulce Base	0.2	# Team of Agents Investigating Dulce Base\n\nT...	7.5
6	6	Dulce Base Mainframe Room Technology Network	0.2	# Dulce Base Mainframe Room Technology Network...	7.5
7	7	Paranormal Military Squad and Operation Dulce	0.4	# Paranormal Military Squad and Operation Dulc...	8.5
8	5	Dulce Base Operational Command: Briefing Room ...	0.4	# Dulce Base Operational Command: Briefing Roo...	7.5
9	2	Alien Technology Retrieval and Analysis at Dul...	0.2	# Alien Technology Retrieval and Analysis at D...	9.0

In [11]:

# inspect number of LLM calls and tokens
print(
    f"LLM calls: {result.llm_calls}. Prompt tokens: {result.prompt_tokens}. Output tokens: {result.output_tokens}."
)

# inspect number of LLM calls and tokens print( f"LLM calls: {result.llm_calls}. Prompt tokens: {result.prompt_tokens}. Output tokens: {result.output_tokens}." )

LLM calls: 3. Prompt tokens: 12934. Output tokens: 1567.