Global Search

In [1]:

# Copyright (c) 2024 Microsoft Corporation.
# Licensed under the MIT License.

# Copyright (c) 2024 Microsoft Corporation. # Licensed under the MIT License.

In [2]:

import os

import pandas as pd
from graphrag.query.indexer_adapters import (
    read_indexer_communities,
    read_indexer_entities,
    read_indexer_reports,
)
from graphrag.query.structured_search.global_search.community_context import (
    GlobalCommunityContext,
)
from graphrag.query.structured_search.global_search.search import GlobalSearch
from graphrag.tokenizer.get_tokenizer import get_tokenizer
from graphrag_llm.completion import create_completion
from graphrag_llm.config import ModelConfig

import os import pandas as pd from graphrag.query.indexer_adapters import ( read_indexer_communities, read_indexer_entities, read_indexer_reports, ) from graphrag.query.structured_search.global_search.community_context import ( GlobalCommunityContext, ) from graphrag.query.structured_search.global_search.search import GlobalSearch from graphrag.tokenizer.get_tokenizer import get_tokenizer from graphrag_llm.completion import create_completion from graphrag_llm.config import ModelConfig

Global Search example¶

Global search method generates answers by searching over all AI-generated community reports in a map-reduce fashion. This is a resource-intensive method, but often gives good responses for questions that require an understanding of the dataset as a whole (e.g. What are the most significant values of the herbs mentioned in this notebook?).

LLM setup¶

In [3]:

api_key = os.environ["GRAPHRAG_API_KEY"]

config = ModelConfig(
    type="litellm",
    model_provider="openai",
    model="gpt-4.1",
    api_key=api_key,
)
model = create_completion(config)

tokenizer = get_tokenizer(config)

api_key = os.environ["GRAPHRAG_API_KEY"] config = ModelConfig( type="litellm", model_provider="openai", model="gpt-4.1", api_key=api_key, ) model = create_completion(config) tokenizer = get_tokenizer(config)

Load community reports as context for global search¶

• Load all community reports in the community_reports table from GraphRAG, to be used as context data for global search.
• Load entities from the entities tables from GraphRAG, to be used for calculating community weights for context ranking. Note that this is optional (if no entities are provided, we will not calculate community weights and only use the rank attribute in the community reports table for context ranking)
• Load all communities in the communities table from the GraphRAG, to be used to reconstruct the community graph hierarchy for dynamic community selection.

In [4]:

# parquet files generated from indexing pipeline
INPUT_DIR = "./inputs/operation dulce"
COMMUNITY_TABLE = "communities"
COMMUNITY_REPORT_TABLE = "community_reports"
ENTITY_TABLE = "entities"

# community level in the Leiden community hierarchy from which we will load the community reports
# higher value means we use reports from more fine-grained communities (at the cost of higher computation cost)
COMMUNITY_LEVEL = 2

# parquet files generated from indexing pipeline INPUT_DIR = "./inputs/operation dulce" COMMUNITY_TABLE = "communities" COMMUNITY_REPORT_TABLE = "community_reports" ENTITY_TABLE = "entities" # community level in the Leiden community hierarchy from which we will load the community reports # higher value means we use reports from more fine-grained communities (at the cost of higher computation cost) COMMUNITY_LEVEL = 2

In [5]:

community_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet")
entity_df = pd.read_parquet(f"{INPUT_DIR}/{ENTITY_TABLE}.parquet")
report_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet")

communities = read_indexer_communities(community_df, report_df)
reports = read_indexer_reports(report_df, community_df, COMMUNITY_LEVEL)
entities = read_indexer_entities(entity_df, community_df, COMMUNITY_LEVEL)

print(f"Total report count: {len(report_df)}")
print(
    f"Report count after filtering by community level {COMMUNITY_LEVEL}: {len(reports)}"
)

report_df.head()

community_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet") entity_df = pd.read_parquet(f"{INPUT_DIR}/{ENTITY_TABLE}.parquet") report_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet") communities = read_indexer_communities(community_df, report_df) reports = read_indexer_reports(report_df, community_df, COMMUNITY_LEVEL) entities = read_indexer_entities(entity_df, community_df, COMMUNITY_LEVEL) print(f"Total report count: {len(report_df)}") print( f"Report count after filtering by community level {COMMUNITY_LEVEL}: {len(reports)}" ) report_df.head()

Total report count: 10
Report count after filtering by community level 2: 10

Out[5]:

	id	human_readable_id	community	level	parent	children	title	summary	full_content	rank	rating_explanation	findings	full_content_json	period	size
0	9785e64b429dacd70cce8238051f3438b387ebcd6920ef...	7	7	1	0	[]	Paranormal Military Squad and Operation Dulce	This community centers on the Paranormal Milit...	# Paranormal Military Squad and Operation Dulc...	8.5	The impact severity rating is high due to the ...	[{'explanation': 'The Paranormal Military Squa...	{\n    "title": "Paranormal Military Squad and...	2026-01-13	4
1	212c8ff1796bff731abe3d54f3216511ab15d5fb0d97eb...	8	8	1	0	[]	Paranormal Military Squad: Leadership and Scie...	This community centers on the Paranormal Milit...	# Paranormal Military Squad: Leadership and Sc...	8.0	The community poses a high impact due to its i...	[{'explanation': 'Taylor Cruz, often referred ...	{\n    "title": "Paranormal Military Squad: Le...	2026-01-13	3
2	dbff99eaacade8ac4337a9bde0f533313445f0643ab3d0...	9	9	1	0	[]	Paranormal Military Squad and Operation: Dulce...	This community centers around the Paranormal M...	# Paranormal Military Squad and Operation: Dul...	8.5	The community poses a high impact severity due...	[{'explanation': 'The military complex is the ...	{\n    "title": "Paranormal Military Squad and...	2026-01-13	3
3	b6e8ab1ac8ecc605bbec09ad78819b95ac08d62c5a9991...	0	0	0	-1	[7, 8, 9]	Paranormal Military Squad and Operation: Dulce	This community centers on the Paranormal Milit...	# Paranormal Military Squad and Operation: Dul...	8.5	The community poses a high impact due to its e...	[{'explanation': 'Taylor Cruz is the central a...	{\n    "title": "Paranormal Military Squad and...	2026-01-13	10
4	97038bee58306d13b709b615fb2b816cae0c11273a9e73...	1	1	0	-1	[]	Team of Agents Investigating Dulce Base	This community centers on a specialized team o...	# Team of Agents Investigating Dulce Base\n\nT...	7.5	The impact severity rating is high due to the ...	[{'explanation': 'The core of this community i...	{\n    "title": "Team of Agents Investigating ...	2026-01-13	5

Build global context based on community reports¶

In [6]:

context_builder = GlobalCommunityContext(
    community_reports=reports,
    communities=communities,
    entities=entities,  # default to None if you don't want to use community weights for ranking
    tokenizer=tokenizer,
)

context_builder = GlobalCommunityContext( community_reports=reports, communities=communities, entities=entities, # default to None if you don't want to use community weights for ranking tokenizer=tokenizer, )

Perform global search¶

In [7]:

context_builder_params = {
    "use_community_summary": False,  # False means using full community reports. True means using community short summaries.
    "shuffle_data": True,
    "include_community_rank": True,
    "min_community_rank": 0,
    "community_rank_name": "rank",
    "include_community_weight": True,
    "community_weight_name": "occurrence weight",
    "normalize_community_weight": True,
    "max_tokens": 12_000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)
    "context_name": "Reports",
}

map_llm_params = {
    "max_tokens": 1000,
    "temperature": 0.0,
}

reduce_llm_params = {
    "max_tokens": 2000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 1000-1500)
    "temperature": 0.0,
}

context_builder_params = { "use_community_summary": False, # False means using full community reports. True means using community short summaries. "shuffle_data": True, "include_community_rank": True, "min_community_rank": 0, "community_rank_name": "rank", "include_community_weight": True, "community_weight_name": "occurrence weight", "normalize_community_weight": True, "max_tokens": 12_000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000) "context_name": "Reports", } map_llm_params = { "max_tokens": 1000, "temperature": 0.0, } reduce_llm_params = { "max_tokens": 2000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 1000-1500) "temperature": 0.0, }

In [8]:

search_engine = GlobalSearch(
    model=model,
    context_builder=context_builder,
    tokenizer=tokenizer,
    max_data_tokens=12_000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)
    map_llm_params=dict(map_llm_params),
    reduce_llm_params=dict(reduce_llm_params),
    allow_general_knowledge=False,  # set this to True will add instruction to encourage the LLM to incorporate general knowledge in the response, which may increase hallucinations, but could be useful in some use cases.
    json_mode=False,
    context_builder_params=context_builder_params,
    concurrent_coroutines=32,
    response_type="multiple paragraphs",  # free form text describing the response type and format, can be anything, e.g. prioritized list, single paragraph, multiple paragraphs, multiple-page report
)

search_engine = GlobalSearch( model=model, context_builder=context_builder, tokenizer=tokenizer, max_data_tokens=12_000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000) map_llm_params=dict(map_llm_params), reduce_llm_params=dict(reduce_llm_params), allow_general_knowledge=False, # set this to True will add instruction to encourage the LLM to incorporate general knowledge in the response, which may increase hallucinations, but could be useful in some use cases. json_mode=False, context_builder_params=context_builder_params, concurrent_coroutines=32, response_type="multiple paragraphs", # free form text describing the response type and format, can be anything, e.g. prioritized list, single paragraph, multiple paragraphs, multiple-page report )

In [9]:

result = await search_engine.search("What is operation dulce?")

print(result.response)

result = await search_engine.search("What is operation dulce?") print(result.response)

## Overview of Operation Dulce

Operation Dulce is a classified, high-stakes mission led by the Paranormal Military Squad, an elite unit specializing in the investigation and management of anomalous phenomena and existential threats. The primary objective of the operation is to investigate the enigmatic Dulce Base, a secretive and technologically advanced underground facility located beneath Dulce. This mission is considered both career-defining for those involved and potentially humanity-altering, due to the profound implications of its findings [Data: Reports (4, 7)].

## Mission Objectives and Scope

The core aims of Operation Dulce include:

- **Uncovering Hidden Technologies:** The operation seeks to identify and analyze advanced, possibly alien, technologies housed within the Dulce Base. This includes the retrieval and study of enigmatic alien technology discovered at a crash site and subsequently brought to the base for further examination. The implications of these findings are significant for both scientific advancement and security [Data: Reports (2, 4)].
- **Investigating Anomalous Phenomena:** The mission involves confronting and managing existential threats and anomalous phenomena, which may challenge conventional understandings of reality and security [Data: Reports (4, 7)].
- **Reconnaissance and Rescue:** The operation includes objectives such as reconnaissance of the base, rescue missions, and the investigation of advanced surveillance systems and electronic anomalies. The team must navigate both psychological and physical dangers, leveraging scientific and technical expertise [Data: Reports (1, 3, 4, +more)].

## Operational Structure and Environment

Operation Dulce is characterized by a hierarchical, mission-centric structure with a clear chain of command. The Paranormal Military Squad operates from a secure military complex, with the operational hub being a high-security briefing room within the Dulce Base itself. This facility supports mission planning and execution, emphasizing security, technological capability, and operational readiness [Data: Reports (5, 7)].

Key features of the operation’s environment and structure include:

- **Advanced Equipment and Infrastructure:** The team utilizes advanced equipment such as encrypted radio transmitters, projectors, and monitors to support secure communications, planning, and data analysis [Data: Reports (0, 5, 9, +more)].
- **Strict Protocols and Discipline:** The mission requires strict adherence to protocols, legal compliance, and operational discipline. Preparation and execution emphasize readiness to confront both conventional and unconventional threats [Data: Reports (0, 1, 9, +more)].
- **Team Coordination:** The operation relies on a tightly coordinated team structure, integrating specialized agents with robust technical and scientific expertise [Data: Reports (0, 3, 8, +more)].

## Significance and Potential Impact

The potential impact of Operation Dulce is substantial. The findings from the mission may reshape understandings of security, technology, and possibly the nature of reality itself, given the base’s reputation for housing cosmic secrets and existential threats. The operation is seen as a pivotal event with far-reaching consequences for both the individuals involved and broader humanity [Data: Reports (4)].

## Summary

In summary, Operation Dulce is a top-secret mission centered on the investigation of the Dulce Base by the Paranormal Military Squad. It involves the exploration and analysis of advanced alien technology, management of anomalous phenomena, and the navigation of significant psychological and physical dangers. The operation is marked by its complexity, strict protocols, advanced technical preparation, and the potential to fundamentally alter scientific and security paradigms [Data: Reports (0, 1, 2, 3, 4, 5, 7, 8, 9, +more)].

In [10]:

# inspect the data used to build the context for the LLM responses
result.context_data["reports"]

# inspect the data used to build the context for the LLM responses result.context_data["reports"]

Out[10]:

	id	title	occurrence weight	content	rank
0	4	Dulce Base and Operation: Dulce Community	1.0	# Dulce Base and Operation: Dulce Community\n\...	9.0
1	0	Paranormal Military Squad and Operation: Dulce	1.0	# Paranormal Military Squad and Operation: Dul...	8.5
2	3	Paranormal Military Squad and Operation: Dulce	1.0	# Paranormal Military Squad and Operation: Dul...	8.5
3	8	Paranormal Military Squad: Leadership and Scie...	1.0	# Paranormal Military Squad: Leadership and Sc...	8.0
4	9	Paranormal Military Squad and Operation: Dulce...	0.2	# Paranormal Military Squad and Operation: Dul...	8.5
5	1	Team of Agents Investigating Dulce Base	0.2	# Team of Agents Investigating Dulce Base\n\nT...	7.5
6	6	Dulce Base Mainframe Room Technology Network	0.2	# Dulce Base Mainframe Room Technology Network...	7.5
7	7	Paranormal Military Squad and Operation Dulce	0.4	# Paranormal Military Squad and Operation Dulc...	8.5
8	5	Dulce Base Operational Command: Briefing Room ...	0.4	# Dulce Base Operational Command: Briefing Roo...	7.5
9	2	Alien Technology Retrieval and Analysis at Dul...	0.2	# Alien Technology Retrieval and Analysis at D...	9.0

In [11]:

# inspect number of LLM calls and tokens
print(
    f"LLM calls: {result.llm_calls}. Prompt tokens: {result.prompt_tokens}. Output tokens: {result.output_tokens}."
)

# inspect number of LLM calls and tokens print( f"LLM calls: {result.llm_calls}. Prompt tokens: {result.prompt_tokens}. Output tokens: {result.output_tokens}." )

LLM calls: 3. Prompt tokens: 12906. Output tokens: 1534.