How to Reproduce LLM-Based Question Generation for Query-Focused Summarization from the Graph RAG Paper?

[perfeus]

Hi everyone,

In the paper "From Local to Global: A Graph RAG Approach to Query-Focused Summarization", the authors describe a method for generating evaluation questions by prompting an LLM with a dataset description. Specifically, they:

• Identify N potential users of the dataset
• Generate N tasks per user
• Then create N questions per (user, task) pair that require global understanding of the dataset, not just fact retrieval.

The goal is to simulate real-world sensemaking rather than simple fact lookup. I'm trying to reproduce this pipeline for my own dataset.

Could anyone:

• Provide examples of how the original paper formulated the LLM prompts or questions?
• Share a Python code example to generate these questions automatically based on a dataset description (similar to the one used in the paper)?

I'd love to see a full example that shows how to:

1. Input a short dataset description
2. Prompt the LLM to generate users, tasks, and questions
3. Output the questions grouped by (user, task) in a structured format.

Thanks a lot in advance! 🚀

[Juyel (juyelhushen)]

A practical way to reproduce this is to treat the process as a three-stage generation pipeline: dataset description → personas → tasks → global questions.

For example, you can structure the prompts so that each stage produces JSON that becomes the input to the next stage:

import json
from openai import OpenAI

client = OpenAI()

dataset_description = """
A collection of internal company documents containing information about
products, engineering initiatives, customer feedback, business performance,
and strategic priorities over several years.
"""

def ask_llm(prompt):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.7,
    )
    return response.choices[0].message.content


# 1. Generate potential users/personas
users = ask_llm(f"""
Given the following dataset description:

{dataset_description}

Identify 5 realistic types of users who might need to perform
sensemaking over this dataset.

For each user provide:
- user
- role
- information_needs

Return JSON only.
""")

# 2. Generate tasks for each persona
tasks = ask_llm(f"""
Dataset:
{dataset_description}

Users:
{users}

For each user, generate 3 realistic analytical/sensemaking tasks.

The tasks should require understanding information across the dataset,
rather than retrieving a single fact.

Return JSON grouped by user.
""")

# 3. Generate questions for each user/task pair
questions = ask_llm(f"""
Dataset:
{dataset_description}

Users:
{users}

Tasks:
{tasks}

For every (user, task) pair, generate 3 questions that require
GLOBAL understanding of the dataset.

Avoid questions that can be answered by retrieving one isolated fact.
Questions should require synthesizing information from multiple
documents or topics.

Return JSON in this structure:

[
  {{
    "user": "...",
    "task": "...",
    "questions": [
      "...",
      "...",
      "..."
    ]
  }}
]
""")

print(json.dumps(json.loads(questions), indent=2))

The important part is the distinction between fact retrieval and global sensemaking. For example:

❌ "What was the revenue in 2024?"

This can be answered from a single piece of information.

✅ "How did the company's product strategy evolve between 2022 and 2024, and what major business or customer signals appear to have influenced those changes?"

This requires synthesizing information from multiple parts of the dataset.

For a more faithful reproduction of the paper's evaluation setup, I'd also keep the generated users/tasks/questions as a separate evaluation dataset rather than mixing them into the GraphRAG indexing corpus. Then you can run the questions through your GraphRAG system and evaluate the generated answers separately.