Upload 42 files

.gitattributes

@@ -1,35 +1,8 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.ipynb linguist-vendoredtests/pdfs/2023-annual-report-truncated.pdf filter=lfs diff=lfs merge=lfs -text
+tests/pdfs/2023-annual-report.pdf filter=lfs diff=lfs merge=lfs -text
+tests/pdfs/earthmover.pdf filter=lfs diff=lfs merge=lfs -text
+tests/pdfs/four-lectures.pdf filter=lfs diff=lfs merge=lfs -text
+tests/pdfs/PRML.pdf filter=lfs diff=lfs merge=lfs -text
+tests/pdfs/q1-fy25-earnings.pdf filter=lfs diff=lfs merge=lfs -text
+tests/pdfs/Regulation[[:space:]]Best[[:space:]]Interest_Interpretive[[:space:]]release.pdf filter=lfs diff=lfs merge=lfs -text
+tests/pdfs/Regulation[[:space:]]Best[[:space:]]Interest_proposed[[:space:]]rule.pdf filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,15 @@
+.ipynb_checkpoints
+__pycache__
+files
+index
+temp/*
+chroma-collections.parquet
+chroma-embeddings.parquet
+.DS_Store
+.env*
+notebook
+SDK/*
+log/*
+logs/
+parts/*
+json_results/*

CHANGELOG.md

@@ -0,0 +1,16 @@
+# Change Log
+All notable changes to this project will be documented in this file.
+
+## Beta - 2025-04-23
+
+### Fixed
+- [x] Fixed a bug introduced on April 18 where `start_index` was incorrectly passed.
+
+## Beta - 2025-04-03
+
+### Added
+- [x] Add node_id, node summary
+- [x] Add document discription
+
+### Changed
+- [x] Change "child_nodes" -> "nodes" to simplify the structure

Dockerfile

@@ -0,0 +1,23 @@
+
+FROM python:3.10-slim
+
+WORKDIR /app
+
+# Copy requirements first to leverage Docker cache
+COPY requirements.txt .
+
+# Install dependencies
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy the rest of the application
+COPY . .
+
+# Expose the port used by Gradio
+EXPOSE 7860
+
+# Set environment variables if needed (though secrets should be set in HF Space settings)
+# ENV NVIDIA_API_KEY=...
+# ENV MISTRAL_API_KEY=...
+
+# Run the application
+CMD ["python", "app.py"]

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Vectify AI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,11 +1,261 @@
----
-title: PageIndexAPI
-emoji: 📚
-colorFrom: red
-colorTo: blue
-sdk: docker
-pinned: false
-license: apache-2.0
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+---
+title: PageIndex RAG
+emoji: 📑
+colorFrom: blue
+colorTo: indigo
+sdk: docker
+app_file: app.py
+pinned: false
+---
+
+<div align="center">
+  
+<a href="https://vectify.ai/pageindex" target="_blank">
+  <img src="https://github.com/user-attachments/assets/46201e72-675b-43bc-bfbd-081cc6b65a1d" alt="PageIndex Banner" />
+</a>
+
+<br/>
+<br/>
+
+<p align="center">
+  <a href="https://trendshift.io/repositories/14736" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14736" alt="VectifyAI%2FPageIndex | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
+</p>
+
+# PageIndex: Vectorless, Reasoning-based RAG
+
+<p align="center"><b>Reasoning-based RAG&nbsp; ◦ &nbsp;No Vector DB&nbsp; ◦ &nbsp;No Chunking&nbsp; ◦ &nbsp;Human-like Retrieval</b></p>
+
+<h4 align="center">
+  <a href="https://vectify.ai">🏠 Homepage</a>&nbsp; • &nbsp;
+  <a href="https://chat.pageindex.ai">🖥️ Chat Platform</a>&nbsp; • &nbsp;
+  <a href="https://pageindex.ai/mcp">🔌 MCP</a>&nbsp; • &nbsp;
+  <a href="https://docs.pageindex.ai">📚 Docs</a>&nbsp; • &nbsp;
+  <a href="https://discord.com/invite/VuXuf29EUj">💬 Discord</a>&nbsp; • &nbsp;
+  <a href="https://ii2abc2jejf.typeform.com/to/tK3AXl8T">✉️ Contact</a>&nbsp;
+</h4>
+  
+</div>
+
+
+<details open>
+<summary><h3>📢 Latest Updates</h3></summary>
+
+ **🔥 Releases:**
+- [**PageIndex Chat**](https://chat.pageindex.ai): The first human-like document-analysis agent [platform](https://chat.pageindex.ai) built for professional long documents. Can also be integrated via [MCP](https://pageindex.ai/mcp) or [API](https://docs.pageindex.ai/quickstart) (beta).
+<!-- - [**PageIndex Chat API**](https://docs.pageindex.ai/quickstart): An API that brings PageIndex's advanced long-document intelligence directly into your applications and workflows. -->
+<!-- - [PageIndex MCP](https://pageindex.ai/mcp): Bring PageIndex into Claude, Cursor, or any MCP-enabled agent. Chat with long PDFs in a reasoning-based, human-like way. -->
+ 
+ **📝 Articles:**
+- [**PageIndex Framework**](https://pageindex.ai/blog/pageindex-intro): Introduces the PageIndex framework — an *agentic, in-context* *tree index* that enables LLMs to perform *reasoning-based*, *human-like retrieval* over long documents, without vector DB or chunking.
+<!-- - [Do We Still Need OCR?](https://pageindex.ai/blog/do-we-need-ocr): Explores how vision-based, reasoning-native RAG challenges the traditional OCR pipeline, and why the future of document AI might be *vectorless* and *vision-based*. -->
+
+ **🧪 Cookbooks:**
+- [Vectorless RAG](https://docs.pageindex.ai/cookbook/vectorless-rag-pageindex): A minimal, hands-on example of reasoning-based RAG using PageIndex. No vectors, no chunking, and human-like retrieval.
+- [Vision-based Vectorless RAG](https://docs.pageindex.ai/cookbook/vision-rag-pageindex): OCR-free, vision-only RAG with PageIndex's reasoning-native retrieval workflow that works directly over PDF page images.
+</details>
+
+---
+
+# 📑 Introduction to PageIndex
+
+Are you frustrated with vector database retrieval accuracy for long professional documents? Traditional vector-based RAG relies on semantic *similarity* rather than true *relevance*. But **similarity ≠ relevance** — what we truly need in retrieval is **relevance**, and that requires **reasoning**. When working with professional documents that demand domain expertise and multi-step reasoning, similarity search often falls short.
+
+Inspired by AlphaGo, we propose **[PageIndex](https://vectify.ai/pageindex)** — a **vectorless**, **reasoning-based RAG** system that builds a **hierarchical tree index** from long documents and uses LLMs to **reason** *over that index* for **agentic, context-aware retrieval**.
+It simulates how *human experts* navigate and extract knowledge from complex documents through *tree search*, enabling LLMs to *think* and *reason* their way to the most relevant document sections. PageIndex performs retrieval in two steps:
+
+1. Generate a “Table-of-Contents” **tree structure index** of documents
+2. Perform reasoning-based retrieval through **tree search**
+
+<div align="center">
+  <a href="https://pageindex.ai/blog/pageindex-intro" target="_blank" title="The PageIndex Framework">
+    <img src="https://docs.pageindex.ai/images/cookbook/vectorless-rag.png" width="70%">
+  </a>
+</div>
+
+### 🎯 Core Features 
+
+Compared to traditional vector-based RAG, **PageIndex** features:
+- **No Vector DB**: Uses document structure and LLM reasoning for retrieval, instead of vector similarity search.
+- **No Chunking**: Documents are organized into natural sections, not artificial chunks.
+- **Human-like Retrieval**: Simulates how human experts navigate and extract knowledge from complex documents.
+- **Better Explainability and Traceability**: Retrieval is based on reasoning — traceable and interpretable, with page and section references. No more opaque, approximate vector search (“vibe retrieval”).
+
+PageIndex powers a reasoning-based RAG system that achieved **state-of-the-art** [98.7% accuracy](https://github.com/VectifyAI/Mafin2.5-FinanceBench) on FinanceBench, demonstrating superior performance over vector-based RAG solutions in professional document analysis (see our [blog post](https://vectify.ai/blog/Mafin2.5) for details).
+
+### 📍 Explore PageIndex
+
+To learn more, please see a detailed introduction of the [PageIndex framework](https://pageindex.ai/blog/pageindex-intro). Check out this GitHub repo for open-source code, and the [cookbooks](https://docs.pageindex.ai/cookbook), [tutorials](https://docs.pageindex.ai/tutorials), and [blog](https://pageindex.ai/blog) for additional usage guides and examples. 
+
+The PageIndex service is available as a ChatGPT-style [chat platform](https://chat.pageindex.ai), or can be integrated via [MCP](https://pageindex.ai/mcp) or [API](https://docs.pageindex.ai/quickstart).
+
+### 🛠️ Deployment Options
+- Self-host — run locally with this open-source repo.
+- Cloud Service — try instantly with our [Chat Platform](https://chat.pageindex.ai/), or integrate with [MCP](https://pageindex.ai/mcp) or [API](https://docs.pageindex.ai/quickstart).
+- _Enterprise_ — private or on-prem deployment. [Contact us](https://ii2abc2jejf.typeform.com/to/tK3AXl8T) or [book a demo](https://calendly.com/pageindex/meet) for more details.
+
+### 🧪 Quick Hands-on
+
+- Try the [**Vectorless RAG**](https://github.com/VectifyAI/PageIndex/blob/main/cookbook/pageindex_RAG_simple.ipynb) notebook — a *minimal*, hands-on example of reasoning-based RAG using PageIndex.
+- Experiment with [*Vision-based Vectorless RAG*](https://github.com/VectifyAI/PageIndex/blob/main/cookbook/vision_RAG_pageindex.ipynb) — no OCR; a minimal, reasoning-native RAG pipeline that works directly over page images.
+  
+<div align="center">
+  <a href="https://colab.research.google.com/github/VectifyAI/PageIndex/blob/main/cookbook/pageindex_RAG_simple.ipynb" target="_blank" rel="noopener">
+    <img src="https://img.shields.io/badge/Open_In_Colab-Vectorless_RAG-orange?style=for-the-badge&logo=googlecolab" alt="Open in Colab: Vectorless RAG" />
+  </a>
+  &nbsp;&nbsp;
+  <a href="https://colab.research.google.com/github/VectifyAI/PageIndex/blob/main/cookbook/vision_RAG_pageindex.ipynb" target="_blank" rel="noopener">
+    <img src="https://img.shields.io/badge/Open_In_Colab-Vision_RAG-orange?style=for-the-badge&logo=googlecolab" alt="Open in Colab: Vision RAG" />
+  </a>
+</div>
+
+---
+
+# 🌲 PageIndex Tree Structure
+PageIndex can transform lengthy PDF documents into a semantic **tree structure**, similar to a _"table of contents"_ but optimized for use with Large Language Models (LLMs). It's ideal for: financial reports, regulatory filings, academic textbooks, legal or technical manuals, and any document that exceeds LLM context limits.
+
+Below is an example PageIndex tree structure. Also see more example [documents](https://github.com/VectifyAI/PageIndex/tree/main/tests/pdfs) and generated [tree structures](https://github.com/VectifyAI/PageIndex/tree/main/tests/results).
+
+```jsonc
+...
+{
+  "title": "Financial Stability",
+  "node_id": "0006",
+  "start_index": 21,
+  "end_index": 22,
+  "summary": "The Federal Reserve ...",
+  "nodes": [
+    {
+      "title": "Monitoring Financial Vulnerabilities",
+      "node_id": "0007",
+      "start_index": 22,
+      "end_index": 28,
+      "summary": "The Federal Reserve's monitoring ..."
+    },
+    {
+      "title": "Domestic and International Cooperation and Coordination",
+      "node_id": "0008",
+      "start_index": 28,
+      "end_index": 31,
+      "summary": "In 2023, the Federal Reserve collaborated ..."
+    }
+  ]
+}
+...
+```
+
+You can generate the PageIndex tree structure with this open-source repo, or use our [API](https://docs.pageindex.ai/quickstart) 
+
+---
+
+# ⚙️ Package Usage
+
+You can follow these steps to generate a PageIndex tree from a PDF document.
+
+### 1. Install dependencies
+
+```bash
+pip3 install --upgrade -r requirements.txt
+```
+
+### 2. Set your OpenAI API key
+
+Create a `.env` file in the root directory and add your API key:
+
+```bash
+CHATGPT_API_KEY=your_openai_key_here
+```
+
+### 3. Run PageIndex on your PDF
+
+```bash
+python3 run_pageindex.py --pdf_path /path/to/your/document.pdf
+```
+
+<details>
+<summary><strong>Optional parameters</strong></summary>
+<br>
+You can customize the processing with additional optional arguments:
+
+```
+--model                 OpenAI model to use (default: gpt-4o-2024-11-20)
+--toc-check-pages       Pages to check for table of contents (default: 20)
+--max-pages-per-node    Max pages per node (default: 10)
+--max-tokens-per-node   Max tokens per node (default: 20000)
+--if-add-node-id        Add node ID (yes/no, default: yes)
+--if-add-node-summary   Add node summary (yes/no, default: yes)
+--if-add-doc-description Add doc description (yes/no, default: yes)
+```
+</details>
+
+<details>
+<summary><strong>Markdown support</strong></summary>
+<br>
+We also provide markdown support for PageIndex. You can use the `-md_path` flag to generate a tree structure for a markdown file.
+
+```bash
+python3 run_pageindex.py --md_path /path/to/your/document.md
+```
+
+> Note: in this function, we use "#" to determine node heading and their levels. For example, "##" is level 2, "###" is level 3, etc. Make sure your markdown file is formatted correctly. If your Markdown file was converted from a PDF or HTML, we don't recommend using this function, since most existing conversion tools cannot preserve the original hierarchy. Instead, use our [PageIndex OCR](https://pageindex.ai/blog/ocr), which is designed to preserve the original hierarchy, to convert the PDF to a markdown file and then use this function.
+</details>
+
+<!-- 
+# ☁️ Improved Tree Generation with PageIndex OCR
+
+This repo is designed for generating PageIndex tree structure for simple PDFs, but many real-world use cases involve complex PDFs that are hard to parse by classic Python tools. However, extracting high-quality text from PDF documents remains a non-trivial challenge. Most OCR tools only extract page-level content, losing the broader document context and hierarchy.
+
+To address this, we introduced PageIndex OCR — the first long-context OCR model designed to preserve the global structure of documents. PageIndex OCR significantly outperforms other leading OCR tools, such as those from Mistral and Contextual AI, in recognizing true hierarchy and semantic relationships across document pages.
+
+- Experience next-level OCR quality with PageIndex OCR at our [Dashboard](https://dash.pageindex.ai/).
+- Integrate PageIndex OCR seamlessly into your stack via our [API](https://docs.pageindex.ai/quickstart).
+
+<p align="center">
+  <img src="https://github.com/user-attachments/assets/eb35d8ae-865c-4e60-a33b-ebbd00c41732" width="80%">
+</p>
+-->
+
+---
+
+# 📈 Case Study: PageIndex Leads Finance QA Benchmark
+
+[Mafin 2.5](https://vectify.ai/mafin) is a reasoning-based RAG system for financial document analysis, powered by **PageIndex**. It achieved a state-of-the-art [**98.7% accuracy**](https://vectify.ai/blog/Mafin2.5) on the [FinanceBench](https://arxiv.org/abs/2311.11944) benchmark, significantly outperforming traditional vector-based RAG systems.
+
+PageIndex's hierarchical indexing and reasoning-driven retrieval enable precise navigation and extraction of relevant context from complex financial reports, such as SEC filings and earnings disclosures.
+
+Explore the full [benchmark results](https://github.com/VectifyAI/Mafin2.5-FinanceBench) and our [blog post](https://vectify.ai/blog/Mafin2.5) for detailed comparisons and performance metrics.
+
+<div align="center">
+  <a href="https://github.com/VectifyAI/Mafin2.5-FinanceBench">
+    <img src="https://github.com/user-attachments/assets/571aa074-d803-43c7-80c4-a04254b782a3" width="70%">
+  </a>
+</div>
+
+---
+
+# 🧭 Resources
+
+* 🧪 [Cookbooks](https://docs.pageindex.ai/cookbook/vectorless-rag-pageindex): hands-on, runnable examples and advanced use cases.
+* 📖 [Tutorials](https://docs.pageindex.ai/doc-search): practical guides and strategies, including *Document Search* and *Tree Search*.
+* 📝 [Blog](https://pageindex.ai/blog): technical articles, research insights, and product updates.
+* 🔌 [MCP setup](https://pageindex.ai/mcp#quick-setup) & [API docs](https://docs.pageindex.ai/quickstart): integration details and configuration options.
+
+---
+
+# ⭐ Support Us
+
+Leave us a star 🌟 if you like our project. Thank you!  
+
+<p>
+  <img src="https://github.com/user-attachments/assets/eae4ff38-48ae-4a7c-b19f-eab81201d794" width="80%">
+</p>
+
+### Connect with Us
+
+[![Twitter](https://img.shields.io/badge/Twitter-000000?style=for-the-badge&logo=x&logoColor=white)](https://x.com/PageIndexAI)&nbsp;
+[![LinkedIn](https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white)](https://www.linkedin.com/company/vectify-ai/)&nbsp;
+[![Discord](https://img.shields.io/badge/Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white)](https://discord.com/invite/VuXuf29EUj)&nbsp;
+[![Contact Us](https://img.shields.io/badge/Contact_Us-3B82F6?style=for-the-badge&logo=envelope&logoColor=white)](https://ii2abc2jejf.typeform.com/to/tK3AXl8T)
+
+---
+
+© 2025 [Vectify AI](https://vectify.ai)

app.py

@@ -0,0 +1,67 @@
+import gradio as gr
+import os
+from pageindex.core.tree_index import TreeIndex
+from llm_config import get_llm_client, get_model_name
+
+# Initialize clients (checking for environment variables)
+# We do this inside the function or globally, but for robustness inside function is safer if env vars change (less likely in HF Spaces)
+# However, initializing once is better for connection pooling if applicable. Let's do it inside for now to handle errors gracefully.
+
+def process_docling_and_chat(markdown_text, user_query):
+    if not markdown_text:
+        return "Please provide document markdown text."
+    if not user_query:
+        return "Please provide a query."
+
+    try:
+        # 1. Build the PageIndex Tree locally in the Space
+        tree = TreeIndex()
+        tree.build_from_markdown(markdown_text)
+        
+        # 2. Initialize the Navigator (The "Brain")
+        # Try Nvidia first, then Mistral
+        try:
+            client = get_llm_client(provider="nvidia")
+            model = get_model_name(provider="nvidia")
+            # Test connection simply or just proceed
+        except Exception as e:
+            print(f"Nvidia client failed: {e}. Falling back to Mistral.")
+            client = get_llm_client(provider="mistral")
+            model = get_model_name(provider="mistral")
+        
+        # 3. Perform Reasoning Search
+        # This uses the internal logic of the repo to navigate the tree
+        context = tree.reasoning_search(query=user_query, llm_client=client)
+        
+        # 4. Final Answer Extraction
+        # Using the same client for consistency
+        response = client.chat.completions.create(
+            model=model, 
+            messages=[
+                {"role": "system", "content": "You are a helpful assistant. Use the provided context to answer the user's query."},
+                {"role": "user", "content": f"Context:\n{context}\n\nQuery: {user_query}"}
+            ]
+        )
+        return response.choices[0].message.content
+        
+    except Exception as e:
+        return f"An error occurred: {str(e)}"
+
+# Gradio UI setup
+with gr.Blocks(title="Petromind AI - PageIndex RAG") as demo:
+    gr.Markdown("# Oil & Gas Report - PageIndex RAG")
+    gr.Markdown("Upload document content (markdown format) and ask questions to extract specific information using PageIndex reasoning.")
+    
+    with gr.Row():
+        with gr.Column(scale=1):
+            input_md = gr.Textbox(label="Paste Docling Markdown Here", lines=15, placeholder="# Document Title\n\n## Section 1\nContent...")
+        with gr.Column(scale=1):
+            query = gr.Textbox(label="What do you want to extract?", placeholder="e.g., What is the casing size?")
+            btn = gr.Button("Analyze", variant="primary")
+            output = gr.Textbox(label="Result", lines=10, interactive=False)
+    
+    btn.click(fn=process_docling_and_chat, inputs=[input_md, query], outputs=output)
+
+if __name__ == "__main__":
+    # Enable queue for concurrency
+    demo.queue().launch(server_name="0.0.0.0", server_port=7860) 

cookbook/README.md

@@ -0,0 +1,14 @@
+### 🧪 Cookbooks:
+
+* [**Vectorless RAG notebook**](https://github.com/VectifyAI/PageIndex/blob/main/cookbook/pageindex_RAG_simple.ipynb): A *minimal*, hands-on example of reasoning-based RAG using **PageIndex** — no vectors, no chunking, and human-like retrieval.
+* [Vision-based Vectorless RAG notebook](https://github.com/VectifyAI/PageIndex/blob/main/cookbook/vision_RAG_pageindex.ipynb): no OCR; reasoning-native RAG pipeline that retrieves and reasons directly over page images.
+
+<div align="center">
+  <a href="https://colab.research.google.com/github/VectifyAI/PageIndex/blob/main/cookbook/pageindex_RAG_simple.ipynb" target="_blank" rel="noopener">
+    <img src="https://img.shields.io/badge/Open_In_Colab-Vectorless_RAG-orange?style=for-the-badge&logo=googlecolab" alt="Open in Colab: Vectorless RAG" />
+  </a>
+  &nbsp;&nbsp;
+  <a href="https://colab.research.google.com/github/VectifyAI/PageIndex/blob/main/cookbook/vision_RAG_pageindex.ipynb" target="_blank" rel="noopener">
+    <img src="https://img.shields.io/badge/Open_In_Colab-Vision_RAG-orange?style=for-the-badge&logo=googlecolab" alt="Open in Colab: Vision RAG" />
+  </a>
+</div>

cookbook/agentic_retrieval.ipynb

@@ -0,0 +1,899 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "XTboY7brzyp2"
+      },
+      "source": [
+        "![pageindex_banner](https://pageindex.ai/static/images/pageindex_banner.jpg)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "EtjMbl9Pz3S-"
+      },
+      "source": [
+        "<p align=\"center\">Reasoning-based RAG&nbsp; ◦ &nbsp;No Vector DB&nbsp; ◦ &nbsp;No Chunking&nbsp; ◦ &nbsp;Human-like Retrieval</p>\n",
+        "\n",
+        "<p align=\"center\">\n",
+        "  <a href=\"https://vectify.ai\">🏠 Homepage</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://chat.pageindex.ai\">🖥️ Platform</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://docs.pageindex.ai/quickstart\">📚 API Docs</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://github.com/VectifyAI/PageIndex\">📦 GitHub</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://discord.com/invite/VuXuf29EUj\">💬 Discord</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://ii2abc2jejf.typeform.com/to/tK3AXl8T\">✉️ Contact</a>&nbsp;\n",
+        "</p>\n",
+        "\n",
+        "<div align=\"center\">\n",
+        "\n",
+        "[![Star us on GitHub](https://img.shields.io/github/stars/VectifyAI/PageIndex?style=for-the-badge&logo=github&label=⭐️%20Star%20Us)](https://github.com/VectifyAI/PageIndex) &nbsp;&nbsp; [![Follow us on X](https://img.shields.io/badge/Follow%20Us-000000?style=for-the-badge&logo=x&logoColor=white)](https://twitter.com/VectifyAI)\n",
+        "\n",
+        "</div>\n",
+        "\n",
+        "---\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bbC9uLWCz8zl"
+      },
+      "source": [
+        "# Agentic Retrieval with PageIndex Chat API\n",
+        "\n",
+        "Similarity-based RAG based on Vector-DB has shown big limitations in recent AI applications, reasoning-based or agentic retrieval has become important in current developments. However, unlike classic RAG pipeine with embedding input, top-K chunks returns, re-rank, what should a agentic-native retreival API looks like?\n",
+        "\n",
+        "For an agentic-native retrieval system, we need the ability to prompt for retrieval just as naturally as you interact with ChatGPT. Below, we provide an example of how the PageIndex Chat API enables this style of prompt-driven retrieval.\n",
+        "\n",
+        "\n",
+        "## PageIndex Chat API\n",
+        "[PageIndex Chat](https://chat.pageindex.ai/) is a AI assistant that allow you chat with multiple super-long documents without worrying about limited context or context rot problem. It is based on [PageIndex](https://pageindex.ai/blog/pageindex-intro), a vectorless reasoning-based RAG framework which gives more transparent and reliable results like a human expert.\n",
+        "<div align=\"center\">\n",
+        "  <img src=\"https://docs.pageindex.ai/images/cookbook/vectorless-rag.png\" width=\"70%\">\n",
+        "</div>\n",
+        "\n",
+        "You can now access PageIndex Chat with API or SDK.\n",
+        "\n",
+        "## 📝 Notebook Overview\n",
+        "\n",
+        "This notebook demonstrates a simple, minimal example of agentic retrieval with PageIndex. You will learn:\n",
+        "- [x] How to use PageIndex Chat API.\n",
+        "- [x] How to prompt the PageIndex Chat to make it a retrieval system"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "77SQbPoe-LTN"
+      },
+      "source": [
+        "### Install PageIndex SDK"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 36,
+      "metadata": {
+        "id": "6Eiv_cHf0OXz"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install -q --upgrade pageindex"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "UR9-qkdD-Om7"
+      },
+      "source": [
+        "### Setup PageIndex"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 60,
+      "metadata": {
+        "id": "AFzsW4gq0fjh"
+      },
+      "outputs": [],
+      "source": [
+        "from pageindex import PageIndexClient\n",
+        "\n",
+        "# Get your PageIndex API key from https://dash.pageindex.ai/api-keys\n",
+        "PAGEINDEX_API_KEY = \"YOUR_PAGEINDEX_API_KEY\"\n",
+        "pi_client = PageIndexClient(api_key=PAGEINDEX_API_KEY)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "uvzf9oWL-Ts9"
+      },
+      "source": [
+        "### Upload a document"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 39,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "qf7sNRoL0hGw",
+        "outputId": "529f53c1-c827-45a7-cf01-41f567d4feaa"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Downloaded https://arxiv.org/pdf/2507.13334.pdf\n",
+            "Document Submitted: pi-cmi34m6jy01sg0bqzofch62n8\n"
+          ]
+        }
+      ],
+      "source": [
+        "import os, requests\n",
+        "\n",
+        "pdf_url = \"https://arxiv.org/pdf/2507.13334.pdf\"\n",
+        "pdf_path = os.path.join(\"../data\", pdf_url.split('/')[-1])\n",
+        "os.makedirs(os.path.dirname(pdf_path), exist_ok=True)\n",
+        "\n",
+        "response = requests.get(pdf_url)\n",
+        "with open(pdf_path, \"wb\") as f:\n",
+        "    f.write(response.content)\n",
+        "print(f\"Downloaded {pdf_url}\")\n",
+        "\n",
+        "doc_id = pi_client.submit_document(pdf_path)[\"doc_id\"]\n",
+        "print('Document Submitted:', doc_id)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "U4hpLB4T-fCt"
+      },
+      "source": [
+        "### Check the processing status"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 61,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "PB1S_CWd2n87",
+        "outputId": "472a64ab-747d-469c-9e46-3329456df212"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "{'createdAt': '2025-11-16T08:36:41.177000',\n",
+            " 'description': 'This survey provides a comprehensive overview and taxonomy of '\n",
+            "                'Context Engineering for Large Language Models, covering '\n",
+            "                'foundational components, system implementations, evaluation '\n",
+            "                'methods, and future research directions.',\n",
+            " 'id': 'pi-cmi1gp1hg01t20do2l3bgzwz1',\n",
+            " 'name': '2507.13334_19.pdf',\n",
+            " 'pageNum': 166,\n",
+            " 'status': 'completed'}\n",
+            "\n",
+            " Document ready! (166 pages)\n"
+          ]
+        }
+      ],
+      "source": [
+        "from pprint import pprint\n",
+        "\n",
+        "doc_info = pi_client.get_document(doc_id)\n",
+        "pprint(doc_info)\n",
+        "\n",
+        "if doc_info['status'] == 'completed':\n",
+        "  print(f\"\\n Document ready! ({doc_info['pageNum']} pages)\")\n",
+        "elif doc_info['status'] == 'processing':\n",
+        "  print(\"\\n Document is still processing. Please wait and check again.\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "z1C9FOvO-p1m"
+      },
+      "source": [
+        "### Ask a question about this document"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 55,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "X3RbQvy_0nt7",
+        "outputId": "9bfb314b-24ad-4eb2-d26c-01be5728d3cc"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "I'll help you find the evaluation methods used in this paper. Let me start by examining the document structure to locate the relevant sections.{\"doc_name\": \"2507.13334_19.pdf\"}Perfect! I can see there's a dedicated section on \"Evaluation\" (node_id: 0015) that covers pages 45-50. Let me extract the content from those pages to get detailed information about the evaluation methods.{\"doc_name\": \"2507.13334_19.pdf\", \"pages\": \"45-50\"}Based on the comprehensive evaluation section of the paper, here are the **evaluation methods** used:\n",
+            "\n",
+            "## Evaluation Framework Overview\n",
+            "\n",
+            "The paper presents a comprehensive evaluation framework organized into **Component-Level Assessment** and **System-Level Integration Assessment**.\n",
+            "\n",
+            "### 1. **Component-Level Assessment (Intrinsic Evaluation)**\n",
+            "\n",
+            "#### Prompt Engineering Evaluation:\n",
+            "- **Semantic similarity metrics**\n",
+            "- **Response quality assessment**\n",
+            "- **Robustness testing** across diverse input variations\n",
+            "- **Contextual calibration** assessment\n",
+            "\n",
+            "#### Long Context Processing Evaluation:\n",
+            "- **\"Needle in a haystack\"** evaluation paradigm - tests models' ability to retrieve specific information embedded within long contexts\n",
+            "- **Multi-document reasoning tasks** - assess synthesis capabilities\n",
+            "- **Position interpolation techniques** evaluation\n",
+            "- **Information retention, positional bias, and reasoning coherence** metrics\n",
+            "\n",
+            "#### Self-Contextualization Evaluation:\n",
+            "- **Meta-learning assessments**\n",
+            "- **Adaptation speed measurements**\n",
+            "- **Consistency analysis** across multiple iterations\n",
+            "- Self-refinement frameworks: **Self-Refine, Reflexion, N-CRITICS**\n",
+            "- Performance improvements measured (~20% improvement with GPT-4)\n",
+            "\n",
+            "#### Structured/Relational Data Integration:\n",
+            "- **Knowledge graph traversal accuracy**\n",
+            "- **Table comprehension assessment**\n",
+            "- **Database query generation evaluation**\n",
+            "\n",
+            "### 2. **System-Level Integration Assessment (Extrinsic Evaluation)**\n",
+            "\n",
+            "#### Retrieval-Augmented Generation (RAG):\n",
+            "- **Precision, recall, relevance metrics**\n",
+            "- **Factual accuracy assessment**\n",
+            "- **Task decomposition accuracy**\n",
+            "- **Multi-plan selection effectiveness**\n",
+            "- Memory-augmented planning evaluation\n",
+            "\n",
+            "#### Memory Systems Evaluation:\n",
+            "- **LongMemEval benchmark** (500 curated questions covering):\n",
+            "  - Information extraction\n",
+            "  - Temporal reasoning\n",
+            "  - Multi-session reasoning\n",
+            "  - Knowledge updates\n",
+            "- Dedicated benchmarks: **NarrativeQA, QMSum, QuALITY, MEMENTO**\n",
+            "- Accuracy degradation tracking (~30% degradation in extended interactions)\n",
+            "\n",
+            "#### Tool-Integrated Reasoning:\n",
+            "- **MCP-RADAR framework** for standardized evaluation\n",
+            "- **Berkeley Function Calling Leaderboard (BFCL)** - 2,000 test cases\n",
+            "- **T-Eval** - 553 tool-use cases\n",
+            "- **API-Bank** - 73 APIs, 314 dialogues\n",
+            "- **ToolHop** - 995 queries, 3,912 tools\n",
+            "- **StableToolBench** for API instability\n",
+            "- **WebArena** and **Mind2Web** for web agents\n",
+            "- **VideoWebArena** for multimodal agents\n",
+            "- Metrics: tool selection accuracy, parameter extraction precision, execution success rates, error recovery\n",
+            "\n",
+            "#### Multi-Agent Systems:\n",
+            "- **Communication effectiveness metrics**\n",
+            "- **Coordination efficiency assessment**\n",
+            "- **Protocol adherence evaluation**\n",
+            "- **Task decomposition accuracy**\n",
+            "- **Emergent collaborative behaviors** assessment\n",
+            "- Context handling and transaction support evaluation\n",
+            "\n",
+            "### 3. **Emerging Evaluation Paradigms**\n",
+            "\n",
+            "#### Self-Refinement Evaluation:\n",
+            "- Iterative improvement assessment across multiple cycles\n",
+            "- Multi-dimensional feedback mechanisms\n",
+            "- Ensemble-based evaluation approaches\n",
+            "\n",
+            "#### Multi-Aspect Feedback:\n",
+            "- Correctness, relevance, clarity, and robustness dimensions\n",
+            "- Self-rewarding mechanisms for autonomous evolution\n",
+            "\n",
+            "#### Criticism-Guided Evaluation:\n",
+            "- Specialized critic models providing detailed feedback\n",
+            "- Fine-grained assessment of reasoning quality, factual accuracy, logical consistency\n",
+            "\n",
+            "### 4. **Safety and Robustness Assessment**\n",
+            "\n",
+            "- **Adversarial attack resistance testing**\n",
+            "- **Distribution shift evaluation**\n",
+            "- **Input perturbation testing**\n",
+            "- **Alignment assessment** (adherence to intended behaviors)\n",
+            "- **Graceful degradation strategies**\n",
+            "- **Error recovery protocols**\n",
+            "- **Long-term behavior consistency** evaluation\n",
+            "\n",
+            "### Key Benchmarks Mentioned:\n",
+            "- GAIA (general assistant tasks - 92% human vs 15% GPT-4 accuracy)\n",
+            "- GTA benchmark (GPT-4 <50% task completion vs 92% human)\n",
+            "- WebArena Leaderboard (with success rates ranging from 23.5% to 61.7%)\n",
+            "\n",
+            "### Challenges Identified:\n",
+            "- Traditional metrics (BLEU, ROUGE, perplexity) inadequate for complex systems\n",
+            "- Need for \"living\" benchmarks that co-evolve with AI capabilities\n",
+            "- Longitudinal evaluation frameworks for tracking memory fidelity over time\n",
+            "- Compositional generalization assessment\n",
+            "- Evaluation of \"unknown unknowns\" in multi-agent systems\n",
+            "\n",
+            "The paper emphasizes a **paradigm shift from static benchmarks to dynamic, holistic assessments** that evaluate not just task success but reasoning quality, robustness, and long-term autonomy."
+          ]
+        }
+      ],
+      "source": [
+        "query = \"What are the evaluation methods used in this paper?\"\n",
+        "\n",
+        "for chunk in pi_client.chat_completions(\n",
+        "    messages=[{\"role\": \"user\", \"content\": query}],\n",
+        "    doc_id=doc_id,\n",
+        "    stream=True\n",
+        "):\n",
+        "    print(chunk, end='', flush=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "RXrdnxv92LY1"
+      },
+      "source": [
+        "## Agentic Retrieval with PageIndex Chat API\n",
+        "\n",
+        "You can now easily prompt the PageIndex Chat API to be a retrieval assistant."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 62,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "puyOGkAi0wrH",
+        "outputId": "b08c682f-abfb-4381-8bb0-726d210120ad"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "I'll help you find the evaluation methods discussed in this paper. Let me start by examining the document structure to locate relevant sections.{\"doc_name\": \"2507.13334_22.pdf\"}Now I can see the document structure. Section 6 is dedicated to \"Evaluation\" and spans pages 45-50. Let me retrieve the content from those pages to get the detailed evaluation methods.{\"doc_name\": \"2507.13334_22.pdf\", \"pages\": \"45-50\"}Perfect! I have retrieved the comprehensive evaluation section. Now let me compile the evaluation methods in the requested JSON format.\n",
+            "\n",
+            "```json\n",
+            "[\n",
+            "  {\n",
+            "    \"page\": 45,\n",
+            "    \"content\": \"## 6. Evaluation\\n\\nThe evaluation of context-engineered systems presents unprecedented challenges that transcend traditional language model assessment paradigms. These systems exhibit complex, multi-component architectures with dynamic, context-dependent behaviors requiring comprehensive evaluation frameworks that assess component-level diagnostics, task-based performance, and overall system robustness [841, 1141].\\n\\nThe heterogeneous nature of context engineering components-spanning retrieval mechanisms, memory systems, reasoning chains, and multi-agent coordination-demands evaluation methodologies that can capture both individual component effectiveness and emergent system-level behaviors [314, 939].\\n\\n### 6.1. Evaluation Frameworks and Methodologies\\n\\nThis subsection presents comprehensive approaches for evaluating both individual components and integrated systems in context engineering.\\n\\n#### 6.1.1. Component-Level Assessment\\n\\nIntrinsic evaluation focuses on the performance of individual components in isolation, providing foundational insights into system capabilities and failure modes.\\n\\nFor prompt engineering components, evaluation encompasses prompt effectiveness measurement through semantic similarity metrics, response quality assessment, and robustness testing across diverse input variations. Current approaches reveal brittleness and robustness challenges in prompt design, necessitating more sophisticated evaluation frameworks that can assess contextual calibration and adaptive prompt optimization $[1141,669]$.\"\n",
+            "  },\n",
+            "  {\n",
+            "    \"page\": 46,\n",
+            "    \"content\": \"Long context processing evaluation requires specialized metrics addressing information retention, positional bias, and reasoning coherence across extended sequences. The \\\"needle in a haystack\\\" evaluation paradigm tests models' ability to retrieve specific information embedded within long contexts, while multi-document reasoning tasks assess synthesis capabilities across multiple information sources. Position interpolation techniques and ultra-long sequence processing methods face significant computational challenges that limit practical evaluation scenarios [737, 299].\\n\\nSelf-contextualization mechanisms undergo evaluation through meta-learning assessments, adaptation speed measurements, and consistency analysis across multiple iterations. Self-refinement frameworks including Self-Refine, Reflexion, and N-CRITICS demonstrate substantial performance improvements, with GPT-4 achieving approximately 20\\\\% improvement through iterative self-refinement processes [741, 964, 795]. Multi-dimensional feedback mechanisms and ensemble-based evaluation approaches provide comprehensive assessment of autonomous evolution capabilities [583, 710].\\n\\nStructured and relational data integration evaluation examines accuracy in knowledge graph traversal, table comprehension, and database query generation. However, current evaluation frameworks face significant limitations in assessing structural reasoning capabilities, with high-quality structured training data development presenting ongoing challenges. LSTM-based models demonstrate increased errors when sequential and structural information conflict, highlighting the need for more sophisticated benchmarks testing structural understanding $[769,674,167]$.\\n\\n#### 6.1.2. System-Level Integration Assessment\\n\\nExtrinsic evaluation measures end-to-end performance on downstream tasks, providing holistic assessments of system utility through comprehensive benchmarks spanning question answering, reasoning, and real-world applications.\\n\\nSystem-level evaluation must capture emergent behaviors arising from component interactions, including synergistic effects where combined components exceed individual performance and potential interference patterns where component integration degrades overall effectiveness [841, 1141].\\n\\nRetrieval-Augmented Generation evaluation encompasses both retrieval quality and generation effectiveness through comprehensive metrics addressing precision, recall, relevance, and factual accuracy. Agentic RAG systems introduce additional complexity requiring evaluation of task decomposition accuracy, multi-plan selection effectiveness, and memory-augmented planning capabilities. Self-reflection mechanisms demonstrate iterative improvement through feedback loops, with MemoryBank implementations incorporating Ebbinghaus Forgetting Curve principles for enhanced memory evaluation [444, 166, 1372, 1192, 41].\\n\\nMemory systems evaluation encounters substantial difficulties stemming from the absence of standardized assessment frameworks and the inherently stateless characteristics of contemporary LLMs. LongMemEval offers 500 carefully curated questions that evaluate fundamental capabilities encompassing information extraction, temporal reasoning, multi-session reasoning, and knowledge updates. Commercial AI assistants exhibit $30 \\\\%$ accuracy degradation throughout extended interactions, underscoring significant deficiencies in memory persistence and retrieval effectiveness [1340, 1180, 463, 847, 390]. Dedicated benchmarks such as NarrativeQA, QMSum, QuALITY, and MEMENTO tackle episodic memory evaluation challenges [556, 572].\\n\\nTool-integrated reasoning systems require comprehensive evaluation covering the entire interaction trajectory, including tool selection accuracy, parameter extraction precision, execution success rates, and error recovery capabilities. The MCP-RADAR framework provides standardized evaluation employing objective metrics for software engineering and mathematical reasoning domains. Real-world evaluation reveals\"\n",
+            "  },\n",
+            "  {\n",
+            "    \"page\": 47,\n",
+            "    \"content\": \"significant performance gaps, with GPT-4 completing less than 50\\\\% of tasks in the GTA benchmark, compared to human performance of $92 \\\\%$ [314, 1098, 126, 939]. Advanced benchmarks including BFCL (2,000 testing cases), T-Eval (553 tool-use cases), API-Bank (73 APIs, 314 dialogues), and ToolHop ( 995 queries, 3,912 tools) address multi-turn interactions and nested tool calling scenarios [263, 363, 377, 1264, 160, 835].\\n\\nMulti-agent systems evaluation captures communication effectiveness, coordination efficiency, and collective outcome quality through specialized metrics addressing protocol adherence, task decomposition accuracy, and emergent collaborative behaviors. Contemporary orchestration frameworks including LangGraph, AutoGen, and CAMEL demonstrate insufficient transaction support, with validation limitations emerging as systems rely exclusively on LLM self-validation capabilities without independent validation procedures. Context handling failures compound challenges as agents struggle with long-term context maintenance encompassing both episodic and semantic information [128, 394, 901].\\n\\n### 6.2. Benchmark Datasets and Evaluation Paradigms\\n\\nThis subsection reviews specialized benchmarks and evaluation paradigms designed for assessing context engineering system performance.\\n\\n#### 6.2.1. Foundational Component Benchmarks\\n\\nLong context processing evaluation employs specialized benchmark suites designed to test information retention, reasoning, and synthesis across extended sequences. Current benchmarks face significant computational complexity challenges, with $\\\\mathrm{O}\\\\left(\\\\mathrm{n}^{2}\\\\right)$ scaling limitations in attention mechanisms creating substantial memory constraints for ultra-long sequences. Position interpolation and extension techniques require sophisticated evaluation frameworks that can assess both computational efficiency and reasoning quality across varying sequence lengths [737, 299, 1236].\\n\\nAdvanced architectures including LongMamba and specialized position encoding methods demonstrate promising directions for long context processing, though evaluation reveals persistent challenges in maintaining coherence across extended sequences. The development of sliding attention mechanisms and memory-efficient implementations requires comprehensive benchmarks that can assess both computational tractability and task performance [1267, 351].\\n\\nStructured and relational data integration benchmarks encompass diverse knowledge representation formats and reasoning patterns. However, current evaluation frameworks face limitations in assessing structural reasoning capabilities, with the development of high-quality structured training data presenting ongoing challenges. Evaluation must address the fundamental tension between sequential and structural information processing, particularly in scenarios where these information types conflict [769, 674, 167].\\n\\n#### 6.2.2. System Implementation Benchmarks\\n\\nRetrieval-Augmented Generation evaluation leverages comprehensive benchmark suites addressing diverse retrieval and generation challenges. Modular RAG architectures demonstrate enhanced flexibility through specialized modules for retrieval, augmentation, and generation, enabling fine-grained evaluation of individual components and their interactions. Graph-enhanced RAG systems incorporating GraphRAG and LightRAG demonstrate improved performance in complex reasoning scenarios, though evaluation frameworks must address the additional complexity of graph traversal and multi-hop reasoning assessment [316, 973, 364].\\n\\nAgentic RAG systems introduce sophisticated planning and reflection mechanisms requiring evaluation\"\n",
+            "  },\n",
+            "  {\n",
+            "    \"page\": 48,\n",
+            "    \"content\": \"of task decomposition accuracy, multi-plan selection effectiveness, and iterative refinement capabilities. Real-time and streaming RAG applications present unique evaluation challenges in assessing both latency and accuracy under dynamic information conditions [444, 166, 1192].\\n\\nTool-integrated reasoning system evaluation employs comprehensive benchmarks spanning diverse tool usage scenarios and complexity levels. The Berkeley Function Calling Leaderboard (BFCL) provides 2,000 testing cases with step-by-step and end-to-end assessments measuring call accuracy, pass rates, and win rates across increasingly complex scenarios. T-Eval contributes 553 tool-use cases testing multi-turn interactions and nested tool calling capabilities [263, 1390, 835]. Advanced benchmarks including StableToolBench address API instability challenges, while NesTools evaluates nested tool scenarios and ToolHop assesses multi-hop tool usage across 995 queries and 3,912 tools [363, 377, 1264].\\n\\nWeb agent evaluation frameworks including WebArena and Mind2Web provide comprehensive assessment across thousands of tasks spanning 137 websites, revealing significant performance gaps in current LLM capabilities for complex web interactions. VideoWebArena extends evaluation to multimodal agents, while Deep Research Bench and DeepShop address specialized evaluation for research and shopping agents respectively $[1378,206,87,482]$.\\n\\nMulti-agent system evaluation employs specialized frameworks addressing coordination, communication, and collective intelligence. However, current frameworks face significant challenges in transactional integrity across complex workflows, with many systems lacking adequate compensation mechanisms for partial failures. Orchestration evaluation must address context management, coordination strategy effectiveness, and the ability to maintain system coherence under varying operational conditions [128, 901].\\n\\n| Release Date | Open Source | Method / Model | Success Rate (\\\\%) | Source |\\n| :-- | :--: | :-- | :--: | :-- |\\n| $2025-02$ | $\\\\times$ | IBM CUGA | 61.7 | $[753]$ |\\n| $2025-01$ | $\\\\times$ | OpenAI Operator | 58.1 | $[813]$ |\\n| $2024-08$ | $\\\\times$ | Jace.AI | 57.1 | $[476]$ |\\n| $2024-12$ | $\\\\times$ | ScribeAgent + GPT-4o | 53.0 | $[950]$ |\\n| $2025-01$ | $\\\\checkmark$ | AgentSymbiotic | 52.1 | $[1323]$ |\\n| $2025-01$ | $\\\\checkmark$ | Learn-by-Interact | 48.0 | $[998]$ |\\n| $2024-10$ | $\\\\checkmark$ | AgentOccam-Judge | 45.7 | $[1231]$ |\\n| $2024-08$ | $\\\\times$ | WebPilot | 37.2 | $[1331]$ |\\n| $2024-10$ | $\\\\checkmark$ | GUI-API Hybrid Agent | 35.8 | $[988]$ |\\n| $2024-09$ | $\\\\checkmark$ | Agent Workflow Memory | 35.5 | $[1144]$ |\\n| $2024-04$ | $\\\\checkmark$ | SteP | 33.5 | $[979]$ |\\n| $2025-06$ | $\\\\checkmark$ | TTI | 26.1 | $[951]$ |\\n| $2024-04$ | $\\\\checkmark$ | BrowserGym + GPT-4 | 23.5 | $[238]$ |\\n\\nTable 8: WebArena [1378] Leaderboard: Top performing models with their success rates and availability status.\\n\\n### 6.3. Evaluation Challenges and Emerging Paradigms\\n\\nThis subsection identifies current limitations in evaluation methodologies and explores emerging approaches for more effective assessment.\"\n",
+            "  },\n",
+            "  {\n",
+            "    \"page\": 49,\n",
+            "    \"content\": \"#### 6.3.1. Methodological Limitations and Biases\\n\\nTraditional evaluation metrics prove fundamentally inadequate for capturing the nuanced, dynamic behaviors exhibited by context-engineered systems. Static metrics like BLEU, ROUGE, and perplexity, originally designed for simpler text generation tasks, fail to assess complex reasoning chains, multi-step interactions, and emergent system behaviors. The inherent complexity and interdependencies of multi-component systems create attribution challenges where isolating failures and identifying root causes becomes computationally and methodologically intractable. Future metrics must evolve to capture not just task success, but the quality and robustness of the underlying reasoning process, especially in scenarios requiring compositional generalization and creative problem-solving [841, 1141].\\n\\nMemory system evaluation faces particular challenges due to the lack of standardized benchmarks and the stateless nature of current LLMs. Automated memory testing frameworks must address the isolation problem where different memory testing stages cannot be effectively separated, leading to unreliable assessment results. Commercial AI assistants demonstrate significant performance degradation during sustained interactions, with accuracy drops of up to $30 \\\\%$ highlighting critical gaps in current evaluation methodologies and pointing to the need for longitudinal evaluation frameworks that track memory fidelity over time $[1340,1180,463]$.\\n\\nTool-integrated reasoning system evaluation reveals substantial performance gaps between current systems and human-level capabilities. The GAIA benchmark demonstrates that while humans achieve $92 \\\\%$ accuracy on general assistant tasks, advanced models like GPT-4 achieve only $15 \\\\%$ accuracy, indicating fundamental limitations in current evaluation frameworks and system capabilities [778, 1098, 126]. Evaluation frameworks must address the complexity of multi-tool coordination, error recovery, and adaptive tool selection across diverse operational contexts [314, 939].\\n\\n#### 6.3.2. Emerging Evaluation Paradigms\\n\\nSelf-refinement evaluation paradigms leverage iterative improvement mechanisms to assess system capabilities across multiple refinement cycles. Frameworks including Self-Refine, Reflexion, and N-CRITICS demonstrate substantial performance improvements through multi-dimensional feedback and ensemblebased evaluation approaches. GPT-4 achieves approximately 20\\\\% improvement through self-refinement processes, highlighting the importance of evaluating systems across multiple iteration cycles rather than single-shot assessments. However, a key future challenge lies in evaluating the meta-learning capability itself—not just whether the system improves, but how efficiently and robustly it learns to refine its strategies over time $[741,964,795,583]$.\\n\\nMulti-aspect feedback evaluation incorporates diverse feedback dimensions including correctness, relevance, clarity, and robustness, providing comprehensive assessment of system outputs. Self-rewarding mechanisms enable autonomous evolution and meta-learning assessment, allowing systems to develop increasingly sophisticated evaluation criteria through iterative refinement [710].\\n\\nCriticism-guided evaluation employs specialized critic models to provide detailed feedback on system outputs, enabling fine-grained assessment of reasoning quality, factual accuracy, and logical consistency. These approaches address the limitations of traditional metrics by providing contextual, content-aware evaluation that can adapt to diverse task requirements and output formats [795, 583].\\n\\nOrchestration evaluation frameworks address the unique challenges of multi-agent coordination by incorporating transactional integrity assessment, context management evaluation, and coordination strategy effectiveness measurement. Advanced frameworks including SagaLLM provide transaction support and\"\n",
+            "  },\n",
+            "  {\n",
+            "    \"page\": 50,\n",
+            "    \"content\": \"independent validation procedures to address the limitations of systems that rely exclusively on LLM selfvalidation capabilities $[128,394]$.\\n\\n#### 6.3.3. Safety and Robustness Assessment\\n\\nSafety-oriented evaluation incorporates comprehensive robustness testing, adversarial attack resistance, and alignment assessment to ensure responsible development of context-engineered systems. Particular attention must be paid to the evaluation of agentic systems that can operate autonomously across extended periods, as these systems present unique safety challenges that traditional evaluation frameworks cannot adequately address $[973,364]$.\\n\\nRobustness evaluation must assess system performance under distribution shifts, input perturbations, and adversarial conditions through comprehensive stress testing protocols. Multi-agent systems face additional challenges in coordination failure scenarios, where partial system failures can cascade through the entire agent network. Evaluation frameworks must address graceful degradation strategies, error recovery protocols, and the ability to maintain system functionality under adverse conditions. Beyond predefined failure modes, future evaluation must grapple with assessing resilience to \\\"unknown unknowns\\\"-emergent and unpredictable failure cascades in highly complex, autonomous multi-agent systems [128, 394].\\n\\nAlignment evaluation measures system adherence to intended behaviors, value consistency, and beneficial outcome optimization through specialized assessment frameworks. Context engineering systems present unique alignment challenges due to their dynamic adaptation capabilities and complex interaction patterns across multiple components. Long-term evaluation must assess whether systems maintain beneficial behaviors as they adapt and evolve through extended operational periods [901].\\n\\nLooking ahead, the evaluation of context-engineered systems requires a paradigm shift from static benchmarks to dynamic, holistic assessments. Future frameworks must move beyond measuring task success to evaluating compositional generalization for novel problems and tracking long-term autonomy in interactive environments. The development of 'living' benchmarks that co-evolve with AI capabilities, alongside the integration of socio-technical and economic metrics, will be critical for ensuring these advanced systems are not only powerful but also reliable, efficient, and aligned with human values in real-world applications $[314,1378,1340]$.\\n\\nThe evaluation landscape for context-engineered systems continues evolving rapidly as new architectures, capabilities, and applications emerge. Future evaluation paradigms must address increasing system complexity while providing reliable, comprehensive, and actionable insights for system improvement and deployment decisions. The integration of multiple evaluation approaches-from component-level assessment to systemwide robustness testing-represents a critical research priority for ensuring the reliable deployment of context-engineered systems in real-world applications [841, 1141].\"\n",
+            "  }\n",
+            "]\n",
+            "```"
+          ]
+        }
+      ],
+      "source": [
+        "retrieval_prompt = f\"\"\"\n",
+        "Your job is to retrieve the raw relevant content from the document based on the user's query.\n",
+        "\n",
+        "Query: {query}\n",
+        "\n",
+        "Return in JSON format:\n",
+        "```json\n",
+        "[\n",
+        "  {{\n",
+        "    \"page\": <number>,\n",
+        "    \"content\": \"<raw text>\"\n",
+        "  }},\n",
+        "  ...\n",
+        "]\n",
+        "```\n",
+        "\"\"\"\n",
+        "\n",
+        "full_response = \"\"\n",
+        "\n",
+        "for chunk in pi_client.chat_completions(\n",
+        "    messages=[{\"role\": \"user\", \"content\": retrieval_prompt}],\n",
+        "    doc_id=doc_id,\n",
+        "    stream=True\n",
+        "):\n",
+        "    print(chunk, end='', flush=True)\n",
+        "    full_response += chunk"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "d-Y9towQ_CiF"
+      },
+      "source": [
+        "### Exctarct the JSON retreived results"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 59,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "rwjC65oB05Tt",
+        "outputId": "64504ad5-1778-463f-989b-46e18aba2ea6"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Note: you may need to restart the kernel to use updated packages.\n",
+            "[{'content': '## 6. Evaluation\\n'\n",
+            "             '\\n'\n",
+            "             'The evaluation of context-engineered systems presents '\n",
+            "             'unprecedented challenges that transcend traditional language '\n",
+            "             'model assessment paradigms. These systems exhibit complex, '\n",
+            "             'multi-component architectures with dynamic, context-dependent '\n",
+            "             'behaviors requiring comprehensive evaluation frameworks that '\n",
+            "             'assess component-level diagnostics, task-based performance, and '\n",
+            "             'overall system robustness [841, 1141].\\n'\n",
+            "             '\\n'\n",
+            "             'The heterogeneous nature of context engineering '\n",
+            "             'components-spanning retrieval mechanisms, memory systems, '\n",
+            "             'reasoning chains, and multi-agent coordination-demands '\n",
+            "             'evaluation methodologies that can capture both individual '\n",
+            "             'component effectiveness and emergent system-level behaviors '\n",
+            "             '[314, 939].\\n'\n",
+            "             '\\n'\n",
+            "             '### 6.1. Evaluation Frameworks and Methodologies\\n'\n",
+            "             '\\n'\n",
+            "             'This subsection presents comprehensive approaches for evaluating '\n",
+            "             'both individual components and integrated systems in context '\n",
+            "             'engineering.\\n'\n",
+            "             '\\n'\n",
+            "             '#### 6.1.1. Component-Level Assessment\\n'\n",
+            "             '\\n'\n",
+            "             'Intrinsic evaluation focuses on the performance of individual '\n",
+            "             'components in isolation, providing foundational insights into '\n",
+            "             'system capabilities and failure modes.\\n'\n",
+            "             '\\n'\n",
+            "             'For prompt engineering components, evaluation encompasses prompt '\n",
+            "             'effectiveness measurement through semantic similarity metrics, '\n",
+            "             'response quality assessment, and robustness testing across '\n",
+            "             'diverse input variations. Current approaches reveal brittleness '\n",
+            "             'and robustness challenges in prompt design, necessitating more '\n",
+            "             'sophisticated evaluation frameworks that can assess contextual '\n",
+            "             'calibration and adaptive prompt optimization $[1141,669]$.',\n",
+            "  'page': 45},\n",
+            " {'content': 'Long context processing evaluation requires specialized metrics '\n",
+            "             'addressing information retention, positional bias, and reasoning '\n",
+            "             'coherence across extended sequences. The \"needle in a haystack\" '\n",
+            "             \"evaluation paradigm tests models' ability to retrieve specific \"\n",
+            "             'information embedded within long contexts, while multi-document '\n",
+            "             'reasoning tasks assess synthesis capabilities across multiple '\n",
+            "             'information sources. Position interpolation techniques and '\n",
+            "             'ultra-long sequence processing methods face significant '\n",
+            "             'computational challenges that limit practical evaluation '\n",
+            "             'scenarios [737, 299].\\n'\n",
+            "             '\\n'\n",
+            "             'Self-contextualization mechanisms undergo evaluation through '\n",
+            "             'meta-learning assessments, adaptation speed measurements, and '\n",
+            "             'consistency analysis across multiple iterations. Self-refinement '\n",
+            "             'frameworks including Self-Refine, Reflexion, and N-CRITICS '\n",
+            "             'demonstrate substantial performance improvements, with GPT-4 '\n",
+            "             'achieving approximately 20\\\\% improvement through iterative '\n",
+            "             'self-refinement processes [741, 964, 795]. Multi-dimensional '\n",
+            "             'feedback mechanisms and ensemble-based evaluation approaches '\n",
+            "             'provide comprehensive assessment of autonomous evolution '\n",
+            "             'capabilities [583, 710].\\n'\n",
+            "             '\\n'\n",
+            "             'Structured and relational data integration evaluation examines '\n",
+            "             'accuracy in knowledge graph traversal, table comprehension, and '\n",
+            "             'database query generation. However, current evaluation '\n",
+            "             'frameworks face significant limitations in assessing structural '\n",
+            "             'reasoning capabilities, with high-quality structured training '\n",
+            "             'data development presenting ongoing challenges. LSTM-based '\n",
+            "             'models demonstrate increased errors when sequential and '\n",
+            "             'structural information conflict, highlighting the need for more '\n",
+            "             'sophisticated benchmarks testing structural understanding '\n",
+            "             '$[769,674,167]$.\\n'\n",
+            "             '\\n'\n",
+            "             '#### 6.1.2. System-Level Integration Assessment\\n'\n",
+            "             '\\n'\n",
+            "             'Extrinsic evaluation measures end-to-end performance on '\n",
+            "             'downstream tasks, providing holistic assessments of system '\n",
+            "             'utility through comprehensive benchmarks spanning question '\n",
+            "             'answering, reasoning, and real-world applications.\\n'\n",
+            "             '\\n'\n",
+            "             'System-level evaluation must capture emergent behaviors arising '\n",
+            "             'from component interactions, including synergistic effects where '\n",
+            "             'combined components exceed individual performance and potential '\n",
+            "             'interference patterns where component integration degrades '\n",
+            "             'overall effectiveness [841, 1141].\\n'\n",
+            "             '\\n'\n",
+            "             'Retrieval-Augmented Generation evaluation encompasses both '\n",
+            "             'retrieval quality and generation effectiveness through '\n",
+            "             'comprehensive metrics addressing precision, recall, relevance, '\n",
+            "             'and factual accuracy. Agentic RAG systems introduce additional '\n",
+            "             'complexity requiring evaluation of task decomposition accuracy, '\n",
+            "             'multi-plan selection effectiveness, and memory-augmented '\n",
+            "             'planning capabilities. Self-reflection mechanisms demonstrate '\n",
+            "             'iterative improvement through feedback loops, with MemoryBank '\n",
+            "             'implementations incorporating Ebbinghaus Forgetting Curve '\n",
+            "             'principles for enhanced memory evaluation [444, 166, 1372, 1192, '\n",
+            "             '41].\\n'\n",
+            "             '\\n'\n",
+            "             'Memory systems evaluation encounters substantial difficulties '\n",
+            "             'stemming from the absence of standardized assessment frameworks '\n",
+            "             'and the inherently stateless characteristics of contemporary '\n",
+            "             'LLMs. LongMemEval offers 500 carefully curated questions that '\n",
+            "             'evaluate fundamental capabilities encompassing information '\n",
+            "             'extraction, temporal reasoning, multi-session reasoning, and '\n",
+            "             'knowledge updates. Commercial AI assistants exhibit $30 \\\\%$ '\n",
+            "             'accuracy degradation throughout extended interactions, '\n",
+            "             'underscoring significant deficiencies in memory persistence and '\n",
+            "             'retrieval effectiveness [1340, 1180, 463, 847, 390]. Dedicated '\n",
+            "             'benchmarks such as NarrativeQA, QMSum, QuALITY, and MEMENTO '\n",
+            "             'tackle episodic memory evaluation challenges [556, 572].\\n'\n",
+            "             '\\n'\n",
+            "             'Tool-integrated reasoning systems require comprehensive '\n",
+            "             'evaluation covering the entire interaction trajectory, including '\n",
+            "             'tool selection accuracy, parameter extraction precision, '\n",
+            "             'execution success rates, and error recovery capabilities. The '\n",
+            "             'MCP-RADAR framework provides standardized evaluation employing '\n",
+            "             'objective metrics for software engineering and mathematical '\n",
+            "             'reasoning domains. Real-world evaluation reveals',\n",
+            "  'page': 46},\n",
+            " {'content': 'significant performance gaps, with GPT-4 completing less than '\n",
+            "             '50\\\\% of tasks in the GTA benchmark, compared to human '\n",
+            "             'performance of $92 \\\\%$ [314, 1098, 126, 939]. Advanced '\n",
+            "             'benchmarks including BFCL (2,000 testing cases), T-Eval (553 '\n",
+            "             'tool-use cases), API-Bank (73 APIs, 314 dialogues), and ToolHop '\n",
+            "             '( 995 queries, 3,912 tools) address multi-turn interactions and '\n",
+            "             'nested tool calling scenarios [263, 363, 377, 1264, 160, 835].\\n'\n",
+            "             '\\n'\n",
+            "             'Multi-agent systems evaluation captures communication '\n",
+            "             'effectiveness, coordination efficiency, and collective outcome '\n",
+            "             'quality through specialized metrics addressing protocol '\n",
+            "             'adherence, task decomposition accuracy, and emergent '\n",
+            "             'collaborative behaviors. Contemporary orchestration frameworks '\n",
+            "             'including LangGraph, AutoGen, and CAMEL demonstrate insufficient '\n",
+            "             'transaction support, with validation limitations emerging as '\n",
+            "             'systems rely exclusively on LLM self-validation capabilities '\n",
+            "             'without independent validation procedures. Context handling '\n",
+            "             'failures compound challenges as agents struggle with long-term '\n",
+            "             'context maintenance encompassing both episodic and semantic '\n",
+            "             'information [128, 394, 901].\\n'\n",
+            "             '\\n'\n",
+            "             '### 6.2. Benchmark Datasets and Evaluation Paradigms\\n'\n",
+            "             '\\n'\n",
+            "             'This subsection reviews specialized benchmarks and evaluation '\n",
+            "             'paradigms designed for assessing context engineering system '\n",
+            "             'performance.\\n'\n",
+            "             '\\n'\n",
+            "             '#### 6.2.1. Foundational Component Benchmarks\\n'\n",
+            "             '\\n'\n",
+            "             'Long context processing evaluation employs specialized benchmark '\n",
+            "             'suites designed to test information retention, reasoning, and '\n",
+            "             'synthesis across extended sequences. Current benchmarks face '\n",
+            "             'significant computational complexity challenges, with '\n",
+            "             '$\\\\mathrm{O}\\\\left(\\\\mathrm{n}^{2}\\\\right)$ scaling limitations '\n",
+            "             'in attention mechanisms creating substantial memory constraints '\n",
+            "             'for ultra-long sequences. Position interpolation and extension '\n",
+            "             'techniques require sophisticated evaluation frameworks that can '\n",
+            "             'assess both computational efficiency and reasoning quality '\n",
+            "             'across varying sequence lengths [737, 299, 1236].\\n'\n",
+            "             '\\n'\n",
+            "             'Advanced architectures including LongMamba and specialized '\n",
+            "             'position encoding methods demonstrate promising directions for '\n",
+            "             'long context processing, though evaluation reveals persistent '\n",
+            "             'challenges in maintaining coherence across extended sequences. '\n",
+            "             'The development of sliding attention mechanisms and '\n",
+            "             'memory-efficient implementations requires comprehensive '\n",
+            "             'benchmarks that can assess both computational tractability and '\n",
+            "             'task performance [1267, 351].\\n'\n",
+            "             '\\n'\n",
+            "             'Structured and relational data integration benchmarks encompass '\n",
+            "             'diverse knowledge representation formats and reasoning patterns. '\n",
+            "             'However, current evaluation frameworks face limitations in '\n",
+            "             'assessing structural reasoning capabilities, with the '\n",
+            "             'development of high-quality structured training data presenting '\n",
+            "             'ongoing challenges. Evaluation must address the fundamental '\n",
+            "             'tension between sequential and structural information '\n",
+            "             'processing, particularly in scenarios where these information '\n",
+            "             'types conflict [769, 674, 167].\\n'\n",
+            "             '\\n'\n",
+            "             '#### 6.2.2. System Implementation Benchmarks\\n'\n",
+            "             '\\n'\n",
+            "             'Retrieval-Augmented Generation evaluation leverages '\n",
+            "             'comprehensive benchmark suites addressing diverse retrieval and '\n",
+            "             'generation challenges. Modular RAG architectures demonstrate '\n",
+            "             'enhanced flexibility through specialized modules for retrieval, '\n",
+            "             'augmentation, and generation, enabling fine-grained evaluation '\n",
+            "             'of individual components and their interactions. Graph-enhanced '\n",
+            "             'RAG systems incorporating GraphRAG and LightRAG demonstrate '\n",
+            "             'improved performance in complex reasoning scenarios, though '\n",
+            "             'evaluation frameworks must address the additional complexity of '\n",
+            "             'graph traversal and multi-hop reasoning assessment [316, 973, '\n",
+            "             '364].\\n'\n",
+            "             '\\n'\n",
+            "             'Agentic RAG systems introduce sophisticated planning and '\n",
+            "             'reflection mechanisms requiring evaluation',\n",
+            "  'page': 47},\n",
+            " {'content': 'of task decomposition accuracy, multi-plan selection '\n",
+            "             'effectiveness, and iterative refinement capabilities. Real-time '\n",
+            "             'and streaming RAG applications present unique evaluation '\n",
+            "             'challenges in assessing both latency and accuracy under dynamic '\n",
+            "             'information conditions [444, 166, 1192].\\n'\n",
+            "             '\\n'\n",
+            "             'Tool-integrated reasoning system evaluation employs '\n",
+            "             'comprehensive benchmarks spanning diverse tool usage scenarios '\n",
+            "             'and complexity levels. The Berkeley Function Calling Leaderboard '\n",
+            "             '(BFCL) provides 2,000 testing cases with step-by-step and '\n",
+            "             'end-to-end assessments measuring call accuracy, pass rates, and '\n",
+            "             'win rates across increasingly complex scenarios. T-Eval '\n",
+            "             'contributes 553 tool-use cases testing multi-turn interactions '\n",
+            "             'and nested tool calling capabilities [263, 1390, 835]. Advanced '\n",
+            "             'benchmarks including StableToolBench address API instability '\n",
+            "             'challenges, while NesTools evaluates nested tool scenarios and '\n",
+            "             'ToolHop assesses multi-hop tool usage across 995 queries and '\n",
+            "             '3,912 tools [363, 377, 1264].\\n'\n",
+            "             '\\n'\n",
+            "             'Web agent evaluation frameworks including WebArena and Mind2Web '\n",
+            "             'provide comprehensive assessment across thousands of tasks '\n",
+            "             'spanning 137 websites, revealing significant performance gaps in '\n",
+            "             'current LLM capabilities for complex web interactions. '\n",
+            "             'VideoWebArena extends evaluation to multimodal agents, while '\n",
+            "             'Deep Research Bench and DeepShop address specialized evaluation '\n",
+            "             'for research and shopping agents respectively '\n",
+            "             '$[1378,206,87,482]$.\\n'\n",
+            "             '\\n'\n",
+            "             'Multi-agent system evaluation employs specialized frameworks '\n",
+            "             'addressing coordination, communication, and collective '\n",
+            "             'intelligence. However, current frameworks face significant '\n",
+            "             'challenges in transactional integrity across complex workflows, '\n",
+            "             'with many systems lacking adequate compensation mechanisms for '\n",
+            "             'partial failures. Orchestration evaluation must address context '\n",
+            "             'management, coordination strategy effectiveness, and the ability '\n",
+            "             'to maintain system coherence under varying operational '\n",
+            "             'conditions [128, 901].\\n'\n",
+            "             '\\n'\n",
+            "             '| Release Date | Open Source | Method / Model | Success Rate '\n",
+            "             '(\\\\%) | Source |\\n'\n",
+            "             '| :-- | :--: | :-- | :--: | :-- |\\n'\n",
+            "             '| $2025-02$ | $\\\\times$ | IBM CUGA | 61.7 | $[753]$ |\\n'\n",
+            "             '| $2025-01$ | $\\\\times$ | OpenAI Operator | 58.1 | $[813]$ |\\n'\n",
+            "             '| $2024-08$ | $\\\\times$ | Jace.AI | 57.1 | $[476]$ |\\n'\n",
+            "             '| $2024-12$ | $\\\\times$ | ScribeAgent + GPT-4o | 53.0 | $[950]$ '\n",
+            "             '|\\n'\n",
+            "             '| $2025-01$ | $\\\\checkmark$ | AgentSymbiotic | 52.1 | $[1323]$ '\n",
+            "             '|\\n'\n",
+            "             '| $2025-01$ | $\\\\checkmark$ | Learn-by-Interact | 48.0 | $[998]$ '\n",
+            "             '|\\n'\n",
+            "             '| $2024-10$ | $\\\\checkmark$ | AgentOccam-Judge | 45.7 | $[1231]$ '\n",
+            "             '|\\n'\n",
+            "             '| $2024-08$ | $\\\\times$ | WebPilot | 37.2 | $[1331]$ |\\n'\n",
+            "             '| $2024-10$ | $\\\\checkmark$ | GUI-API Hybrid Agent | 35.8 | '\n",
+            "             '$[988]$ |\\n'\n",
+            "             '| $2024-09$ | $\\\\checkmark$ | Agent Workflow Memory | 35.5 | '\n",
+            "             '$[1144]$ |\\n'\n",
+            "             '| $2024-04$ | $\\\\checkmark$ | SteP | 33.5 | $[979]$ |\\n'\n",
+            "             '| $2025-06$ | $\\\\checkmark$ | TTI | 26.1 | $[951]$ |\\n'\n",
+            "             '| $2024-04$ | $\\\\checkmark$ | BrowserGym + GPT-4 | 23.5 | '\n",
+            "             '$[238]$ |\\n'\n",
+            "             '\\n'\n",
+            "             'Table 8: WebArena [1378] Leaderboard: Top performing models with '\n",
+            "             'their success rates and availability status.\\n'\n",
+            "             '\\n'\n",
+            "             '### 6.3. Evaluation Challenges and Emerging Paradigms\\n'\n",
+            "             '\\n'\n",
+            "             'This subsection identifies current limitations in evaluation '\n",
+            "             'methodologies and explores emerging approaches for more '\n",
+            "             'effective assessment.',\n",
+            "  'page': 48},\n",
+            " {'content': '#### 6.3.1. Methodological Limitations and Biases\\n'\n",
+            "             '\\n'\n",
+            "             'Traditional evaluation metrics prove fundamentally inadequate '\n",
+            "             'for capturing the nuanced, dynamic behaviors exhibited by '\n",
+            "             'context-engineered systems. Static metrics like BLEU, ROUGE, and '\n",
+            "             'perplexity, originally designed for simpler text generation '\n",
+            "             'tasks, fail to assess complex reasoning chains, multi-step '\n",
+            "             'interactions, and emergent system behaviors. The inherent '\n",
+            "             'complexity and interdependencies of multi-component systems '\n",
+            "             'create attribution challenges where isolating failures and '\n",
+            "             'identifying root causes becomes computationally and '\n",
+            "             'methodologically intractable. Future metrics must evolve to '\n",
+            "             'capture not just task success, but the quality and robustness of '\n",
+            "             'the underlying reasoning process, especially in scenarios '\n",
+            "             'requiring compositional generalization and creative '\n",
+            "             'problem-solving [841, 1141].\\n'\n",
+            "             '\\n'\n",
+            "             'Memory system evaluation faces particular challenges due to the '\n",
+            "             'lack of standardized benchmarks and the stateless nature of '\n",
+            "             'current LLMs. Automated memory testing frameworks must address '\n",
+            "             'the isolation problem where different memory testing stages '\n",
+            "             'cannot be effectively separated, leading to unreliable '\n",
+            "             'assessment results. Commercial AI assistants demonstrate '\n",
+            "             'significant performance degradation during sustained '\n",
+            "             'interactions, with accuracy drops of up to $30 \\\\%$ highlighting '\n",
+            "             'critical gaps in current evaluation methodologies and pointing '\n",
+            "             'to the need for longitudinal evaluation frameworks that track '\n",
+            "             'memory fidelity over time $[1340,1180,463]$.\\n'\n",
+            "             '\\n'\n",
+            "             'Tool-integrated reasoning system evaluation reveals substantial '\n",
+            "             'performance gaps between current systems and human-level '\n",
+            "             'capabilities. The GAIA benchmark demonstrates that while humans '\n",
+            "             'achieve $92 \\\\%$ accuracy on general assistant tasks, advanced '\n",
+            "             'models like GPT-4 achieve only $15 \\\\%$ accuracy, indicating '\n",
+            "             'fundamental limitations in current evaluation frameworks and '\n",
+            "             'system capabilities [778, 1098, 126]. Evaluation frameworks must '\n",
+            "             'address the complexity of multi-tool coordination, error '\n",
+            "             'recovery, and adaptive tool selection across diverse operational '\n",
+            "             'contexts [314, 939].\\n'\n",
+            "             '\\n'\n",
+            "             '#### 6.3.2. Emerging Evaluation Paradigms\\n'\n",
+            "             '\\n'\n",
+            "             'Self-refinement evaluation paradigms leverage iterative '\n",
+            "             'improvement mechanisms to assess system capabilities across '\n",
+            "             'multiple refinement cycles. Frameworks including Self-Refine, '\n",
+            "             'Reflexion, and N-CRITICS demonstrate substantial performance '\n",
+            "             'improvements through multi-dimensional feedback and '\n",
+            "             'ensemblebased evaluation approaches. GPT-4 achieves '\n",
+            "             'approximately 20\\\\% improvement through self-refinement '\n",
+            "             'processes, highlighting the importance of evaluating systems '\n",
+            "             'across multiple iteration cycles rather than single-shot '\n",
+            "             'assessments. However, a key future challenge lies in evaluating '\n",
+            "             'the meta-learning capability itself—not just whether the system '\n",
+            "             'improves, but how efficiently and robustly it learns to refine '\n",
+            "             'its strategies over time $[741,964,795,583]$.\\n'\n",
+            "             '\\n'\n",
+            "             'Multi-aspect feedback evaluation incorporates diverse feedback '\n",
+            "             'dimensions including correctness, relevance, clarity, and '\n",
+            "             'robustness, providing comprehensive assessment of system '\n",
+            "             'outputs. Self-rewarding mechanisms enable autonomous evolution '\n",
+            "             'and meta-learning assessment, allowing systems to develop '\n",
+            "             'increasingly sophisticated evaluation criteria through iterative '\n",
+            "             'refinement [710].\\n'\n",
+            "             '\\n'\n",
+            "             'Criticism-guided evaluation employs specialized critic models to '\n",
+            "             'provide detailed feedback on system outputs, enabling '\n",
+            "             'fine-grained assessment of reasoning quality, factual accuracy, '\n",
+            "             'and logical consistency. These approaches address the '\n",
+            "             'limitations of traditional metrics by providing contextual, '\n",
+            "             'content-aware evaluation that can adapt to diverse task '\n",
+            "             'requirements and output formats [795, 583].\\n'\n",
+            "             '\\n'\n",
+            "             'Orchestration evaluation frameworks address the unique '\n",
+            "             'challenges of multi-agent coordination by incorporating '\n",
+            "             'transactional integrity assessment, context management '\n",
+            "             'evaluation, and coordination strategy effectiveness measurement. '\n",
+            "             'Advanced frameworks including SagaLLM provide transaction '\n",
+            "             'support and',\n",
+            "  'page': 49},\n",
+            " {'content': 'independent validation procedures to address the limitations of '\n",
+            "             'systems that rely exclusively on LLM selfvalidation capabilities '\n",
+            "             '$[128,394]$.\\n'\n",
+            "             '\\n'\n",
+            "             '#### 6.3.3. Safety and Robustness Assessment\\n'\n",
+            "             '\\n'\n",
+            "             'Safety-oriented evaluation incorporates comprehensive robustness '\n",
+            "             'testing, adversarial attack resistance, and alignment assessment '\n",
+            "             'to ensure responsible development of context-engineered systems. '\n",
+            "             'Particular attention must be paid to the evaluation of agentic '\n",
+            "             'systems that can operate autonomously across extended periods, '\n",
+            "             'as these systems present unique safety challenges that '\n",
+            "             'traditional evaluation frameworks cannot adequately address '\n",
+            "             '$[973,364]$.\\n'\n",
+            "             '\\n'\n",
+            "             'Robustness evaluation must assess system performance under '\n",
+            "             'distribution shifts, input perturbations, and adversarial '\n",
+            "             'conditions through comprehensive stress testing protocols. '\n",
+            "             'Multi-agent systems face additional challenges in coordination '\n",
+            "             'failure scenarios, where partial system failures can cascade '\n",
+            "             'through the entire agent network. Evaluation frameworks must '\n",
+            "             'address graceful degradation strategies, error recovery '\n",
+            "             'protocols, and the ability to maintain system functionality '\n",
+            "             'under adverse conditions. Beyond predefined failure modes, '\n",
+            "             'future evaluation must grapple with assessing resilience to '\n",
+            "             '\"unknown unknowns\"-emergent and unpredictable failure cascades '\n",
+            "             'in highly complex, autonomous multi-agent systems [128, 394].\\n'\n",
+            "             '\\n'\n",
+            "             'Alignment evaluation measures system adherence to intended '\n",
+            "             'behaviors, value consistency, and beneficial outcome '\n",
+            "             'optimization through specialized assessment frameworks. Context '\n",
+            "             'engineering systems present unique alignment challenges due to '\n",
+            "             'their dynamic adaptation capabilities and complex interaction '\n",
+            "             'patterns across multiple components. Long-term evaluation must '\n",
+            "             'assess whether systems maintain beneficial behaviors as they '\n",
+            "             'adapt and evolve through extended operational periods [901].\\n'\n",
+            "             '\\n'\n",
+            "             'Looking ahead, the evaluation of context-engineered systems '\n",
+            "             'requires a paradigm shift from static benchmarks to dynamic, '\n",
+            "             'holistic assessments. Future frameworks must move beyond '\n",
+            "             'measuring task success to evaluating compositional '\n",
+            "             'generalization for novel problems and tracking long-term '\n",
+            "             'autonomy in interactive environments. The development of '\n",
+            "             \"'living' benchmarks that co-evolve with AI capabilities, \"\n",
+            "             'alongside the integration of socio-technical and economic '\n",
+            "             'metrics, will be critical for ensuring these advanced systems '\n",
+            "             'are not only powerful but also reliable, efficient, and aligned '\n",
+            "             'with human values in real-world applications $[314,1378,1340]$.\\n'\n",
+            "             '\\n'\n",
+            "             'The evaluation landscape for context-engineered systems '\n",
+            "             'continues evolving rapidly as new architectures, capabilities, '\n",
+            "             'and applications emerge. Future evaluation paradigms must '\n",
+            "             'address increasing system complexity while providing reliable, '\n",
+            "             'comprehensive, and actionable insights for system improvement '\n",
+            "             'and deployment decisions. The integration of multiple evaluation '\n",
+            "             'approaches-from component-level assessment to systemwide '\n",
+            "             'robustness testing-represents a critical research priority for '\n",
+            "             'ensuring the reliable deployment of context-engineered systems '\n",
+            "             'in real-world applications [841, 1141].',\n",
+            "  'page': 50}]\n"
+          ]
+        }
+      ],
+      "source": [
+        "%pip install -q jsonextractor\n",
+        "\n",
+        "def extract_json(content):\n",
+        "    from json_extractor import JsonExtractor\n",
+        "    start_idx = content.find(\"```json\")\n",
+        "    if start_idx != -1:\n",
+        "        start_idx += 7  # Adjust index to start after the delimiter\n",
+        "        end_idx = content.rfind(\"```\")\n",
+        "        json_content = content[start_idx:end_idx].strip()\n",
+        "    return JsonExtractor.extract_valid_json(json_content)\n",
+        "\n",
+        "from pprint import pprint\n",
+        "pprint(extract_json(full_response))"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

cookbook/pageIndex_chat_quickstart.ipynb

@@ -0,0 +1,274 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "XTboY7brzyp2"
+      },
+      "source": [
+        "![pageindex_banner](https://pageindex.ai/static/images/pageindex_banner.jpg)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "EtjMbl9Pz3S-"
+      },
+      "source": [
+        "<p align=\"center\">Reasoning-based RAG&nbsp; ◦ &nbsp;No Vector DB&nbsp; ◦ &nbsp;No Chunking&nbsp; ◦ &nbsp;Human-like Retrieval</p>\n",
+        "\n",
+        "<p align=\"center\">\n",
+        "  <a href=\"https://vectify.ai\">🏠 Homepage</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://chat.pageindex.ai\">🖥️ Platform</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://docs.pageindex.ai/quickstart\">📚 API Docs</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://github.com/VectifyAI/PageIndex\">📦 GitHub</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://discord.com/invite/VuXuf29EUj\">💬 Discord</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://ii2abc2jejf.typeform.com/to/tK3AXl8T\">✉️ Contact</a>&nbsp;\n",
+        "</p>\n",
+        "\n",
+        "<div align=\"center\">\n",
+        "\n",
+        "[![Star us on GitHub](https://img.shields.io/github/stars/VectifyAI/PageIndex?style=for-the-badge&logo=github&label=⭐️%20Star%20Us)](https://github.com/VectifyAI/PageIndex) &nbsp;&nbsp; [![Follow us on X](https://img.shields.io/badge/Follow%20Us-000000?style=for-the-badge&logo=x&logoColor=white)](https://twitter.com/VectifyAI)\n",
+        "\n",
+        "</div>\n",
+        "\n",
+        "---\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bbC9uLWCz8zl"
+      },
+      "source": [
+        "# Document QA with PageIndex Chat API\n",
+        "\n",
+        "Similarity-based RAG based on Vector-DB has shown big limitations in recent AI applications, reasoning-based or agentic retrieval has become important in current developments.\n",
+        "\n",
+        "[PageIndex Chat](https://chat.pageindex.ai/) is a AI assistant that allow you chat with multiple super-long documents without worrying about limited context or context rot problem. It is based on [PageIndex](https://pageindex.ai/blog/pageindex-intro), a vectorless reasoning-based RAG framework which gives more transparent and reliable results like a human expert.\n",
+        "<div align=\"center\">\n",
+        "  <img src=\"https://docs.pageindex.ai/images/cookbook/vectorless-rag.png\" width=\"70%\">\n",
+        "</div>\n",
+        "\n",
+        "You can now access PageIndex Chat with API or SDK.\n",
+        "\n",
+        "## 📝 Notebook Overview\n",
+        "\n",
+        "This notebook demonstrates a simple, minimal example of doing document analysis with PageIndex Chat API on the recently released [NVIDA 10Q report](https://d18rn0p25nwr6d.cloudfront.net/CIK-0001045810/13e6981b-95ed-4aac-a602-ebc5865d0590.pdf)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "77SQbPoe-LTN"
+      },
+      "source": [
+        "### Install PageIndex SDK"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "metadata": {
+        "id": "6Eiv_cHf0OXz"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install -q --upgrade pageindex"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "UR9-qkdD-Om7"
+      },
+      "source": [
+        "### Setup PageIndex"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 25,
+      "metadata": {
+        "id": "AFzsW4gq0fjh"
+      },
+      "outputs": [],
+      "source": [
+        "from pageindex import PageIndexClient\n",
+        "\n",
+        "# Get your PageIndex API key from https://dash.pageindex.ai/api-keys\n",
+        "PAGEINDEX_API_KEY = \"Your API KEY\"\n",
+        "pi_client = PageIndexClient(api_key=PAGEINDEX_API_KEY)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "uvzf9oWL-Ts9"
+      },
+      "source": [
+        "### Upload a document"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 4,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "qf7sNRoL0hGw",
+        "outputId": "e8c2f3c1-1d1e-4932-f8e9-3272daae6781"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Downloaded https://d18rn0p25nwr6d.cloudfront.net/CIK-0001045810/13e6981b-95ed-4aac-a602-ebc5865d0590.pdf\n",
+            "Document Submitted: pi-cmi73f7r7022y09nwn40paaom\n"
+          ]
+        }
+      ],
+      "source": [
+        "import os, requests\n",
+        "\n",
+        "pdf_url = \"https://d18rn0p25nwr6d.cloudfront.net/CIK-0001045810/13e6981b-95ed-4aac-a602-ebc5865d0590.pdf\"\n",
+        "pdf_path = os.path.join(\"../data\", pdf_url.split('/')[-1])\n",
+        "os.makedirs(os.path.dirname(pdf_path), exist_ok=True)\n",
+        "\n",
+        "response = requests.get(pdf_url)\n",
+        "with open(pdf_path, \"wb\") as f:\n",
+        "    f.write(response.content)\n",
+        "print(f\"Downloaded {pdf_url}\")\n",
+        "\n",
+        "doc_id = pi_client.submit_document(pdf_path)[\"doc_id\"]\n",
+        "print('Document Submitted:', doc_id)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "U4hpLB4T-fCt"
+      },
+      "source": [
+        "### Check the processing status"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 22,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "PB1S_CWd2n87",
+        "outputId": "c1416161-a1d6-4f9e-873c-7f6e26c8fa5f"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "{'createdAt': '2025-11-20T07:11:44.669000',\n",
+            " 'description': \"This document is NVIDIA Corporation's Form 10-Q Quarterly \"\n",
+            "                'Report for the period ending October 26, 2025, detailing its '\n",
+            "                'financial performance, operational results, market risks, and '\n",
+            "                'legal proceedings.',\n",
+            " 'id': 'pi-cmi73f7r7022y09nwn40paaom',\n",
+            " 'name': '13e6981b-95ed-4aac-a602-ebc5865d0590.pdf',\n",
+            " 'pageNum': 48,\n",
+            " 'status': 'completed'}\n",
+            "\n",
+            " Document ready! (48 pages)\n"
+          ]
+        }
+      ],
+      "source": [
+        "from pprint import pprint\n",
+        "\n",
+        "doc_info = pi_client.get_document(doc_id)\n",
+        "pprint(doc_info)\n",
+        "\n",
+        "if doc_info['status'] == 'completed':\n",
+        "  print(f\"\\n Document ready! ({doc_info['pageNum']} pages)\")\n",
+        "elif doc_info['status'] == 'processing':\n",
+        "  print(\"\\n Document is still processing. Please wait and check again.\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "z1C9FOvO-p1m"
+      },
+      "source": [
+        "### Ask a question about this document"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 24,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "X3RbQvy_0nt7",
+        "outputId": "a8bb5468-3f5f-4225-a5fd-60004041a644"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "I'll help you find the revenue information from NVIDIA's 10-Q report. Let me first check the document structure to locate the financial information.{\"doc_name\": \"13e6981b-95ed-4aac-a602-ebc5865d0590.pdf\"}Now let me get the content from pages that contain the revenue information. Based on the structure, revenue information should be in the financial statements section, which appears early in the document. Let me check pages 3-8 which should contain the Condensed Consolidated Statements of Income.{\"doc_name\": \"13e6981b-95ed-4aac-a602-ebc5865d0590.pdf\", \"pages\": \"3-5\"}Perfect! I found the revenue information. Let me provide you with the details:\n",
+            "\n",
+            "## Revenue Information from NVIDIA's 10-Q Report\n",
+            "\n",
+            "**You can find the revenue on Page 3** of the document in the **Condensed Consolidated Statements of Income**.\n",
+            "\n",
+            "### Revenue Details:\n",
+            "\n",
+            "**For the Three Months Ended October 26, 2025 (Q3 FY2026):**\n",
+            "- **Revenue: $57,006 million** ($57.0 billion)\n",
+            "- Compared to Q3 FY2025 (October 27, 2024): $35,082 million\n",
+            "\n",
+            "**For the Nine Months Ended October 26, 2025 (First 9 months of FY2026):**\n",
+            "- **Revenue: $147,811 million** ($147.8 billion)\n",
+            "- Compared to same period FY2025: $91,166 million\n",
+            "\n",
+            "### Key Highlights:\n",
+            "- Q3 revenue increased by **62.5%** year-over-year ($21.9 billion increase)\n",
+            "- Nine-month revenue increased by **62.1%** year-over-year ($56.6 billion increase)\n",
+            "- This represents strong growth driven primarily by Data Center compute and networking platforms for AI and accelerated computing, with Blackwell architectures being a major contributor\n",
+            "\n",
+            "The revenue figures are clearly displayed at the top of the Condensed Consolidated Statements of Income on **Page 3** of the 10-Q report."
+          ]
+        }
+      ],
+      "source": [
+        "query = \"what is the revenue? Also show me which page I can find it.\"\n",
+        "\n",
+        "for chunk in pi_client.chat_completions(\n",
+        "    messages=[{\"role\": \"user\", \"content\": query}],\n",
+        "    doc_id=doc_id,\n",
+        "    stream=True\n",
+        "):\n",
+        "    print(chunk, end='', flush=True)"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

cookbook/pageindex_RAG_simple.ipynb

@@ -0,0 +1,609 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "TCh9BTedHJK1"
+      },
+      "source": [
+        "![pageindex_banner](https://pageindex.ai/static/images/pageindex_banner.jpg)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "nD0hb4TFHWTt"
+      },
+      "source": [
+        "<p align=\"center\"><i>Reasoning-based RAG&nbsp; ✧ &nbsp;No Vector DB&nbsp; ✧ &nbsp;No Chunking&nbsp; ✧ &nbsp;Human-like Retrieval</i></p>\n",
+        "\n",
+        "<p align=\"center\">\n",
+        "  <a href=\"https://vectify.ai\">🏠 Homepage</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://dash.pageindex.ai\">🖥️ Dashboard</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://docs.pageindex.ai/quickstart\">📚 API Docs</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://github.com/VectifyAI/PageIndex\">📦 GitHub</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://discord.com/invite/VuXuf29EUj\">💬 Discord</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://ii2abc2jejf.typeform.com/to/tK3AXl8T\">✉️ Contact</a>&nbsp;\n",
+        "</p>\n",
+        "\n",
+        "<div align=\"center\">\n",
+        "\n",
+        "[![Star us on GitHub](https://img.shields.io/github/stars/VectifyAI/PageIndex?style=for-the-badge&logo=github&label=⭐️%20Star%20Us)](https://github.com/VectifyAI/PageIndex) &nbsp;&nbsp; [![Follow us on X](https://img.shields.io/badge/Follow%20Us-000000?style=for-the-badge&logo=x&logoColor=white)](https://twitter.com/VectifyAI)\n",
+        "\n",
+        "</div>\n",
+        "\n",
+        "---"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Ebvn5qfpcG1K"
+      },
+      "source": [
+        "# Simple Vectorless RAG with PageIndex"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## PageIndex Introduction\n",
+        "PageIndex is a new **reasoning-based**, **vectorless RAG** framework that performs retrieval in two steps:  \n",
+        "1. Generate a tree structure index of documents  \n",
+        "2. Perform reasoning-based retrieval through tree search  \n",
+        "\n",
+        "<div align=\"center\">\n",
+        "  <img src=\"https://docs.pageindex.ai/images/cookbook/vectorless-rag.png\" width=\"70%\">\n",
+        "</div>\n",
+        "\n",
+        "Compared to traditional vector-based RAG, PageIndex features:\n",
+        "- **No Vectors Needed**: Uses document structure and LLM reasoning for retrieval.\n",
+        "- **No Chunking Needed**: Documents are organized into natural sections rather than artificial chunks.\n",
+        "- **Human-like Retrieval**: Simulates how human experts navigate and extract knowledge from complex documents. \n",
+        "- **Transparent Retrieval Process**: Retrieval based on reasoning — say goodbye to approximate semantic search (\"vibe retrieval\")."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## 📝 Notebook Overview\n",
+        "\n",
+        "This notebook demonstrates a simple, minimal example of **vectorless RAG** with PageIndex. You will learn how to:\n",
+        "- [x] Build a PageIndex tree structure of a document\n",
+        "- [x] Perform reasoning-based retrieval with tree search\n",
+        "- [x] Generate answers based on the retrieved context\n",
+        "\n",
+        "> ⚡ Note: This is a **minimal example** to illustrate PageIndex's core philosophy and idea, not its full capabilities. More advanced examples are coming soon.\n",
+        "\n",
+        "---"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "7ziuTbbWcG1L"
+      },
+      "source": [
+        "## Step 0: Preparation\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "edTfrizMFK4c"
+      },
+      "source": [
+        "#### 0.1 Install PageIndex"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": true,
+        "id": "LaoB58wQFNDh"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install -q --upgrade pageindex"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "WVEWzPKGcG1M"
+      },
+      "source": [
+        "#### 0.2 Setup PageIndex"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "StvqfcK4cG1M"
+      },
+      "outputs": [],
+      "source": [
+        "from pageindex import PageIndexClient\n",
+        "import pageindex.utils as utils\n",
+        "\n",
+        "# Get your PageIndex API key from https://dash.pageindex.ai/api-keys\n",
+        "PAGEINDEX_API_KEY = \"YOUR_PAGEINDEX_API_KEY\"\n",
+        "pi_client = PageIndexClient(api_key=PAGEINDEX_API_KEY)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 0.3 Setup LLM\n",
+        "\n",
+        "Choose your preferred LLM for reasoning-based retrieval. In this example, we use OpenAI’s GPT-4.1."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import openai\n",
+        "OPENAI_API_KEY = \"YOUR_OPENAI_API_KEY\"\n",
+        "\n",
+        "async def call_llm(prompt, model=\"gpt-4.1\", temperature=0):\n",
+        "    client = openai.AsyncOpenAI(api_key=OPENAI_API_KEY)\n",
+        "    response = await client.chat.completions.create(\n",
+        "        model=model,\n",
+        "        messages=[{\"role\": \"user\", \"content\": prompt}],\n",
+        "        temperature=temperature\n",
+        "    )\n",
+        "    return response.choices[0].message.content.strip()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "heGtIMOVcG1N"
+      },
+      "source": [
+        "## Step 1: PageIndex Tree Generation"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Mzd1VWjwMUJL"
+      },
+      "source": [
+        "#### 1.1 Submit a document for generating PageIndex tree"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "f6--eZPLcG1N",
+        "outputId": "ca688cfd-6c4b-4a57-dac2-f3c2604c4112"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Downloaded https://arxiv.org/pdf/2501.12948.pdf\n",
+            "Document Submitted: pi-cmeseq08w00vt0bo3u6tr244g\n"
+          ]
+        }
+      ],
+      "source": [
+        "import os, requests\n",
+        "\n",
+        "# You can also use our GitHub repo to generate PageIndex tree\n",
+        "# https://github.com/VectifyAI/PageIndex\n",
+        "\n",
+        "pdf_url = \"https://arxiv.org/pdf/2501.12948.pdf\"\n",
+        "pdf_path = os.path.join(\"../data\", pdf_url.split('/')[-1])\n",
+        "os.makedirs(os.path.dirname(pdf_path), exist_ok=True)\n",
+        "\n",
+        "response = requests.get(pdf_url)\n",
+        "with open(pdf_path, \"wb\") as f:\n",
+        "    f.write(response.content)\n",
+        "print(f\"Downloaded {pdf_url}\")\n",
+        "\n",
+        "doc_id = pi_client.submit_document(pdf_path)[\"doc_id\"]\n",
+        "print('Document Submitted:', doc_id)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4-Hrh0azcG1N"
+      },
+      "source": [
+        "#### 1.2 Get the generated PageIndex tree structure"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 1000
+        },
+        "id": "b1Q1g6vrcG1O",
+        "outputId": "dc944660-38ad-47ea-d358-be422edbae53"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Simplified Tree Structure of the Document:\n",
+            "[{'title': 'DeepSeek-R1: Incentivizing Reasoning Cap...',\n",
+            "  'node_id': '0000',\n",
+            "  'prefix_summary': '# DeepSeek-R1: Incentivizing Reasoning C...',\n",
+            "  'nodes': [{'title': 'Abstract',\n",
+            "             'node_id': '0001',\n",
+            "             'summary': 'The partial document introduces two reas...'},\n",
+            "            {'title': 'Contents',\n",
+            "             'node_id': '0002',\n",
+            "             'summary': 'This partial document provides a detaile...'},\n",
+            "            {'title': '1. Introduction',\n",
+            "             'node_id': '0003',\n",
+            "             'prefix_summary': 'The partial document introduces recent a...',\n",
+            "             'nodes': [{'title': '1.1. Contributions',\n",
+            "                        'node_id': '0004',\n",
+            "                        'summary': 'This partial document outlines the main ...'},\n",
+            "                       {'title': '1.2. Summary of Evaluation Results',\n",
+            "                        'node_id': '0005',\n",
+            "                        'summary': 'The partial document provides a summary ...'}]},\n",
+            "            {'title': '2. Approach',\n",
+            "             'node_id': '0006',\n",
+            "             'prefix_summary': '## 2. Approach\\n',\n",
+            "             'nodes': [{'title': '2.1. Overview',\n",
+            "                        'node_id': '0007',\n",
+            "                        'summary': '### 2.1. Overview\\n\\nPrevious work has hea...'},\n",
+            "                       {'title': '2.2. DeepSeek-R1-Zero: Reinforcement Lea...',\n",
+            "                        'node_id': '0008',\n",
+            "                        'prefix_summary': '### 2.2. DeepSeek-R1-Zero: Reinforcement...',\n",
+            "                        'nodes': [{'title': '2.2.1. Reinforcement Learning Algorithm',\n",
+            "                                   'node_id': '0009',\n",
+            "                                   'summary': 'The partial document describes the Group...'},\n",
+            "                                  {'title': '2.2.2. Reward Modeling',\n",
+            "                                   'node_id': '0010',\n",
+            "                                   'summary': 'This partial document discusses the rewa...'},\n",
+            "                                  {'title': '2.2.3. Training Template',\n",
+            "                                   'node_id': '0011',\n",
+            "                                   'summary': '#### 2.2.3. Training Template\\n\\nTo train ...'},\n",
+            "                                  {'title': '2.2.4. Performance, Self-evolution Proce...',\n",
+            "                                   'node_id': '0012',\n",
+            "                                   'summary': 'This partial document discusses the perf...'}]},\n",
+            "                       {'title': '2.3. DeepSeek-R1: Reinforcement Learning...',\n",
+            "                        'node_id': '0013',\n",
+            "                        'summary': 'This partial document describes the trai...'},\n",
+            "                       {'title': '2.4. Distillation: Empower Small Models ...',\n",
+            "                        'node_id': '0014',\n",
+            "                        'summary': 'This partial document discusses the proc...'}]},\n",
+            "            {'title': '3. Experiment',\n",
+            "             'node_id': '0015',\n",
+            "             'prefix_summary': 'The partial document describes the exper...',\n",
+            "             'nodes': [{'title': '3.1. DeepSeek-R1 Evaluation',\n",
+            "                        'node_id': '0016',\n",
+            "                        'summary': 'This partial document presents a compreh...'},\n",
+            "                       {'title': '3.2. Distilled Model Evaluation',\n",
+            "                        'node_id': '0017',\n",
+            "                        'summary': 'This partial document presents an evalua...'}]},\n",
+            "            {'title': '4. Discussion',\n",
+            "             'node_id': '0018',\n",
+            "             'summary': 'This partial document discusses the comp...'},\n",
+            "            {'title': '5. Conclusion, Limitations, and Future W...',\n",
+            "             'node_id': '0019',\n",
+            "             'summary': 'This partial document presents the concl...'},\n",
+            "            {'title': 'References',\n",
+            "             'node_id': '0020',\n",
+            "             'summary': 'This partial document consists of the re...'},\n",
+            "            {'title': 'Appendix', 'node_id': '0021', 'summary': '## Appendix\\n'},\n",
+            "            {'title': 'A. Contributions and Acknowledgments',\n",
+            "             'node_id': '0022',\n",
+            "             'summary': 'This partial document section details th...'}]}]\n"
+          ]
+        }
+      ],
+      "source": [
+        "if pi_client.is_retrieval_ready(doc_id):\n",
+        "    tree = pi_client.get_tree(doc_id, node_summary=True)['result']\n",
+        "    print('Simplified Tree Structure of the Document:')\n",
+        "    utils.print_tree(tree)\n",
+        "else:\n",
+        "    print(\"Processing document, please try again later...\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "USoCLOiQcG1O"
+      },
+      "source": [
+        "## Step 2: Reasoning-Based Retrieval with Tree Search"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 2.1 Use LLM for tree search and identify nodes that might contain relevant context"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 21,
+      "metadata": {
+        "id": "LLHNJAtTcG1O"
+      },
+      "outputs": [],
+      "source": [
+        "import json\n",
+        "\n",
+        "query = \"What are the conclusions in this document?\"\n",
+        "\n",
+        "tree_without_text = utils.remove_fields(tree.copy(), fields=['text'])\n",
+        "\n",
+        "search_prompt = f\"\"\"\n",
+        "You are given a question and a tree structure of a document.\n",
+        "Each node contains a node id, node title, and a corresponding summary.\n",
+        "Your task is to find all nodes that are likely to contain the answer to the question.\n",
+        "\n",
+        "Question: {query}\n",
+        "\n",
+        "Document tree structure:\n",
+        "{json.dumps(tree_without_text, indent=2)}\n",
+        "\n",
+        "Please reply in the following JSON format:\n",
+        "{{\n",
+        "    \"thinking\": \"<Your thinking process on which nodes are relevant to the question>\",\n",
+        "    \"node_list\": [\"node_id_1\", \"node_id_2\", ..., \"node_id_n\"]\n",
+        "}}\n",
+        "Directly return the final JSON structure. Do not output anything else.\n",
+        "\"\"\"\n",
+        "\n",
+        "tree_search_result = await call_llm(search_prompt)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 2.2 Print retrieved nodes and reasoning process"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 57,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 206
+        },
+        "id": "P8DVUOuAen5u",
+        "outputId": "6bb6d052-ef30-4716-f88e-be98bcb7ebdb"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Reasoning Process:\n",
+            "The question asks for the conclusions in the document. Typically, conclusions are found in sections\n",
+            "explicitly titled 'Conclusion' or in sections summarizing the findings and implications of the work.\n",
+            "In this document tree, node 0019 ('5. Conclusion, Limitations, and Future Work') is the most\n",
+            "directly relevant, as it is dedicated to the conclusion and related topics. Additionally, the\n",
+            "'Abstract' (node 0001) may contain a high-level summary that sometimes includes concluding remarks,\n",
+            "but it is less likely to contain the full conclusions. Other sections like 'Discussion' (node 0018)\n",
+            "may discuss implications but are not explicitly conclusions. Therefore, the primary node is 0019.\n",
+            "\n",
+            "Retrieved Nodes:\n",
+            "Node ID: 0019\t Page: 16\t Title: 5. Conclusion, Limitations, and Future Work\n"
+          ]
+        }
+      ],
+      "source": [
+        "node_map = utils.create_node_mapping(tree)\n",
+        "tree_search_result_json = json.loads(tree_search_result)\n",
+        "\n",
+        "print('Reasoning Process:')\n",
+        "utils.print_wrapped(tree_search_result_json['thinking'])\n",
+        "\n",
+        "print('\\nRetrieved Nodes:')\n",
+        "for node_id in tree_search_result_json[\"node_list\"]:\n",
+        "    node = node_map[node_id]\n",
+        "    print(f\"Node ID: {node['node_id']}\\t Page: {node['page_index']}\\t Title: {node['title']}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "10wOZDG_cG1O"
+      },
+      "source": [
+        "## Step 3: Answer Generation"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 3.1 Extract relevant context from retrieved nodes"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 58,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 279
+        },
+        "id": "a7UCBnXlcG1O",
+        "outputId": "8a026ea3-4ef3-473a-a57b-b4565409749e"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Retrieved Context:\n",
+            "\n",
+            "## 5. Conclusion, Limitations, and Future Work\n",
+            "\n",
+            "In this work, we share our journey in enhancing model reasoning abilities through reinforcement\n",
+            "learning. DeepSeek-R1-Zero represents a pure RL approach without relying on cold-start data,\n",
+            "achieving strong performance across various tasks. DeepSeek-R1 is more powerful, leveraging cold-\n",
+            "start data alongside iterative RL fine-tuning. Ultimately, DeepSeek-R1 achieves performance\n",
+            "comparable to OpenAI-o1-1217 on a range of tasks.\n",
+            "\n",
+            "We further explore distillation the reasoning capability to small dense models. We use DeepSeek-R1\n",
+            "as the teacher model to generate 800K training samples, and fine-tune several small dense models.\n",
+            "The results are promising: DeepSeek-R1-Distill-Qwen-1.5B outperforms GPT-4o and Claude-3.5-Sonnet on\n",
+            "math benchmarks with $28.9 \\%$ on AIME and $83.9 \\%$ on MATH. Other dense models also achieve\n",
+            "impressive results, significantly outperforming other instructiontuned models based on the same\n",
+            "underlying checkpoints.\n",
+            "\n",
+            "In the fut...\n"
+          ]
+        }
+      ],
+      "source": [
+        "node_list = json.loads(tree_search_result)[\"node_list\"]\n",
+        "relevant_content = \"\\n\\n\".join(node_map[node_id][\"text\"] for node_id in node_list)\n",
+        "\n",
+        "print('Retrieved Context:\\n')\n",
+        "utils.print_wrapped(relevant_content[:1000] + '...')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 3.2 Generate answer based on retrieved context"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 59,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 210
+        },
+        "id": "tcp_PhHzcG1O",
+        "outputId": "187ff116-9bb0-4ab4-bacb-13944460b5ff"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Generated Answer:\n",
+            "\n",
+            "The conclusions in this document are:\n",
+            "\n",
+            "- DeepSeek-R1-Zero, a pure reinforcement learning (RL) approach without cold-start data, achieves\n",
+            "strong performance across various tasks.\n",
+            "- DeepSeek-R1, which combines cold-start data with iterative RL fine-tuning, is more powerful and\n",
+            "achieves performance comparable to OpenAI-o1-1217 on a range of tasks.\n",
+            "- Distilling DeepSeek-R1’s reasoning capabilities into smaller dense models is promising; for\n",
+            "example, DeepSeek-R1-Distill-Qwen-1.5B outperforms GPT-4o and Claude-3.5-Sonnet on math benchmarks,\n",
+            "and other dense models also show significant improvements over similar instruction-tuned models.\n",
+            "\n",
+            "These results demonstrate the effectiveness of the RL-based approach and the potential for\n",
+            "distilling reasoning abilities into smaller models.\n"
+          ]
+        }
+      ],
+      "source": [
+        "answer_prompt = f\"\"\"\n",
+        "Answer the question based on the context:\n",
+        "\n",
+        "Question: {query}\n",
+        "Context: {relevant_content}\n",
+        "\n",
+        "Provide a clear, concise answer based only on the context provided.\n",
+        "\"\"\"\n",
+        "\n",
+        "print('Generated Answer:\\n')\n",
+        "answer = await call_llm(answer_prompt)\n",
+        "utils.print_wrapped(answer)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "_1kaGD3GcG1O"
+      },
+      "source": [
+        "---\n",
+        "\n",
+        "## 🎯 What's Next\n",
+        "\n",
+        "This notebook has demonstrated a **basic**, **minimal** example of **reasoning-based**, **vectorless** RAG with PageIndex. The workflow illustrates the core idea:\n",
+        "> *Generating a hierarchical tree structure from a document, reasoning over that tree structure, and extracting relevant context, without relying on a vector database or top-k similarity search*.\n",
+        "\n",
+        "While this notebook highlights a minimal workflow, the PageIndex framework is built to support **far more advanced** use cases. In upcoming tutorials, we will introduce:\n",
+        "* **Multi-Node Reasoning with Content Extraction** — Scale tree search to extract and select relevant content from multiple nodes.\n",
+        "* **Multi-Document Search** — Enable reasoning-based navigation across large document collections, extending beyond a single file.\n",
+        "* **Efficient Tree Search** — Improve tree search efficiency for long documents with a large number of nodes.\n",
+        "* **Expert Knowledge Integration and Preference Alignment** — Incorporate user preferences or expert insights by adding knowledge directly into the LLM tree search, without the need for fine-tuning.\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## 🔎 Learn More About PageIndex\n",
+        "  <a href=\"https://vectify.ai\">🏠 Homepage</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://dash.pageindex.ai\">🖥️ Dashboard</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://docs.pageindex.ai/quickstart\">📚 API Docs</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://github.com/VectifyAI/PageIndex\">📦 GitHub</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://discord.com/invite/VuXuf29EUj\">💬 Discord</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://ii2abc2jejf.typeform.com/to/tK3AXl8T\">✉️ Contact</a>\n",
+        "\n",
+        "<br>\n",
+        "\n",
+        "© 2025 [Vectify AI](https://vectify.ai)"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.11.9"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

cookbook/vision_RAG_pageindex.ipynb

@@ -0,0 +1,667 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "TCh9BTedHJK1"
+      },
+      "source": [
+        "![pageindex_banner](https://pageindex.ai/static/images/pageindex_banner.jpg)\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "nD0hb4TFHWTt"
+      },
+      "source": [
+        "<div align=\"center\">\n",
+        "<p><i>Reasoning-based RAG&nbsp; ◦ &nbsp;No Vector DB&nbsp; ◦ &nbsp;No Chunking&nbsp; ◦ &nbsp;Human-like Retrieval</i></p>\n",
+        "</div>\n",
+        "\n",
+        "<div align=\"center\">\n",
+        "<p>\n",
+        "  <a href=\"https://vectify.ai\">🏠 Homepage</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://chat.pageindex.ai\">💻 Chat</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://pageindex.ai/mcp\">🔌 MCP</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://docs.pageindex.ai/quickstart\">📚 API</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://github.com/VectifyAI/PageIndex\">📦 GitHub</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://discord.com/invite/VuXuf29EUj\">💬 Discord</a>&nbsp; • &nbsp;\n",
+        "  <a href=\"https://ii2abc2jejf.typeform.com/to/tK3AXl8T\">✉️ Contact</a>&nbsp;\n",
+        "</p>\n",
+        "</div>\n",
+        "\n",
+        "<div align=\"center\">\n",
+        "\n",
+        "[![Star us on GitHub](https://img.shields.io/github/stars/VectifyAI/PageIndex?style=for-the-badge&logo=github&label=⭐️%20Star%20Us)](https://github.com/VectifyAI/PageIndex) &nbsp;&nbsp; [![Follow us on X](https://img.shields.io/badge/Follow%20Us-000000?style=for-the-badge&logo=x&logoColor=white)](https://twitter.com/VectifyAI)\n",
+        "\n",
+        "</div>\n",
+        "\n",
+        "---"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "> Check out our blog post, \"[Do We Still Need OCR?](https://pageindex.ai/blog/do-we-need-ocr)\", for a more detailed discussion."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Ebvn5qfpcG1K"
+      },
+      "source": [
+        "# A Vision-based, Vectorless RAG System for Long Documents\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "In modern document question answering (QA) systems, Optical Character Recognition (OCR) serves an important role by converting PDF pages into text that can be processed by Large Language Models (LLMs). The resulting text can provide contextual input that enables LLMs to perform question answering over document content.\n",
+        "\n",
+        "Traditional OCR systems typically use a two-stage process that first detects the layout of a PDF — dividing it into text, tables, and images — and then recognizes and converts these elements into plain text. With the rise of vision-language models (VLMs) (such as [Qwen-VL](https://github.com/QwenLM/Qwen3-VL) and [GPT-4.1](https://openai.com/index/gpt-4-1/)), new end-to-end OCR models like [DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR) have emerged. These models jointly understand visual and textual information, enabling direct interpretation of PDFs without an explicit layout detection step.\n",
+        "\n",
+        "However, this paradigm shift raises an important question: \n",
+        "\n",
+        "\n",
+        "> **If a VLM can already process both the document images and the query to produce an answer directly, do we still need the intermediate OCR step?**\n",
+        "\n",
+        "In this notebook, we give a practical implementation of a vision-based question-answering system for long documents, without relying on OCR. Specifically, we use PageIndex as a reasoning-based retrieval layer and OpenAI's multimodal GPT-4.1 as the VLM for visual reasoning and answer generation.\n",
+        "\n",
+        "See the original [blog post](https://pageindex.ai/blog/do-we-need-ocr) for a more detailed discussion on how VLMs can replace traditional OCR pipelines in document question-answering."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## 📝 Notebook Overview\n",
+        "\n",
+        "This notebook demonstrates a *minimal*, **vision-based vectorless RAG** pipeline for long documents with PageIndex, using only visual context from PDF pages. You will learn how to:\n",
+        "- [x] Build a PageIndex tree structure of a document\n",
+        "- [x] Perform reasoning-based retrieval with tree search\n",
+        "- [x] Extract PDF page images of retrieved tree nodes for visual context\n",
+        "- [x] Generate answers using VLM with PDF image inputs only (no OCR required)\n",
+        "\n",
+        "> ⚡ Note: This example uses PageIndex's reasoning-based retrieval with OpenAI's multimodal GPT-4.1 model for both tree search and visual context reasoning.\n",
+        "\n",
+        "---"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "7ziuTbbWcG1L"
+      },
+      "source": [
+        "## Step 0: Preparation\n",
+        "\n",
+        "This notebook demonstrates **Vision-based RAG** with PageIndex, using PDF page images as visual context for retrieval and answer generation.\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "edTfrizMFK4c"
+      },
+      "source": [
+        "#### 0.1 Install PageIndex"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": true,
+        "id": "LaoB58wQFNDh"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install -q --upgrade pageindex requests openai PyMuPDF"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "WVEWzPKGcG1M"
+      },
+      "source": [
+        "#### 0.2 Setup PageIndex"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "StvqfcK4cG1M"
+      },
+      "outputs": [],
+      "source": [
+        "from pageindex import PageIndexClient\n",
+        "import pageindex.utils as utils\n",
+        "\n",
+        "# Get your PageIndex API key from https://dash.pageindex.ai/api-keys\n",
+        "PAGEINDEX_API_KEY = \"YOUR_PAGEINDEX_API_KEY\"\n",
+        "pi_client = PageIndexClient(api_key=PAGEINDEX_API_KEY)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 0.3 Setup VLM\n",
+        "\n",
+        "Choose your preferred VLM — in this notebook, we use OpenAI's multimodal GPT-4.1 as the VLM."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import openai, fitz, base64, os\n",
+        "\n",
+        "# Setup OpenAI client\n",
+        "OPENAI_API_KEY = \"YOUR_OPENAI_API_KEY\"\n",
+        "\n",
+        "async def call_vlm(prompt, image_paths=None, model=\"gpt-4.1\"):\n",
+        "    client = openai.AsyncOpenAI(api_key=OPENAI_API_KEY)\n",
+        "    messages = [{\"role\": \"user\", \"content\": prompt}]\n",
+        "    if image_paths:\n",
+        "        content = [{\"type\": \"text\", \"text\": prompt}]\n",
+        "        for image in image_paths:\n",
+        "            if os.path.exists(image):\n",
+        "                with open(image, \"rb\") as image_file:\n",
+        "                    image_data = base64.b64encode(image_file.read()).decode('utf-8')\n",
+        "                    content.append({\n",
+        "                        \"type\": \"image_url\",\n",
+        "                        \"image_url\": {\n",
+        "                            \"url\": f\"data:image/jpeg;base64,{image_data}\"\n",
+        "                        }\n",
+        "                    })\n",
+        "        messages[0][\"content\"] = content\n",
+        "    response = await client.chat.completions.create(model=model, messages=messages, temperature=0)\n",
+        "    return response.choices[0].message.content.strip()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 0.4 PDF Image Extraction Helper Functions\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "def extract_pdf_page_images(pdf_path, output_dir=\"pdf_images\"):\n",
+        "    os.makedirs(output_dir, exist_ok=True)\n",
+        "    pdf_document = fitz.open(pdf_path)\n",
+        "    page_images = {}\n",
+        "    total_pages = len(pdf_document)\n",
+        "    for page_number in range(len(pdf_document)):\n",
+        "        page = pdf_document.load_page(page_number)\n",
+        "        # Convert page to image\n",
+        "        mat = fitz.Matrix(2.0, 2.0)  # 2x zoom for better quality\n",
+        "        pix = page.get_pixmap(matrix=mat)\n",
+        "        img_data = pix.tobytes(\"jpeg\")\n",
+        "        image_path = os.path.join(output_dir, f\"page_{page_number + 1}.jpg\")\n",
+        "        with open(image_path, \"wb\") as image_file:\n",
+        "            image_file.write(img_data)\n",
+        "        page_images[page_number + 1] = image_path\n",
+        "        print(f\"Saved page {page_number + 1} image: {image_path}\")\n",
+        "    pdf_document.close()\n",
+        "    return page_images, total_pages\n",
+        "\n",
+        "def get_page_images_for_nodes(node_list, node_map, page_images):\n",
+        "    # Get PDF page images for retrieved nodes\n",
+        "    image_paths = []\n",
+        "    seen_pages = set()\n",
+        "    for node_id in node_list:\n",
+        "        node_info = node_map[node_id]\n",
+        "        for page_num in range(node_info['start_index'], node_info['end_index'] + 1):\n",
+        "            if page_num not in seen_pages:\n",
+        "                image_paths.append(page_images[page_num])\n",
+        "                seen_pages.add(page_num)\n",
+        "    return image_paths\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "heGtIMOVcG1N"
+      },
+      "source": [
+        "## Step 1: PageIndex Tree Generation"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Mzd1VWjwMUJL"
+      },
+      "source": [
+        "#### 1.1 Submit a document for generating PageIndex tree"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/"
+        },
+        "id": "f6--eZPLcG1N",
+        "outputId": "ca688cfd-6c4b-4a57-dac2-f3c2604c4112"
+      },
+      "outputs": [],
+      "source": [
+        "import os, requests\n",
+        "\n",
+        "# You can also use our GitHub repo to generate PageIndex tree\n",
+        "# https://github.com/VectifyAI/PageIndex\n",
+        "\n",
+        "pdf_url = \"https://arxiv.org/pdf/1706.03762.pdf\"  # the \"Attention Is All You Need\" paper\n",
+        "pdf_path = os.path.join(\"../data\", pdf_url.split('/')[-1])\n",
+        "os.makedirs(os.path.dirname(pdf_path), exist_ok=True)\n",
+        "\n",
+        "response = requests.get(pdf_url)\n",
+        "with open(pdf_path, \"wb\") as f:\n",
+        "    f.write(response.content)\n",
+        "print(f\"Downloaded {pdf_url}\\n\")\n",
+        "\n",
+        "# Extract page images from PDF\n",
+        "print(\"Extracting page images...\")\n",
+        "page_images, total_pages = extract_pdf_page_images(pdf_path)\n",
+        "print(f\"Extracted {len(page_images)} page images from {total_pages} total pages.\\n\")\n",
+        "\n",
+        "doc_id = pi_client.submit_document(pdf_path)[\"doc_id\"]\n",
+        "print('Document Submitted:', doc_id)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4-Hrh0azcG1N"
+      },
+      "source": [
+        "#### 1.2 Get the generated PageIndex tree structure"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 65,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 1000
+        },
+        "id": "b1Q1g6vrcG1O",
+        "outputId": "dc944660-38ad-47ea-d358-be422edbae53"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Simplified Tree Structure of the Document:\n",
+            "[{'title': 'Attention Is All You Need',\n",
+            "  'node_id': '0000',\n",
+            "  'page_index': 1,\n",
+            "  'prefix_summary': '# Attention Is All You Need\\n\\nAshish Vasw...',\n",
+            "  'nodes': [{'title': 'Abstract',\n",
+            "             'node_id': '0001',\n",
+            "             'page_index': 1,\n",
+            "             'summary': 'The text introduces the Transformer, a n...'},\n",
+            "            {'title': '1 Introduction',\n",
+            "             'node_id': '0002',\n",
+            "             'page_index': 2,\n",
+            "             'summary': 'The text introduces the Transformer, a n...'},\n",
+            "            {'title': '2 Background',\n",
+            "             'node_id': '0003',\n",
+            "             'page_index': 2,\n",
+            "             'summary': 'This section discusses the Transformer m...'},\n",
+            "            {'title': '3 Model Architecture',\n",
+            "             'node_id': '0004',\n",
+            "             'page_index': 2,\n",
+            "             'prefix_summary': 'The text describes the encoder-decoder a...',\n",
+            "             'nodes': [{'title': '3.1 Encoder and Decoder Stacks',\n",
+            "                        'node_id': '0005',\n",
+            "                        'page_index': 3,\n",
+            "                        'summary': 'The text describes the encoder and decod...'},\n",
+            "                       {'title': '3.2 Attention',\n",
+            "                        'node_id': '0006',\n",
+            "                        'page_index': 3,\n",
+            "                        'prefix_summary': '### 3.2 Attention\\n\\nAn attention function...',\n",
+            "                        'nodes': [{'title': '3.2.1 Scaled Dot-Product Attention',\n",
+            "                                   'node_id': '0007',\n",
+            "                                   'page_index': 4,\n",
+            "                                   'summary': 'The text describes Scaled Dot-Product At...'},\n",
+            "                                  {'title': '3.2.2 Multi-Head Attention',\n",
+            "                                   'node_id': '0008',\n",
+            "                                   'page_index': 4,\n",
+            "                                   'summary': 'The text describes Multi-Head Attention,...'},\n",
+            "                                  {'title': '3.2.3 Applications of Attention in our M...',\n",
+            "                                   'node_id': '0009',\n",
+            "                                   'page_index': 5,\n",
+            "                                   'summary': 'The text describes the three application...'}]},\n",
+            "                       {'title': '3.3 Position-wise Feed-Forward Networks',\n",
+            "                        'node_id': '0010',\n",
+            "                        'page_index': 5,\n",
+            "                        'summary': '### 3.3 Position-wise Feed-Forward Netwo...'},\n",
+            "                       {'title': '3.4 Embeddings and Softmax',\n",
+            "                        'node_id': '0011',\n",
+            "                        'page_index': 5,\n",
+            "                        'summary': 'The text describes the use of learned em...'},\n",
+            "                       {'title': '3.5 Positional Encoding',\n",
+            "                        'node_id': '0012',\n",
+            "                        'page_index': 6,\n",
+            "                        'summary': 'This section explains the necessity of p...'}]},\n",
+            "            {'title': '4 Why Self-Attention',\n",
+            "             'node_id': '0013',\n",
+            "             'page_index': 6,\n",
+            "             'summary': 'This text compares self-attention layers...'},\n",
+            "            {'title': '5 Training',\n",
+            "             'node_id': '0014',\n",
+            "             'page_index': 7,\n",
+            "             'prefix_summary': '## 5 Training\\n\\nThis section describes th...',\n",
+            "             'nodes': [{'title': '5.1 Training Data and Batching',\n",
+            "                        'node_id': '0015',\n",
+            "                        'page_index': 7,\n",
+            "                        'summary': '### 5.1 Training Data and Batching\\n\\nWe t...'},\n",
+            "                       {'title': '5.2 Hardware and Schedule',\n",
+            "                        'node_id': '0016',\n",
+            "                        'page_index': 7,\n",
+            "                        'summary': '### 5.2 Hardware and Schedule\\n\\nWe traine...'},\n",
+            "                       {'title': '5.3 Optimizer',\n",
+            "                        'node_id': '0017',\n",
+            "                        'page_index': 7,\n",
+            "                        'summary': '### 5.3 Optimizer\\n\\nWe used the Adam opti...'},\n",
+            "                       {'title': '5.4 Regularization',\n",
+            "                        'node_id': '0018',\n",
+            "                        'page_index': 7,\n",
+            "                        'summary': 'The text details three regularization te...'}]},\n",
+            "            {'title': '6 Results',\n",
+            "             'node_id': '0019',\n",
+            "             'page_index': 8,\n",
+            "             'prefix_summary': '## 6 Results\\n',\n",
+            "             'nodes': [{'title': '6.1 Machine Translation',\n",
+            "                        'node_id': '0020',\n",
+            "                        'page_index': 8,\n",
+            "                        'summary': 'The text details the performance of a Tr...'},\n",
+            "                       {'title': '6.2 Model Variations',\n",
+            "                        'node_id': '0021',\n",
+            "                        'page_index': 8,\n",
+            "                        'summary': 'This text details experiments varying co...'},\n",
+            "                       {'title': '6.3 English Constituency Parsing',\n",
+            "                        'node_id': '0022',\n",
+            "                        'page_index': 9,\n",
+            "                        'summary': 'The text describes experiments evaluatin...'}]},\n",
+            "            {'title': '7 Conclusion',\n",
+            "             'node_id': '0023',\n",
+            "             'page_index': 10,\n",
+            "             'summary': 'This text concludes by presenting the Tr...'},\n",
+            "            {'title': 'References',\n",
+            "             'node_id': '0024',\n",
+            "             'page_index': 10,\n",
+            "             'summary': 'The provided text is a collection of ref...'},\n",
+            "            {'title': 'Attention Visualizations',\n",
+            "             'node_id': '0025',\n",
+            "             'page_index': 13,\n",
+            "             'summary': 'The text provides examples of attention ...'}]}]\n"
+          ]
+        }
+      ],
+      "source": [
+        "if pi_client.is_retrieval_ready(doc_id):\n",
+        "    tree = pi_client.get_tree(doc_id, node_summary=True)['result']\n",
+        "    print('Simplified Tree Structure of the Document:')\n",
+        "    utils.print_tree(tree, exclude_fields=['text'])\n",
+        "else:\n",
+        "    print(\"Processing document, please try again later...\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "USoCLOiQcG1O"
+      },
+      "source": [
+        "## Step 2: Reasoning-Based Retrieval with Tree Search"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 2.1 Reasoning-based retrieval with PageIndex to identify nodes that might contain relevant context"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "LLHNJAtTcG1O"
+      },
+      "outputs": [],
+      "source": [
+        "import json\n",
+        "\n",
+        "query = \"What is the last operation in the Scaled Dot-Product Attention figure?\"\n",
+        "\n",
+        "tree_without_text = utils.remove_fields(tree.copy(), fields=['text'])\n",
+        "\n",
+        "search_prompt = f\"\"\"\n",
+        "You are given a question and a tree structure of a document.\n",
+        "Each node contains a node id, node title, and a corresponding summary.\n",
+        "Your task is to find all tree nodes that are likely to contain the answer to the question.\n",
+        "\n",
+        "Question: {query}\n",
+        "\n",
+        "Document tree structure:\n",
+        "{json.dumps(tree_without_text, indent=2)}\n",
+        "\n",
+        "Please reply in the following JSON format:\n",
+        "{{\n",
+        "    \"thinking\": \"<Your thinking process on which nodes are relevant to the question>\",\n",
+        "    \"node_list\": [\"node_id_1\", \"node_id_2\", ..., \"node_id_n\"]\n",
+        "}}\n",
+        "Directly return the final JSON structure. Do not output anything else.\n",
+        "\"\"\"\n",
+        "\n",
+        "tree_search_result = await call_vlm(search_prompt)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 2.2 Print retrieved nodes and reasoning process"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 87,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 206
+        },
+        "id": "P8DVUOuAen5u",
+        "outputId": "6bb6d052-ef30-4716-f88e-be98bcb7ebdb"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Reasoning Process:\n",
+            "\n",
+            "The question asks about the last operation in the Scaled Dot-Product Attention figure. The most\n",
+            "relevant section is the one that describes Scaled Dot-Product Attention in detail, including its\n",
+            "computation and the figure itself. This is likely found in section 3.2.1 'Scaled Dot-Product\n",
+            "Attention' (node_id: 0007), which is a subsection of 3.2 'Attention' (node_id: 0006). The parent\n",
+            "section 3.2 may also contain the figure and its caption, as the summary mentions Figure 2 (which is\n",
+            "the Scaled Dot-Product Attention figure). Therefore, both node 0006 and node 0007 are likely to\n",
+            "contain the answer.\n",
+            "\n",
+            "Retrieved Nodes:\n",
+            "\n",
+            "Node ID: 0006\t Pages: 3-4\t Title: 3.2 Attention\n",
+            "Node ID: 0007\t Pages: 4\t Title: 3.2.1 Scaled Dot-Product Attention\n"
+          ]
+        }
+      ],
+      "source": [
+        "node_map = utils.create_node_mapping(tree, include_page_ranges=True, max_page=total_pages)\n",
+        "tree_search_result_json = json.loads(tree_search_result)\n",
+        "\n",
+        "print('Reasoning Process:\\n')\n",
+        "utils.print_wrapped(tree_search_result_json['thinking'])\n",
+        "\n",
+        "print('\\nRetrieved Nodes:\\n')\n",
+        "for node_id in tree_search_result_json[\"node_list\"]:\n",
+        "    node_info = node_map[node_id]\n",
+        "    node = node_info['node']\n",
+        "    start_page = node_info['start_index']\n",
+        "    end_page = node_info['end_index']\n",
+        "    page_range = start_page if start_page == end_page else f\"{start_page}-{end_page}\"\n",
+        "    print(f\"Node ID: {node['node_id']}\\t Pages: {page_range}\\t Title: {node['title']}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 2.3 Get corresponding PDF page images of retrieved nodes"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 81,
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "\n",
+            "Retrieved 2 PDF page image(s) for visual context.\n"
+          ]
+        }
+      ],
+      "source": [
+        "retrieved_nodes = tree_search_result_json[\"node_list\"]\n",
+        "retrieved_page_images = get_page_images_for_nodes(retrieved_nodes, node_map, page_images)\n",
+        "print(f'\\nRetrieved {len(retrieved_page_images)} PDF page image(s) for visual context.')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "10wOZDG_cG1O"
+      },
+      "source": [
+        "## Step 3: Answer Generation"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "#### 3.1 Generate answer using VLM with visual context"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 210
+        },
+        "id": "tcp_PhHzcG1O",
+        "outputId": "187ff116-9bb0-4ab4-bacb-13944460b5ff"
+      },
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Generated answer using VLM with retrieved PDF page images as visual context:\n",
+            "\n",
+            "The last operation in the **Scaled Dot-Product Attention** figure is a **MatMul** (matrix\n",
+            "multiplication). This operation multiplies the attention weights (after softmax) by the value matrix\n",
+            "\\( V \\).\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Generate answer using VLM with only PDF page images as visual context\n",
+        "answer_prompt = f\"\"\"\n",
+        "Answer the question based on the images of the document pages as context.\n",
+        "\n",
+        "Question: {query}\n",
+        "\n",
+        "Provide a clear, concise answer based only on the context provided.\n",
+        "\"\"\"\n",
+        "\n",
+        "print('Generated answer using VLM with retrieved PDF page images as visual context:\\n')\n",
+        "answer = await call_vlm(answer_prompt, retrieved_page_images)\n",
+        "utils.print_wrapped(answer)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Conclusion\n",
+        "\n",
+        "In this notebook, we demonstrated a *minimal* **vision-based, vectorless RAG pipeline** using PageIndex and a VLM. The system retrieves relevant pages by reasoning over the document’s hierarchical tree index and answers questions directly from PDF images — no OCR required.\n",
+        "\n",
+        "If you’re interested in building your own **reasoning-based document QA system**, try [PageIndex Chat](https://chat.pageindex.ai), or integrate via [PageIndex MCP](https://pageindex.ai/mcp) and the [API](https://docs.pageindex.ai/quickstart). You can also explore the [GitHub repo](https://github.com/VectifyAI/PageIndex) for open-source implementations and additional examples."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "\n",
+        "\n",
+        "© 2025 [Vectify AI](https://vectify.ai)"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.11.9"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

llm_config.py

@@ -0,0 +1,39 @@
+import os
+from openai import OpenAI
+
+def get_llm_client(provider="nvidia"):
+    """
+    Returns an OpenAI client configured for the specified provider.
+    
+    Args:
+        provider (str): "nvidia" or "mistral"
+    
+    Returns:
+        OpenAI: The configured client
+    """
+    if provider == "nvidia":
+        # Llama 4 Maverick via NVIDIA NIM
+        api_key = os.getenv("NVIDIA_API_KEY")
+        if not api_key:
+            print("Warning: NVIDIA_API_KEY not found in environment variables.")
+            
+        return OpenAI(
+            base_url="https://integrate.api.nvidia.com/v1",
+            api_key=api_key
+        )
+    else:
+        # Mistral Large 3 via Mistral API
+        api_key = os.getenv("MISTRAL_API_KEY")
+        if not api_key:
+            print("Warning: MISTRAL_API_KEY not found in environment variables.")
+
+        return OpenAI(
+            base_url="https://api.mistral.ai/v1",
+            api_key=api_key
+        )
+
+def get_model_name(provider="nvidia"):
+    if provider == "nvidia":
+        return "meta/llama-4-maverick"
+    else:
+        return "mistral-large-latest" 

pageindex/__init__.py

@@ -0,0 +1,2 @@
+from .page_index import *
+from .page_index_md import md_to_tree

pageindex/config.yaml

@@ -0,0 +1,8 @@
+model: "gpt-4o-2024-11-20"
+toc_check_page_num: 20
+max_page_num_each_node: 10
+max_token_num_each_node: 20000
+if_add_node_id: "yes"
+if_add_node_summary: "yes"
+if_add_doc_description: "no"
+if_add_node_text: "no"

pageindex/core/tree_index.py

@@ -0,0 +1,226 @@
+import json
+import re
+import asyncio
+from typing import List, Dict, Any, Optional
+
+# Import potentially available util functions or re-implement if necessary
+# Assuming we are in pageindex/core/tree_index.py, and page_index_md is in pageindex/page_index_md.py
+try:
+    from ...pageindex.page_index_md import (
+        extract_nodes_from_markdown,
+        extract_node_text_content,
+        build_tree_from_nodes,
+        count_tokens  # Assuming this exists in utils or page_index_md imports
+    )
+except ImportError:
+    # If imports fail due to path issues, we might need to adjust python path or copy logic.
+    # For now, let's assume we can import from the sibling directory if the package structure is correct.
+    # Alternatively, we can re-implement the core logic here to be self-contained.
+    pass
+
+class TreeIndex:
+    def __init__(self, model: str = "gpt-4-turbo"):
+        self.model = model
+        self.tree = None
+
+    def build_from_markdown(self, markdown_text: str) -> str:
+        """
+        Builds the tree index from markdown text.
+        
+        Args:
+            markdown_text (str): The markdown content.
+            
+        Returns:
+            str: A document ID or confirmation (for this implementation, it returns 'doc_id').
+        """
+        print(f"Extracting nodes from markdown content...")
+        # Re-implementing core logic here to ensure stability and avoid complex relative imports for now
+        # This matches page_index_md.py logic
+        
+        # 1. Extract nodes
+        node_list, lines = self._extract_nodes(markdown_text)
+        
+        # 2. Extract text content
+        nodes_with_content = self._extract_node_text(node_list, lines)
+        
+        # 3. Build tree
+        self.tree = self._build_tree(nodes_with_content)
+        
+        # 4. Add Node IDs (simple counter)
+        self._add_node_ids(self.tree)
+
+        print(f"Tree built successfully.")
+        return "doc_1" # In a real app this would be a UUID
+
+    def _extract_nodes(self, markdown_content):
+        header_pattern = r'^(#{1,6})\s+(.+)$'
+        code_block_pattern = r'^```'
+        node_list = []
+        
+        lines = markdown_content.split('\n')
+        in_code_block = False
+        
+        for line_num, line in enumerate(lines, 1):
+            stripped_line = line.strip()
+            
+            if re.match(code_block_pattern, stripped_line):
+                in_code_block = not in_code_block
+                continue
+            
+            if not stripped_line:
+                continue
+            
+            if not in_code_block:
+                match = re.match(header_pattern, stripped_line)
+                if match:
+                    title = match.group(2).strip()
+                    node_list.append({'node_title': title, 'line_num': line_num})
+        
+        return node_list, lines
+
+    def _extract_node_text(self, node_list, lines):
+        all_nodes = []
+        for node in node_list:
+            line_content = lines[node['line_num'] - 1]
+            header_match = re.match(r'^(#{1,6})', line_content)
+            
+            if header_match:
+                all_nodes.append({
+                    'title': node['node_title'],
+                    'line_num': node['line_num'],
+                    'level': len(header_match.group(1))
+                })
+        
+        for i, node in enumerate(all_nodes):
+            start_line = node['line_num'] - 1 
+            if i + 1 < len(all_nodes):
+                end_line = all_nodes[i + 1]['line_num'] - 1 
+            else:
+                end_line = len(lines)
+            
+            node['text'] = '\n'.join(lines[start_line:end_line]).strip()    
+        return all_nodes
+
+    def _build_tree(self, node_list):
+        if not node_list:
+            return []
+        
+        stack = []
+        root_nodes = []
+        
+        for node in node_list:
+            current_level = node['level']
+            
+            tree_node = {
+                'title': node['title'],
+                'text': node['text'],
+                'line_num': node['line_num'],
+                'nodes': []
+            }
+            
+            while stack and stack[-1][1] >= current_level:
+                stack.pop()
+            
+            if not stack:
+                root_nodes.append(tree_node)
+            else:
+                parent_node, parent_level = stack[-1]
+                parent_node['nodes'].append(tree_node)
+            
+            stack.append((tree_node, current_level))
+        
+        return root_nodes
+
+    def _add_node_ids(self, nodes, prefix=""):
+        for i, node in enumerate(nodes):
+            node_id = f"{prefix}{i+1}"
+            node['node_id'] = node_id
+            if node.get('nodes'):
+                self._add_node_ids(node['nodes'], prefix=f"{node_id}.")
+
+    def reasoning_search(self, query: str, llm_client: Any) -> str:
+        """
+        Performs a tree search to find relevant nodes for the query.
+        
+        Args:
+            query (str): The user query.
+            llm_client: The initialized OpenAI client.
+            
+        Returns:
+            str: Context concatenated from relevant nodes.
+        """
+        if not self.tree:
+            return "Tree not index built. Please upload a document first."
+
+        # Simplify tree for prompt (remove raw text to save tokens, keep titles and IDs)
+        tree_summary = self._get_tree_structure_summary(self.tree)
+        
+        prompt = f"""
+        You are given a query and the tree structure of a document.
+        You need to find all nodes that are likely to contain the answer.
+
+        Query: {query}
+
+        Document tree structure: 
+        {json.dumps(tree_summary, indent=2)}
+
+        Reply in the following JSON format:
+        {{
+          "thinking": <your reasoning about which nodes are relevant>,
+          "node_list": ["node_id1", "node_id2", ...]
+        }}
+        """
+
+        try:
+            response = llm_client.chat.completions.create(
+                model="gpt-4-turbo", # Provide a default, though the client might override or ignored it for NIM
+                messages=[{"role": "user", "content": prompt}],
+                temperature=0.1
+            )
+            
+            content = response.choices[0].message.content
+            # Basic JSON extraction if LLM adds markdown
+            if "```json" in content:
+                content = content.split("```json")[1].split("```")[0].strip()
+            elif "```" in content:
+                content = content.split("```")[1].split("```")[0].strip()
+            
+            result = json.loads(content)
+            relevant_node_ids = result.get("node_list", [])
+            
+            # Retrieve text for these nodes
+            context_parts = []
+            for node_id in relevant_node_ids:
+                node = self._find_node_by_id(self.tree, node_id)
+                if node:
+                    context_parts.append(f"--- Section: {node['title']} (ID: {node_id}) ---\n{node['text']}\n")
+            
+            full_context = "\n".join(context_parts)
+            if not full_context:
+                return "No relevant context found in the document tree."
+            return full_context
+
+        except Exception as e:
+            return f"Error during reasoning search: {str(e)}"
+
+    def _get_tree_structure_summary(self, nodes):
+        summary = []
+        for node in nodes:
+            node_summary = {
+                'title': node['title'],
+                'node_id': node['node_id']
+            }
+            if node.get('nodes'):
+                node_summary['nodes'] = self._get_tree_structure_summary(node['nodes'])
+            summary.append(node_summary)
+        return summary
+
+    def _find_node_by_id(self, nodes, target_id):
+        for node in nodes:
+            if node.get('node_id') == target_id:
+                return node
+            if node.get('nodes'):
+                found = self._find_node_by_id(node['nodes'], target_id)
+                if found:
+                    return found
+        return None

pageindex/page_index.py

@@ -0,0 +1,1144 @@
+import os
+import json
+import copy
+import math
+import random
+import re
+from .utils import *
+import os
+from concurrent.futures import ThreadPoolExecutor, as_completed
+
+
+################### check title in page #########################################################
+async def check_title_appearance(item, page_list, start_index=1, model=None):    
+    title=item['title']
+    if 'physical_index' not in item or item['physical_index'] is None:
+        return {'list_index': item.get('list_index'), 'answer': 'no', 'title':title, 'page_number': None}
+    
+    
+    page_number = item['physical_index']
+    page_text = page_list[page_number-start_index][0]
+
+    
+    prompt = f"""
+    Your job is to check if the given section appears or starts in the given page_text.
+
+    Note: do fuzzy matching, ignore any space inconsistency in the page_text.
+
+    The given section title is {title}.
+    The given page_text is {page_text}.
+    
+    Reply format:
+    {{
+        
+        "thinking": <why do you think the section appears or starts in the page_text>
+        "answer": "yes or no" (yes if the section appears or starts in the page_text, no otherwise)
+    }}
+    Directly return the final JSON structure. Do not output anything else."""
+
+    response = await ChatGPT_API_async(model=model, prompt=prompt)
+    response = extract_json(response)
+    if 'answer' in response:
+        answer = response['answer']
+    else:
+        answer = 'no'
+    return {'list_index': item['list_index'], 'answer': answer, 'title': title, 'page_number': page_number}
+
+
+async def check_title_appearance_in_start(title, page_text, model=None, logger=None):    
+    prompt = f"""
+    You will be given the current section title and the current page_text.
+    Your job is to check if the current section starts in the beginning of the given page_text.
+    If there are other contents before the current section title, then the current section does not start in the beginning of the given page_text.
+    If the current section title is the first content in the given page_text, then the current section starts in the beginning of the given page_text.
+
+    Note: do fuzzy matching, ignore any space inconsistency in the page_text.
+
+    The given section title is {title}.
+    The given page_text is {page_text}.
+    
+    reply format:
+    {{
+        "thinking": <why do you think the section appears or starts in the page_text>
+        "start_begin": "yes or no" (yes if the section starts in the beginning of the page_text, no otherwise)
+    }}
+    Directly return the final JSON structure. Do not output anything else."""
+
+    response = await ChatGPT_API_async(model=model, prompt=prompt)
+    response = extract_json(response)
+    if logger:
+        logger.info(f"Response: {response}")
+    return response.get("start_begin", "no")
+
+
+async def check_title_appearance_in_start_concurrent(structure, page_list, model=None, logger=None):
+    if logger:
+        logger.info("Checking title appearance in start concurrently")
+    
+    # skip items without physical_index
+    for item in structure:
+        if item.get('physical_index') is None:
+            item['appear_start'] = 'no'
+
+    # only for items with valid physical_index
+    tasks = []
+    valid_items = []
+    for item in structure:
+        if item.get('physical_index') is not None:
+            page_text = page_list[item['physical_index'] - 1][0]
+            tasks.append(check_title_appearance_in_start(item['title'], page_text, model=model, logger=logger))
+            valid_items.append(item)
+
+    results = await asyncio.gather(*tasks, return_exceptions=True)
+    for item, result in zip(valid_items, results):
+        if isinstance(result, Exception):
+            if logger:
+                logger.error(f"Error checking start for {item['title']}: {result}")
+            item['appear_start'] = 'no'
+        else:
+            item['appear_start'] = result
+
+    return structure
+
+
+def toc_detector_single_page(content, model=None):
+    prompt = f"""
+    Your job is to detect if there is a table of content provided in the given text.
+
+    Given text: {content}
+
+    return the following JSON format:
+    {{
+        "thinking": <why do you think there is a table of content in the given text>
+        "toc_detected": "<yes or no>",
+    }}
+
+    Directly return the final JSON structure. Do not output anything else.
+    Please note: abstract,summary, notation list, figure list, table list, etc. are not table of contents."""
+
+    response = ChatGPT_API(model=model, prompt=prompt)
+    # print('response', response)
+    json_content = extract_json(response)    
+    return json_content['toc_detected']
+
+
+def check_if_toc_extraction_is_complete(content, toc, model=None):
+    prompt = f"""
+    You are given a partial document  and a  table of contents.
+    Your job is to check if the  table of contents is complete, which it contains all the main sections in the partial document.
+
+    Reply format:
+    {{
+        "thinking": <why do you think the table of contents is complete or not>
+        "completed": "yes" or "no"
+    }}
+    Directly return the final JSON structure. Do not output anything else."""
+
+    prompt = prompt + '\n Document:\n' + content + '\n Table of contents:\n' + toc
+    response = ChatGPT_API(model=model, prompt=prompt)
+    json_content = extract_json(response)
+    return json_content['completed']
+
+
+def check_if_toc_transformation_is_complete(content, toc, model=None):
+    prompt = f"""
+    You are given a raw table of contents and a  table of contents.
+    Your job is to check if the  table of contents is complete.
+
+    Reply format:
+    {{
+        "thinking": <why do you think the cleaned table of contents is complete or not>
+        "completed": "yes" or "no"
+    }}
+    Directly return the final JSON structure. Do not output anything else."""
+
+    prompt = prompt + '\n Raw Table of contents:\n' + content + '\n Cleaned Table of contents:\n' + toc
+    response = ChatGPT_API(model=model, prompt=prompt)
+    json_content = extract_json(response)
+    return json_content['completed']
+
+def extract_toc_content(content, model=None):
+    prompt = f"""
+    Your job is to extract the full table of contents from the given text, replace ... with :
+
+    Given text: {content}
+
+    Directly return the full table of contents content. Do not output anything else."""
+
+    response, finish_reason = ChatGPT_API_with_finish_reason(model=model, prompt=prompt)
+    
+    if_complete = check_if_toc_transformation_is_complete(content, response, model)
+    if if_complete == "yes" and finish_reason == "finished":
+        return response
+    
+    chat_history = [
+        {"role": "user", "content": prompt}, 
+        {"role": "assistant", "content": response},    
+    ]
+    prompt = f"""please continue the generation of table of contents , directly output the remaining part of the structure"""
+    new_response, finish_reason = ChatGPT_API_with_finish_reason(model=model, prompt=prompt, chat_history=chat_history)
+    response = response + new_response
+    if_complete = check_if_toc_transformation_is_complete(content, response, model)
+    
+    while not (if_complete == "yes" and finish_reason == "finished"):
+        chat_history = [
+            {"role": "user", "content": prompt}, 
+            {"role": "assistant", "content": response},    
+        ]
+        prompt = f"""please continue the generation of table of contents , directly output the remaining part of the structure"""
+        new_response, finish_reason = ChatGPT_API_with_finish_reason(model=model, prompt=prompt, chat_history=chat_history)
+        response = response + new_response
+        if_complete = check_if_toc_transformation_is_complete(content, response, model)
+        
+        # Optional: Add a maximum retry limit to prevent infinite loops
+        if len(chat_history) > 5:  # Arbitrary limit of 10 attempts
+            raise Exception('Failed to complete table of contents after maximum retries')
+    
+    return response
+
+def detect_page_index(toc_content, model=None):
+    print('start detect_page_index')
+    prompt = f"""
+    You will be given a table of contents.
+
+    Your job is to detect if there are page numbers/indices given within the table of contents.
+
+    Given text: {toc_content}
+
+    Reply format:
+    {{
+        "thinking": <why do you think there are page numbers/indices given within the table of contents>
+        "page_index_given_in_toc": "<yes or no>"
+    }}
+    Directly return the final JSON structure. Do not output anything else."""
+
+    response = ChatGPT_API(model=model, prompt=prompt)
+    json_content = extract_json(response)
+    return json_content['page_index_given_in_toc']
+
+def toc_extractor(page_list, toc_page_list, model):
+    def transform_dots_to_colon(text):
+        text = re.sub(r'\.{5,}', ': ', text)
+        # Handle dots separated by spaces
+        text = re.sub(r'(?:\. ){5,}\.?', ': ', text)
+        return text
+    
+    toc_content = ""
+    for page_index in toc_page_list:
+        toc_content += page_list[page_index][0]
+    toc_content = transform_dots_to_colon(toc_content)
+    has_page_index = detect_page_index(toc_content, model=model)
+    
+    return {
+        "toc_content": toc_content,
+        "page_index_given_in_toc": has_page_index
+    }
+
+
+
+
+def toc_index_extractor(toc, content, model=None):
+    print('start toc_index_extractor')
+    tob_extractor_prompt = """
+    You are given a table of contents in a json format and several pages of a document, your job is to add the physical_index to the table of contents in the json format.
+
+    The provided pages contains tags like <physical_index_X> and <physical_index_X> to indicate the physical location of the page X.
+
+    The structure variable is the numeric system which represents the index of the hierarchy section in the table of contents. For example, the first section has structure index 1, the first subsection has structure index 1.1, the second subsection has structure index 1.2, etc.
+
+    The response should be in the following JSON format: 
+    [
+        {
+            "structure": <structure index, "x.x.x" or None> (string),
+            "title": <title of the section>,
+            "physical_index": "<physical_index_X>" (keep the format)
+        },
+        ...
+    ]
+
+    Only add the physical_index to the sections that are in the provided pages.
+    If the section is not in the provided pages, do not add the physical_index to it.
+    Directly return the final JSON structure. Do not output anything else."""
+
+    prompt = tob_extractor_prompt + '\nTable of contents:\n' + str(toc) + '\nDocument pages:\n' + content
+    response = ChatGPT_API(model=model, prompt=prompt)
+    json_content = extract_json(response)    
+    return json_content
+
+
+
+def toc_transformer(toc_content, model=None):
+    print('start toc_transformer')
+    init_prompt = """
+    You are given a table of contents, You job is to transform the whole table of content into a JSON format included table_of_contents.
+
+    structure is the numeric system which represents the index of the hierarchy section in the table of contents. For example, the first section has structure index 1, the first subsection has structure index 1.1, the second subsection has structure index 1.2, etc.
+
+    The response should be in the following JSON format: 
+    {
+    table_of_contents: [
+        {
+            "structure": <structure index, "x.x.x" or None> (string),
+            "title": <title of the section>,
+            "page": <page number or None>,
+        },
+        ...
+        ],
+    }
+    You should transform the full table of contents in one go.
+    Directly return the final JSON structure, do not output anything else. """
+
+    prompt = init_prompt + '\n Given table of contents\n:' + toc_content
+    last_complete, finish_reason = ChatGPT_API_with_finish_reason(model=model, prompt=prompt)
+    if_complete = check_if_toc_transformation_is_complete(toc_content, last_complete, model)
+    if if_complete == "yes" and finish_reason == "finished":
+        last_complete = extract_json(last_complete)
+        cleaned_response=convert_page_to_int(last_complete['table_of_contents'])
+        return cleaned_response
+    
+    last_complete = get_json_content(last_complete)
+    while not (if_complete == "yes" and finish_reason == "finished"):
+        position = last_complete.rfind('}')
+        if position != -1:
+            last_complete = last_complete[:position+2]
+        prompt = f"""
+        Your task is to continue the table of contents json structure, directly output the remaining part of the json structure.
+        The response should be in the following JSON format: 
+
+        The raw table of contents json structure is:
+        {toc_content}
+
+        The incomplete transformed table of contents json structure is:
+        {last_complete}
+
+        Please continue the json structure, directly output the remaining part of the json structure."""
+
+        new_complete, finish_reason = ChatGPT_API_with_finish_reason(model=model, prompt=prompt)
+
+        if new_complete.startswith('```json'):
+            new_complete =  get_json_content(new_complete)
+            last_complete = last_complete+new_complete
+
+        if_complete = check_if_toc_transformation_is_complete(toc_content, last_complete, model)
+        
+
+    last_complete = json.loads(last_complete)
+
+    cleaned_response=convert_page_to_int(last_complete['table_of_contents'])
+    return cleaned_response
+    
+
+
+
+def find_toc_pages(start_page_index, page_list, opt, logger=None):
+    print('start find_toc_pages')
+    last_page_is_yes = False
+    toc_page_list = []
+    i = start_page_index
+    
+    while i < len(page_list):
+        # Only check beyond max_pages if we're still finding TOC pages
+        if i >= opt.toc_check_page_num and not last_page_is_yes:
+            break
+        detected_result = toc_detector_single_page(page_list[i][0],model=opt.model)
+        if detected_result == 'yes':
+            if logger:
+                logger.info(f'Page {i} has toc')
+            toc_page_list.append(i)
+            last_page_is_yes = True
+        elif detected_result == 'no' and last_page_is_yes:
+            if logger:
+                logger.info(f'Found the last page with toc: {i-1}')
+            break
+        i += 1
+    
+    if not toc_page_list and logger:
+        logger.info('No toc found')
+        
+    return toc_page_list
+
+def remove_page_number(data):
+    if isinstance(data, dict):
+        data.pop('page_number', None)  
+        for key in list(data.keys()):
+            if 'nodes' in key:
+                remove_page_number(data[key])
+    elif isinstance(data, list):
+        for item in data:
+            remove_page_number(item)
+    return data
+
+def extract_matching_page_pairs(toc_page, toc_physical_index, start_page_index):
+    pairs = []
+    for phy_item in toc_physical_index:
+        for page_item in toc_page:
+            if phy_item.get('title') == page_item.get('title'):
+                physical_index = phy_item.get('physical_index')
+                if physical_index is not None and int(physical_index) >= start_page_index:
+                    pairs.append({
+                        'title': phy_item.get('title'),
+                        'page': page_item.get('page'),
+                        'physical_index': physical_index
+                    })
+    return pairs
+
+
+def calculate_page_offset(pairs):
+    differences = []
+    for pair in pairs:
+        try:
+            physical_index = pair['physical_index']
+            page_number = pair['page']
+            difference = physical_index - page_number
+            differences.append(difference)
+        except (KeyError, TypeError):
+            continue
+    
+    if not differences:
+        return None
+    
+    difference_counts = {}
+    for diff in differences:
+        difference_counts[diff] = difference_counts.get(diff, 0) + 1
+    
+    most_common = max(difference_counts.items(), key=lambda x: x[1])[0]
+    
+    return most_common
+
+def add_page_offset_to_toc_json(data, offset):
+    for i in range(len(data)):
+        if data[i].get('page') is not None and isinstance(data[i]['page'], int):
+            data[i]['physical_index'] = data[i]['page'] + offset
+            del data[i]['page']
+    
+    return data
+
+
+
+def page_list_to_group_text(page_contents, token_lengths, max_tokens=20000, overlap_page=1):    
+    num_tokens = sum(token_lengths)
+    
+    if num_tokens <= max_tokens:
+        # merge all pages into one text
+        page_text = "".join(page_contents)
+        return [page_text]
+    
+    subsets = []
+    current_subset = []
+    current_token_count = 0
+
+    expected_parts_num = math.ceil(num_tokens / max_tokens)
+    average_tokens_per_part = math.ceil(((num_tokens / expected_parts_num) + max_tokens) / 2)
+    
+    for i, (page_content, page_tokens) in enumerate(zip(page_contents, token_lengths)):
+        if current_token_count + page_tokens > average_tokens_per_part:
+
+            subsets.append(''.join(current_subset))
+            # Start new subset from overlap if specified
+            overlap_start = max(i - overlap_page, 0)
+            current_subset = page_contents[overlap_start:i]
+            current_token_count = sum(token_lengths[overlap_start:i])
+        
+        # Add current page to the subset
+        current_subset.append(page_content)
+        current_token_count += page_tokens
+
+    # Add the last subset if it contains any pages
+    if current_subset:
+        subsets.append(''.join(current_subset))
+    
+    print('divide page_list to groups', len(subsets))
+    return subsets
+
+def add_page_number_to_toc(part, structure, model=None):
+    fill_prompt_seq = """
+    You are given an JSON structure of a document and a partial part of the document. Your task is to check if the title that is described in the structure is started in the partial given document.
+
+    The provided text contains tags like <physical_index_X> and <physical_index_X> to indicate the physical location of the page X. 
+
+    If the full target section starts in the partial given document, insert the given JSON structure with the "start": "yes", and "start_index": "<physical_index_X>".
+
+    If the full target section does not start in the partial given document, insert "start": "no",  "start_index": None.
+
+    The response should be in the following format. 
+        [
+            {
+                "structure": <structure index, "x.x.x" or None> (string),
+                "title": <title of the section>,
+                "start": "<yes or no>",
+                "physical_index": "<physical_index_X> (keep the format)" or None
+            },
+            ...
+        ]    
+    The given structure contains the result of the previous part, you need to fill the result of the current part, do not change the previous result.
+    Directly return the final JSON structure. Do not output anything else."""
+
+    prompt = fill_prompt_seq + f"\n\nCurrent Partial Document:\n{part}\n\nGiven Structure\n{json.dumps(structure, indent=2)}\n"
+    current_json_raw = ChatGPT_API(model=model, prompt=prompt)
+    json_result = extract_json(current_json_raw)
+    
+    for item in json_result:
+        if 'start' in item:
+            del item['start']
+    return json_result
+
+
+def remove_first_physical_index_section(text):
+    """
+    Removes the first section between <physical_index_X> and <physical_index_X> tags,
+    and returns the remaining text.
+    """
+    pattern = r'<physical_index_\d+>.*?<physical_index_\d+>'
+    match = re.search(pattern, text, re.DOTALL)
+    if match:
+        # Remove the first matched section
+        return text.replace(match.group(0), '', 1)
+    return text
+
+### add verify completeness
+def generate_toc_continue(toc_content, part, model="gpt-4o-2024-11-20"):
+    print('start generate_toc_continue')
+    prompt = """
+    You are an expert in extracting hierarchical tree structure.
+    You are given a tree structure of the previous part and the text of the current part.
+    Your task is to continue the tree structure from the previous part to include the current part.
+
+    The structure variable is the numeric system which represents the index of the hierarchy section in the table of contents. For example, the first section has structure index 1, the first subsection has structure index 1.1, the second subsection has structure index 1.2, etc.
+
+    For the title, you need to extract the original title from the text, only fix the space inconsistency.
+
+    The provided text contains tags like <physical_index_X> and <physical_index_X> to indicate the start and end of page X. \
+    
+    For the physical_index, you need to extract the physical index of the start of the section from the text. Keep the <physical_index_X> format.
+
+    The response should be in the following format. 
+        [
+            {
+                "structure": <structure index, "x.x.x"> (string),
+                "title": <title of the section, keep the original title>,
+                "physical_index": "<physical_index_X> (keep the format)"
+            },
+            ...
+        ]    
+
+    Directly return the additional part of the final JSON structure. Do not output anything else."""
+
+    prompt = prompt + '\nGiven text\n:' + part + '\nPrevious tree structure\n:' + json.dumps(toc_content, indent=2)
+    response, finish_reason = ChatGPT_API_with_finish_reason(model=model, prompt=prompt)
+    if finish_reason == 'finished':
+        return extract_json(response)
+    else:
+        raise Exception(f'finish reason: {finish_reason}')
+    
+### add verify completeness
+def generate_toc_init(part, model=None):
+    print('start generate_toc_init')
+    prompt = """
+    You are an expert in extracting hierarchical tree structure, your task is to generate the tree structure of the document.
+
+    The structure variable is the numeric system which represents the index of the hierarchy section in the table of contents. For example, the first section has structure index 1, the first subsection has structure index 1.1, the second subsection has structure index 1.2, etc.
+
+    For the title, you need to extract the original title from the text, only fix the space inconsistency.
+
+    The provided text contains tags like <physical_index_X> and <physical_index_X> to indicate the start and end of page X. 
+
+    For the physical_index, you need to extract the physical index of the start of the section from the text. Keep the <physical_index_X> format.
+
+    The response should be in the following format. 
+        [
+            {{
+                "structure": <structure index, "x.x.x"> (string),
+                "title": <title of the section, keep the original title>,
+                "physical_index": "<physical_index_X> (keep the format)"
+            }},
+            
+        ],
+
+
+    Directly return the final JSON structure. Do not output anything else."""
+
+    prompt = prompt + '\nGiven text\n:' + part
+    response, finish_reason = ChatGPT_API_with_finish_reason(model=model, prompt=prompt)
+
+    if finish_reason == 'finished':
+         return extract_json(response)
+    else:
+        raise Exception(f'finish reason: {finish_reason}')
+
+def process_no_toc(page_list, start_index=1, model=None, logger=None):
+    page_contents=[]
+    token_lengths=[]
+    for page_index in range(start_index, start_index+len(page_list)):
+        page_text = f"<physical_index_{page_index}>\n{page_list[page_index-start_index][0]}\n<physical_index_{page_index}>\n\n"
+        page_contents.append(page_text)
+        token_lengths.append(count_tokens(page_text, model))
+    group_texts = page_list_to_group_text(page_contents, token_lengths)
+    logger.info(f'len(group_texts): {len(group_texts)}')
+
+    toc_with_page_number= generate_toc_init(group_texts[0], model)
+    for group_text in group_texts[1:]:
+        toc_with_page_number_additional = generate_toc_continue(toc_with_page_number, group_text, model)    
+        toc_with_page_number.extend(toc_with_page_number_additional)
+    logger.info(f'generate_toc: {toc_with_page_number}')
+
+    toc_with_page_number = convert_physical_index_to_int(toc_with_page_number)
+    logger.info(f'convert_physical_index_to_int: {toc_with_page_number}')
+
+    return toc_with_page_number
+
+def process_toc_no_page_numbers(toc_content, toc_page_list, page_list,  start_index=1, model=None, logger=None):
+    page_contents=[]
+    token_lengths=[]
+    toc_content = toc_transformer(toc_content, model)
+    logger.info(f'toc_transformer: {toc_content}')
+    for page_index in range(start_index, start_index+len(page_list)):
+        page_text = f"<physical_index_{page_index}>\n{page_list[page_index-start_index][0]}\n<physical_index_{page_index}>\n\n"
+        page_contents.append(page_text)
+        token_lengths.append(count_tokens(page_text, model))
+    
+    group_texts = page_list_to_group_text(page_contents, token_lengths)
+    logger.info(f'len(group_texts): {len(group_texts)}')
+
+    toc_with_page_number=copy.deepcopy(toc_content)
+    for group_text in group_texts:
+        toc_with_page_number = add_page_number_to_toc(group_text, toc_with_page_number, model)
+    logger.info(f'add_page_number_to_toc: {toc_with_page_number}')
+
+    toc_with_page_number = convert_physical_index_to_int(toc_with_page_number)
+    logger.info(f'convert_physical_index_to_int: {toc_with_page_number}')
+
+    return toc_with_page_number
+
+
+
+def process_toc_with_page_numbers(toc_content, toc_page_list, page_list, toc_check_page_num=None, model=None, logger=None):
+    toc_with_page_number = toc_transformer(toc_content, model)
+    logger.info(f'toc_with_page_number: {toc_with_page_number}')
+
+    toc_no_page_number = remove_page_number(copy.deepcopy(toc_with_page_number))
+    
+    start_page_index = toc_page_list[-1] + 1
+    main_content = ""
+    for page_index in range(start_page_index, min(start_page_index + toc_check_page_num, len(page_list))):
+        main_content += f"<physical_index_{page_index+1}>\n{page_list[page_index][0]}\n<physical_index_{page_index+1}>\n\n"
+
+    toc_with_physical_index = toc_index_extractor(toc_no_page_number, main_content, model)
+    logger.info(f'toc_with_physical_index: {toc_with_physical_index}')
+
+    toc_with_physical_index = convert_physical_index_to_int(toc_with_physical_index)
+    logger.info(f'toc_with_physical_index: {toc_with_physical_index}')
+
+    matching_pairs = extract_matching_page_pairs(toc_with_page_number, toc_with_physical_index, start_page_index)
+    logger.info(f'matching_pairs: {matching_pairs}')
+
+    offset = calculate_page_offset(matching_pairs)
+    logger.info(f'offset: {offset}')
+
+    toc_with_page_number = add_page_offset_to_toc_json(toc_with_page_number, offset)
+    logger.info(f'toc_with_page_number: {toc_with_page_number}')
+
+    toc_with_page_number = process_none_page_numbers(toc_with_page_number, page_list, model=model)
+    logger.info(f'toc_with_page_number: {toc_with_page_number}')
+
+    return toc_with_page_number
+
+
+
+##check if needed to process none page numbers
+def process_none_page_numbers(toc_items, page_list, start_index=1, model=None):
+    for i, item in enumerate(toc_items):
+        if "physical_index" not in item:
+            # logger.info(f"fix item: {item}")
+            # Find previous physical_index
+            prev_physical_index = 0  # Default if no previous item exists
+            for j in range(i - 1, -1, -1):
+                if toc_items[j].get('physical_index') is not None:
+                    prev_physical_index = toc_items[j]['physical_index']
+                    break
+            
+            # Find next physical_index
+            next_physical_index = -1  # Default if no next item exists
+            for j in range(i + 1, len(toc_items)):
+                if toc_items[j].get('physical_index') is not None:
+                    next_physical_index = toc_items[j]['physical_index']
+                    break
+
+            page_contents = []
+            for page_index in range(prev_physical_index, next_physical_index+1):
+                # Add bounds checking to prevent IndexError
+                list_index = page_index - start_index
+                if list_index >= 0 and list_index < len(page_list):
+                    page_text = f"<physical_index_{page_index}>\n{page_list[list_index][0]}\n<physical_index_{page_index}>\n\n"
+                    page_contents.append(page_text)
+                else:
+                    continue
+
+            item_copy = copy.deepcopy(item)
+            del item_copy['page']
+            result = add_page_number_to_toc(page_contents, item_copy, model)
+            if isinstance(result[0]['physical_index'], str) and result[0]['physical_index'].startswith('<physical_index'):
+                item['physical_index'] = int(result[0]['physical_index'].split('_')[-1].rstrip('>').strip())
+                del item['page']
+    
+    return toc_items
+
+
+
+
+def check_toc(page_list, opt=None):
+    toc_page_list = find_toc_pages(start_page_index=0, page_list=page_list, opt=opt)
+    if len(toc_page_list) == 0:
+        print('no toc found')
+        return {'toc_content': None, 'toc_page_list': [], 'page_index_given_in_toc': 'no'}
+    else:
+        print('toc found')
+        toc_json = toc_extractor(page_list, toc_page_list, opt.model)
+
+        if toc_json['page_index_given_in_toc'] == 'yes':
+            print('index found')
+            return {'toc_content': toc_json['toc_content'], 'toc_page_list': toc_page_list, 'page_index_given_in_toc': 'yes'}
+        else:
+            current_start_index = toc_page_list[-1] + 1
+            
+            while (toc_json['page_index_given_in_toc'] == 'no' and 
+                   current_start_index < len(page_list) and 
+                   current_start_index < opt.toc_check_page_num):
+                
+                additional_toc_pages = find_toc_pages(
+                    start_page_index=current_start_index,
+                    page_list=page_list,
+                    opt=opt
+                )
+                
+                if len(additional_toc_pages) == 0:
+                    break
+
+                additional_toc_json = toc_extractor(page_list, additional_toc_pages, opt.model)
+                if additional_toc_json['page_index_given_in_toc'] == 'yes':
+                    print('index found')
+                    return {'toc_content': additional_toc_json['toc_content'], 'toc_page_list': additional_toc_pages, 'page_index_given_in_toc': 'yes'}
+
+                else:
+                    current_start_index = additional_toc_pages[-1] + 1
+            print('index not found')
+            return {'toc_content': toc_json['toc_content'], 'toc_page_list': toc_page_list, 'page_index_given_in_toc': 'no'}
+
+
+
+
+
+
+################### fix incorrect toc #########################################################
+def single_toc_item_index_fixer(section_title, content, model="gpt-4o-2024-11-20"):
+    tob_extractor_prompt = """
+    You are given a section title and several pages of a document, your job is to find the physical index of the start page of the section in the partial document.
+
+    The provided pages contains tags like <physical_index_X> and <physical_index_X> to indicate the physical location of the page X.
+
+    Reply in a JSON format:
+    {
+        "thinking": <explain which page, started and closed by <physical_index_X>, contains the start of this section>,
+        "physical_index": "<physical_index_X>" (keep the format)
+    }
+    Directly return the final JSON structure. Do not output anything else."""
+
+    prompt = tob_extractor_prompt + '\nSection Title:\n' + str(section_title) + '\nDocument pages:\n' + content
+    response = ChatGPT_API(model=model, prompt=prompt)
+    json_content = extract_json(response)    
+    return convert_physical_index_to_int(json_content['physical_index'])
+
+
+
+async def fix_incorrect_toc(toc_with_page_number, page_list, incorrect_results, start_index=1, model=None, logger=None):
+    print(f'start fix_incorrect_toc with {len(incorrect_results)} incorrect results')
+    incorrect_indices = {result['list_index'] for result in incorrect_results}
+    
+    end_index = len(page_list) + start_index - 1
+    
+    incorrect_results_and_range_logs = []
+    # Helper function to process and check a single incorrect item
+    async def process_and_check_item(incorrect_item):
+        list_index = incorrect_item['list_index']
+        
+        # Check if list_index is valid
+        if list_index < 0 or list_index >= len(toc_with_page_number):
+            # Return an invalid result for out-of-bounds indices
+            return {
+                'list_index': list_index,
+                'title': incorrect_item['title'],
+                'physical_index': incorrect_item.get('physical_index'),
+                'is_valid': False
+            }
+        
+        # Find the previous correct item
+        prev_correct = None
+        for i in range(list_index-1, -1, -1):
+            if i not in incorrect_indices and i >= 0 and i < len(toc_with_page_number):
+                physical_index = toc_with_page_number[i].get('physical_index')
+                if physical_index is not None:
+                    prev_correct = physical_index
+                    break
+        # If no previous correct item found, use start_index
+        if prev_correct is None:
+            prev_correct = start_index - 1
+        
+        # Find the next correct item
+        next_correct = None
+        for i in range(list_index+1, len(toc_with_page_number)):
+            if i not in incorrect_indices and i >= 0 and i < len(toc_with_page_number):
+                physical_index = toc_with_page_number[i].get('physical_index')
+                if physical_index is not None:
+                    next_correct = physical_index
+                    break
+        # If no next correct item found, use end_index
+        if next_correct is None:
+            next_correct = end_index
+        
+        incorrect_results_and_range_logs.append({
+            'list_index': list_index,
+            'title': incorrect_item['title'],
+            'prev_correct': prev_correct,
+            'next_correct': next_correct
+        })
+
+        page_contents=[]
+        for page_index in range(prev_correct, next_correct+1):
+            # Add bounds checking to prevent IndexError
+            list_index = page_index - start_index
+            if list_index >= 0 and list_index < len(page_list):
+                page_text = f"<physical_index_{page_index}>\n{page_list[list_index][0]}\n<physical_index_{page_index}>\n\n"
+                page_contents.append(page_text)
+            else:
+                continue
+        content_range = ''.join(page_contents)
+        
+        physical_index_int = single_toc_item_index_fixer(incorrect_item['title'], content_range, model)
+        
+        # Check if the result is correct
+        check_item = incorrect_item.copy()
+        check_item['physical_index'] = physical_index_int
+        check_result = await check_title_appearance(check_item, page_list, start_index, model)
+
+        return {
+            'list_index': list_index,
+            'title': incorrect_item['title'],
+            'physical_index': physical_index_int,
+            'is_valid': check_result['answer'] == 'yes'
+        }
+
+    # Process incorrect items concurrently
+    tasks = [
+        process_and_check_item(item)
+        for item in incorrect_results
+    ]
+    results = await asyncio.gather(*tasks, return_exceptions=True)
+    for item, result in zip(incorrect_results, results):
+        if isinstance(result, Exception):
+            print(f"Processing item {item} generated an exception: {result}")
+            continue
+    results = [result for result in results if not isinstance(result, Exception)]
+
+    # Update the toc_with_page_number with the fixed indices and check for any invalid results
+    invalid_results = []
+    for result in results:
+        if result['is_valid']:
+            # Add bounds checking to prevent IndexError
+            list_idx = result['list_index']
+            if 0 <= list_idx < len(toc_with_page_number):
+                toc_with_page_number[list_idx]['physical_index'] = result['physical_index']
+            else:
+                # Index is out of bounds, treat as invalid
+                invalid_results.append({
+                    'list_index': result['list_index'],
+                    'title': result['title'],
+                    'physical_index': result['physical_index'],
+                })
+        else:
+            invalid_results.append({
+                'list_index': result['list_index'],
+                'title': result['title'],
+                'physical_index': result['physical_index'],
+            })
+
+    logger.info(f'incorrect_results_and_range_logs: {incorrect_results_and_range_logs}')
+    logger.info(f'invalid_results: {invalid_results}')
+
+    return toc_with_page_number, invalid_results
+
+
+
+async def fix_incorrect_toc_with_retries(toc_with_page_number, page_list, incorrect_results, start_index=1, max_attempts=3, model=None, logger=None):
+    print('start fix_incorrect_toc')
+    fix_attempt = 0
+    current_toc = toc_with_page_number
+    current_incorrect = incorrect_results
+
+    while current_incorrect:
+        print(f"Fixing {len(current_incorrect)} incorrect results")
+        
+        current_toc, current_incorrect = await fix_incorrect_toc(current_toc, page_list, current_incorrect, start_index, model, logger)
+                
+        fix_attempt += 1
+        if fix_attempt >= max_attempts:
+            logger.info("Maximum fix attempts reached")
+            break
+    
+    return current_toc, current_incorrect
+
+
+
+
+################### verify toc #########################################################
+async def verify_toc(page_list, list_result, start_index=1, N=None, model=None):
+    print('start verify_toc')
+    # Find the last non-None physical_index
+    last_physical_index = None
+    for item in reversed(list_result):
+        if item.get('physical_index') is not None:
+            last_physical_index = item['physical_index']
+            break
+    
+    # Early return if we don't have valid physical indices
+    if last_physical_index is None or last_physical_index < len(page_list)/2:
+        return 0, []
+    
+    # Determine which items to check
+    if N is None:
+        print('check all items')
+        sample_indices = range(0, len(list_result))
+    else:
+        N = min(N, len(list_result))
+        print(f'check {N} items')
+        sample_indices = random.sample(range(0, len(list_result)), N)
+
+    # Prepare items with their list indices
+    indexed_sample_list = []
+    for idx in sample_indices:
+        item = list_result[idx]
+        # Skip items with None physical_index (these were invalidated by validate_and_truncate_physical_indices)
+        if item.get('physical_index') is not None:
+            item_with_index = item.copy()
+            item_with_index['list_index'] = idx  # Add the original index in list_result
+            indexed_sample_list.append(item_with_index)
+
+    # Run checks concurrently
+    tasks = [
+        check_title_appearance(item, page_list, start_index, model)
+        for item in indexed_sample_list
+    ]
+    results = await asyncio.gather(*tasks)
+    
+    # Process results
+    correct_count = 0
+    incorrect_results = []
+    for result in results:
+        if result['answer'] == 'yes':
+            correct_count += 1
+        else:
+            incorrect_results.append(result)
+    
+    # Calculate accuracy
+    checked_count = len(results)
+    accuracy = correct_count / checked_count if checked_count > 0 else 0
+    print(f"accuracy: {accuracy*100:.2f}%")
+    return accuracy, incorrect_results
+
+
+
+
+
+################### main process #########################################################
+async def meta_processor(page_list, mode=None, toc_content=None, toc_page_list=None, start_index=1, opt=None, logger=None):
+    print(mode)
+    print(f'start_index: {start_index}')
+    
+    if mode == 'process_toc_with_page_numbers':
+        toc_with_page_number = process_toc_with_page_numbers(toc_content, toc_page_list, page_list, toc_check_page_num=opt.toc_check_page_num, model=opt.model, logger=logger)
+    elif mode == 'process_toc_no_page_numbers':
+        toc_with_page_number = process_toc_no_page_numbers(toc_content, toc_page_list, page_list, model=opt.model, logger=logger)
+    else:
+        toc_with_page_number = process_no_toc(page_list, start_index=start_index, model=opt.model, logger=logger)
+            
+    toc_with_page_number = [item for item in toc_with_page_number if item.get('physical_index') is not None] 
+    
+    toc_with_page_number = validate_and_truncate_physical_indices(
+        toc_with_page_number, 
+        len(page_list), 
+        start_index=start_index, 
+        logger=logger
+    )
+    
+    accuracy, incorrect_results = await verify_toc(page_list, toc_with_page_number, start_index=start_index, model=opt.model)
+        
+    logger.info({
+        'mode': 'process_toc_with_page_numbers',
+        'accuracy': accuracy,
+        'incorrect_results': incorrect_results
+    })
+    if accuracy == 1.0 and len(incorrect_results) == 0:
+        return toc_with_page_number
+    if accuracy > 0.6 and len(incorrect_results) > 0:
+        toc_with_page_number, incorrect_results = await fix_incorrect_toc_with_retries(toc_with_page_number, page_list, incorrect_results,start_index=start_index, max_attempts=3, model=opt.model, logger=logger)
+        return toc_with_page_number
+    else:
+        if mode == 'process_toc_with_page_numbers':
+            return await meta_processor(page_list, mode='process_toc_no_page_numbers', toc_content=toc_content, toc_page_list=toc_page_list, start_index=start_index, opt=opt, logger=logger)
+        elif mode == 'process_toc_no_page_numbers':
+            return await meta_processor(page_list, mode='process_no_toc', start_index=start_index, opt=opt, logger=logger)
+        else:
+            raise Exception('Processing failed')
+        
+ 
+async def process_large_node_recursively(node, page_list, opt=None, logger=None):
+    node_page_list = page_list[node['start_index']-1:node['end_index']]
+    token_num = sum([page[1] for page in node_page_list])
+    
+    if node['end_index'] - node['start_index'] > opt.max_page_num_each_node and token_num >= opt.max_token_num_each_node:
+        print('large node:', node['title'], 'start_index:', node['start_index'], 'end_index:', node['end_index'], 'token_num:', token_num)
+
+        node_toc_tree = await meta_processor(node_page_list, mode='process_no_toc', start_index=node['start_index'], opt=opt, logger=logger)
+        node_toc_tree = await check_title_appearance_in_start_concurrent(node_toc_tree, page_list, model=opt.model, logger=logger)
+        
+        # Filter out items with None physical_index before post_processing
+        valid_node_toc_items = [item for item in node_toc_tree if item.get('physical_index') is not None]
+        
+        if valid_node_toc_items and node['title'].strip() == valid_node_toc_items[0]['title'].strip():
+            node['nodes'] = post_processing(valid_node_toc_items[1:], node['end_index'])
+            node['end_index'] = valid_node_toc_items[1]['start_index'] if len(valid_node_toc_items) > 1 else node['end_index']
+        else:
+            node['nodes'] = post_processing(valid_node_toc_items, node['end_index'])
+            node['end_index'] = valid_node_toc_items[0]['start_index'] if valid_node_toc_items else node['end_index']
+        
+    if 'nodes' in node and node['nodes']:
+        tasks = [
+            process_large_node_recursively(child_node, page_list, opt, logger=logger)
+            for child_node in node['nodes']
+        ]
+        await asyncio.gather(*tasks)
+    
+    return node
+
+async def tree_parser(page_list, opt, doc=None, logger=None):
+    check_toc_result = check_toc(page_list, opt)
+    logger.info(check_toc_result)
+
+    if check_toc_result.get("toc_content") and check_toc_result["toc_content"].strip() and check_toc_result["page_index_given_in_toc"] == "yes":
+        toc_with_page_number = await meta_processor(
+            page_list, 
+            mode='process_toc_with_page_numbers', 
+            start_index=1, 
+            toc_content=check_toc_result['toc_content'], 
+            toc_page_list=check_toc_result['toc_page_list'], 
+            opt=opt,
+            logger=logger)
+    else:
+        toc_with_page_number = await meta_processor(
+            page_list, 
+            mode='process_no_toc', 
+            start_index=1, 
+            opt=opt,
+            logger=logger)
+
+    toc_with_page_number = add_preface_if_needed(toc_with_page_number)
+    toc_with_page_number = await check_title_appearance_in_start_concurrent(toc_with_page_number, page_list, model=opt.model, logger=logger)
+    
+    # Filter out items with None physical_index before post_processings
+    valid_toc_items = [item for item in toc_with_page_number if item.get('physical_index') is not None]
+    
+    toc_tree = post_processing(valid_toc_items, len(page_list))
+    tasks = [
+        process_large_node_recursively(node, page_list, opt, logger=logger)
+        for node in toc_tree
+    ]
+    await asyncio.gather(*tasks)
+    
+    return toc_tree
+
+
+def page_index_main(doc, opt=None):
+    logger = JsonLogger(doc)
+    
+    is_valid_pdf = (
+        (isinstance(doc, str) and os.path.isfile(doc) and doc.lower().endswith(".pdf")) or 
+        isinstance(doc, BytesIO)
+    )
+    if not is_valid_pdf:
+        raise ValueError("Unsupported input type. Expected a PDF file path or BytesIO object.")
+
+    print('Parsing PDF...')
+    page_list = get_page_tokens(doc)
+
+    logger.info({'total_page_number': len(page_list)})
+    logger.info({'total_token': sum([page[1] for page in page_list])})
+
+    async def page_index_builder():
+        structure = await tree_parser(page_list, opt, doc=doc, logger=logger)
+        if opt.if_add_node_id == 'yes':
+            write_node_id(structure)    
+        if opt.if_add_node_text == 'yes':
+            add_node_text(structure, page_list)
+        if opt.if_add_node_summary == 'yes':
+            if opt.if_add_node_text == 'no':
+                add_node_text(structure, page_list)
+            await generate_summaries_for_structure(structure, model=opt.model)
+            if opt.if_add_node_text == 'no':
+                remove_structure_text(structure)
+            if opt.if_add_doc_description == 'yes':
+                # Create a clean structure without unnecessary fields for description generation
+                clean_structure = create_clean_structure_for_description(structure)
+                doc_description = generate_doc_description(clean_structure, model=opt.model)
+                return {
+                    'doc_name': get_pdf_name(doc),
+                    'doc_description': doc_description,
+                    'structure': structure,
+                }
+        return {
+            'doc_name': get_pdf_name(doc),
+            'structure': structure,
+        }
+
+    return asyncio.run(page_index_builder())
+
+
+def page_index(doc, model=None, toc_check_page_num=None, max_page_num_each_node=None, max_token_num_each_node=None,
+               if_add_node_id=None, if_add_node_summary=None, if_add_doc_description=None, if_add_node_text=None):
+    
+    user_opt = {
+        arg: value for arg, value in locals().items()
+        if arg != "doc" and value is not None
+    }
+    opt = ConfigLoader().load(user_opt)
+    return page_index_main(doc, opt)
+
+
+def validate_and_truncate_physical_indices(toc_with_page_number, page_list_length, start_index=1, logger=None):
+    """
+    Validates and truncates physical indices that exceed the actual document length.
+    This prevents errors when TOC references pages that don't exist in the document (e.g. the file is broken or incomplete).
+    """
+    if not toc_with_page_number:
+        return toc_with_page_number
+    
+    max_allowed_page = page_list_length + start_index - 1
+    truncated_items = []
+    
+    for i, item in enumerate(toc_with_page_number):
+        if item.get('physical_index') is not None:
+            original_index = item['physical_index']
+            if original_index > max_allowed_page:
+                item['physical_index'] = None
+                truncated_items.append({
+                    'title': item.get('title', 'Unknown'),
+                    'original_index': original_index
+                })
+                if logger:
+                    logger.info(f"Removed physical_index for '{item.get('title', 'Unknown')}' (was {original_index}, too far beyond document)")
+    
+    if truncated_items and logger:
+        logger.info(f"Total removed items: {len(truncated_items)}")
+        
+    print(f"Document validation: {page_list_length} pages, max allowed index: {max_allowed_page}")
+    if truncated_items:
+        print(f"Truncated {len(truncated_items)} TOC items that exceeded document length")
+     
+    return toc_with_page_number

pageindex/page_index_md.py

@@ -0,0 +1,339 @@
+import asyncio
+import json
+import re
+import os
+try:
+    from .utils import *
+except:
+    from utils import *
+
+async def get_node_summary(node, summary_token_threshold=200, model=None):
+    node_text = node.get('text')
+    num_tokens = count_tokens(node_text, model=model)
+    if num_tokens < summary_token_threshold:
+        return node_text
+    else:
+        return await generate_node_summary(node, model=model)
+
+
+async def generate_summaries_for_structure_md(structure, summary_token_threshold, model=None):
+    nodes = structure_to_list(structure)
+    tasks = [get_node_summary(node, summary_token_threshold=summary_token_threshold, model=model) for node in nodes]
+    summaries = await asyncio.gather(*tasks)
+    
+    for node, summary in zip(nodes, summaries):
+        if not node.get('nodes'):
+            node['summary'] = summary
+        else:
+            node['prefix_summary'] = summary
+    return structure
+
+
+def extract_nodes_from_markdown(markdown_content):
+    header_pattern = r'^(#{1,6})\s+(.+)$'
+    code_block_pattern = r'^```'
+    node_list = []
+    
+    lines = markdown_content.split('\n')
+    in_code_block = False
+    
+    for line_num, line in enumerate(lines, 1):
+        stripped_line = line.strip()
+        
+        # Check for code block delimiters (triple backticks)
+        if re.match(code_block_pattern, stripped_line):
+            in_code_block = not in_code_block
+            continue
+        
+        # Skip empty lines
+        if not stripped_line:
+            continue
+        
+        # Only look for headers when not inside a code block
+        if not in_code_block:
+            match = re.match(header_pattern, stripped_line)
+            if match:
+                title = match.group(2).strip()
+                node_list.append({'node_title': title, 'line_num': line_num})
+
+    return node_list, lines
+
+
+def extract_node_text_content(node_list, markdown_lines):    
+    all_nodes = []
+    for node in node_list:
+        line_content = markdown_lines[node['line_num'] - 1]
+        header_match = re.match(r'^(#{1,6})', line_content)
+        
+        if header_match is None:
+            print(f"Warning: Line {node['line_num']} does not contain a valid header: '{line_content}'")
+            continue
+            
+        processed_node = {
+            'title': node['node_title'],
+            'line_num': node['line_num'],
+            'level': len(header_match.group(1))
+        }
+        all_nodes.append(processed_node)
+    
+    for i, node in enumerate(all_nodes):
+        start_line = node['line_num'] - 1 
+        if i + 1 < len(all_nodes):
+            end_line = all_nodes[i + 1]['line_num'] - 1 
+        else:
+            end_line = len(markdown_lines)
+        
+        node['text'] = '\n'.join(markdown_lines[start_line:end_line]).strip()    
+    return all_nodes
+
+def update_node_list_with_text_token_count(node_list, model=None):
+
+    def find_all_children(parent_index, parent_level, node_list):
+        """Find all direct and indirect children of a parent node"""
+        children_indices = []
+        
+        # Look for children after the parent
+        for i in range(parent_index + 1, len(node_list)):
+            current_level = node_list[i]['level']
+            
+            # If we hit a node at same or higher level than parent, stop
+            if current_level <= parent_level:
+                break
+                
+            # This is a descendant
+            children_indices.append(i)
+        
+        return children_indices
+    
+    # Make a copy to avoid modifying the original
+    result_list = node_list.copy()
+    
+    # Process nodes from end to beginning to ensure children are processed before parents
+    for i in range(len(result_list) - 1, -1, -1):
+        current_node = result_list[i]
+        current_level = current_node['level']
+        
+        # Get all children of this node
+        children_indices = find_all_children(i, current_level, result_list)
+        
+        # Start with the node's own text
+        node_text = current_node.get('text', '')
+        total_text = node_text
+        
+        # Add all children's text
+        for child_index in children_indices:
+            child_text = result_list[child_index].get('text', '')
+            if child_text:
+                total_text += '\n' + child_text
+        
+        # Calculate token count for combined text
+        result_list[i]['text_token_count'] = count_tokens(total_text, model=model)
+    
+    return result_list
+
+
+def tree_thinning_for_index(node_list, min_node_token=None, model=None):
+    def find_all_children(parent_index, parent_level, node_list):
+        children_indices = []
+        
+        for i in range(parent_index + 1, len(node_list)):
+            current_level = node_list[i]['level']
+            
+            if current_level <= parent_level:
+                break
+                
+            children_indices.append(i)
+        
+        return children_indices
+    
+    result_list = node_list.copy()
+    nodes_to_remove = set()
+    
+    for i in range(len(result_list) - 1, -1, -1):
+        if i in nodes_to_remove:
+            continue
+            
+        current_node = result_list[i]
+        current_level = current_node['level']
+        
+        total_tokens = current_node.get('text_token_count', 0)
+        
+        if total_tokens < min_node_token:
+            children_indices = find_all_children(i, current_level, result_list)
+            
+            children_texts = []
+            for child_index in sorted(children_indices):
+                if child_index not in nodes_to_remove:
+                    child_text = result_list[child_index].get('text', '')
+                    if child_text.strip():
+                        children_texts.append(child_text)
+                    nodes_to_remove.add(child_index)
+            
+            if children_texts:
+                parent_text = current_node.get('text', '')
+                merged_text = parent_text
+                for child_text in children_texts:
+                    if merged_text and not merged_text.endswith('\n'):
+                        merged_text += '\n\n'
+                    merged_text += child_text
+                
+                result_list[i]['text'] = merged_text
+                
+                result_list[i]['text_token_count'] = count_tokens(merged_text, model=model)
+    
+    for index in sorted(nodes_to_remove, reverse=True):
+        result_list.pop(index)
+    
+    return result_list
+
+
+def build_tree_from_nodes(node_list):
+    if not node_list:
+        return []
+    
+    stack = []
+    root_nodes = []
+    node_counter = 1
+    
+    for node in node_list:
+        current_level = node['level']
+        
+        tree_node = {
+            'title': node['title'],
+            'node_id': str(node_counter).zfill(4),
+            'text': node['text'],
+            'line_num': node['line_num'],
+            'nodes': []
+        }
+        node_counter += 1
+        
+        while stack and stack[-1][1] >= current_level:
+            stack.pop()
+        
+        if not stack:
+            root_nodes.append(tree_node)
+        else:
+            parent_node, parent_level = stack[-1]
+            parent_node['nodes'].append(tree_node)
+        
+        stack.append((tree_node, current_level))
+    
+    return root_nodes
+
+
+def clean_tree_for_output(tree_nodes):
+    cleaned_nodes = []
+    
+    for node in tree_nodes:
+        cleaned_node = {
+            'title': node['title'],
+            'node_id': node['node_id'],
+            'text': node['text'],
+            'line_num': node['line_num']
+        }
+        
+        if node['nodes']:
+            cleaned_node['nodes'] = clean_tree_for_output(node['nodes'])
+        
+        cleaned_nodes.append(cleaned_node)
+    
+    return cleaned_nodes
+
+
+async def md_to_tree(md_path, if_thinning=False, min_token_threshold=None, if_add_node_summary='no', summary_token_threshold=None, model=None, if_add_doc_description='no', if_add_node_text='no', if_add_node_id='yes'):
+    with open(md_path, 'r', encoding='utf-8') as f:
+        markdown_content = f.read()
+    
+    print(f"Extracting nodes from markdown...")
+    node_list, markdown_lines = extract_nodes_from_markdown(markdown_content)
+
+    print(f"Extracting text content from nodes...")
+    nodes_with_content = extract_node_text_content(node_list, markdown_lines)
+    
+    if if_thinning:
+        nodes_with_content = update_node_list_with_text_token_count(nodes_with_content, model=model)
+        print(f"Thinning nodes...")
+        nodes_with_content = tree_thinning_for_index(nodes_with_content, min_token_threshold, model=model)
+    
+    print(f"Building tree from nodes...")
+    tree_structure = build_tree_from_nodes(nodes_with_content)
+
+    if if_add_node_id == 'yes':
+        write_node_id(tree_structure)
+
+    print(f"Formatting tree structure...")
+    
+    if if_add_node_summary == 'yes':
+        # Always include text for summary generation
+        tree_structure = format_structure(tree_structure, order = ['title', 'node_id', 'summary', 'prefix_summary', 'text', 'line_num', 'nodes'])
+        
+        print(f"Generating summaries for each node...")
+        tree_structure = await generate_summaries_for_structure_md(tree_structure, summary_token_threshold=summary_token_threshold, model=model)
+        
+        if if_add_node_text == 'no':
+            # Remove text after summary generation if not requested
+            tree_structure = format_structure(tree_structure, order = ['title', 'node_id', 'summary', 'prefix_summary', 'line_num', 'nodes'])
+        
+        if if_add_doc_description == 'yes':
+            print(f"Generating document description...")
+            # Create a clean structure without unnecessary fields for description generation
+            clean_structure = create_clean_structure_for_description(tree_structure)
+            doc_description = generate_doc_description(clean_structure, model=model)
+            return {
+                'doc_name': os.path.splitext(os.path.basename(md_path))[0],
+                'doc_description': doc_description,
+                'structure': tree_structure,
+            }
+    else:
+        # No summaries needed, format based on text preference
+        if if_add_node_text == 'yes':
+            tree_structure = format_structure(tree_structure, order = ['title', 'node_id', 'summary', 'prefix_summary', 'text', 'line_num', 'nodes'])
+        else:
+            tree_structure = format_structure(tree_structure, order = ['title', 'node_id', 'summary', 'prefix_summary', 'line_num', 'nodes'])
+    
+    return {
+        'doc_name': os.path.splitext(os.path.basename(md_path))[0],
+        'structure': tree_structure,
+    }
+
+
+if __name__ == "__main__":
+    import os
+    import json
+    
+    # MD_NAME = 'Detect-Order-Construct'
+    MD_NAME = 'cognitive-load'
+    MD_PATH = os.path.join(os.path.dirname(__file__), '..', 'tests/markdowns/', f'{MD_NAME}.md')
+
+
+    MODEL="gpt-4.1"
+    IF_THINNING=False
+    THINNING_THRESHOLD=5000
+    SUMMARY_TOKEN_THRESHOLD=200
+    IF_SUMMARY=True
+
+    tree_structure = asyncio.run(md_to_tree(
+        md_path=MD_PATH, 
+        if_thinning=IF_THINNING, 
+        min_token_threshold=THINNING_THRESHOLD, 
+        if_add_node_summary='yes' if IF_SUMMARY else 'no', 
+        summary_token_threshold=SUMMARY_TOKEN_THRESHOLD, 
+        model=MODEL))
+    
+    print('\n' + '='*60)
+    print('TREE STRUCTURE')
+    print('='*60)
+    print_json(tree_structure)
+
+    print('\n' + '='*60)
+    print('TABLE OF CONTENTS')
+    print('='*60)
+    print_toc(tree_structure['structure'])
+
+    output_path = os.path.join(os.path.dirname(__file__), '..', 'results', f'{MD_NAME}_structure.json')
+    os.makedirs(os.path.dirname(output_path), exist_ok=True)
+    
+    with open(output_path, 'w', encoding='utf-8') as f:
+        json.dump(tree_structure, f, indent=2, ensure_ascii=False)
+    
+    print(f"\nTree structure saved to: {output_path}")

pageindex/utils.py

@@ -0,0 +1,712 @@
+import tiktoken
+import openai
+import logging
+import os
+from datetime import datetime
+import time
+import json
+import PyPDF2
+import copy
+import asyncio
+import pymupdf
+from io import BytesIO
+from dotenv import load_dotenv
+load_dotenv()
+import logging
+import yaml
+from pathlib import Path
+from types import SimpleNamespace as config
+
+CHATGPT_API_KEY = os.getenv("CHATGPT_API_KEY")
+
+def count_tokens(text, model=None):
+    if not text:
+        return 0
+    enc = tiktoken.encoding_for_model(model)
+    tokens = enc.encode(text)
+    return len(tokens)
+
+def ChatGPT_API_with_finish_reason(model, prompt, api_key=CHATGPT_API_KEY, chat_history=None):
+    max_retries = 10
+    client = openai.OpenAI(api_key=api_key)
+    for i in range(max_retries):
+        try:
+            if chat_history:
+                messages = chat_history
+                messages.append({"role": "user", "content": prompt})
+            else:
+                messages = [{"role": "user", "content": prompt}]
+            
+            response = client.chat.completions.create(
+                model=model,
+                messages=messages,
+                temperature=0,
+            )
+            if response.choices[0].finish_reason == "length":
+                return response.choices[0].message.content, "max_output_reached"
+            else:
+                return response.choices[0].message.content, "finished"
+
+        except Exception as e:
+            print('************* Retrying *************')
+            logging.error(f"Error: {e}")
+            if i < max_retries - 1:
+                time.sleep(1)  # Wait for 1秒 before retrying
+            else:
+                logging.error('Max retries reached for prompt: ' + prompt)
+                return "Error"
+
+
+
+def ChatGPT_API(model, prompt, api_key=CHATGPT_API_KEY, chat_history=None):
+    max_retries = 10
+    client = openai.OpenAI(api_key=api_key)
+    for i in range(max_retries):
+        try:
+            if chat_history:
+                messages = chat_history
+                messages.append({"role": "user", "content": prompt})
+            else:
+                messages = [{"role": "user", "content": prompt}]
+            
+            response = client.chat.completions.create(
+                model=model,
+                messages=messages,
+                temperature=0,
+            )
+   
+            return response.choices[0].message.content
+        except Exception as e:
+            print('************* Retrying *************')
+            logging.error(f"Error: {e}")
+            if i < max_retries - 1:
+                time.sleep(1)  # Wait for 1秒 before retrying
+            else:
+                logging.error('Max retries reached for prompt: ' + prompt)
+                return "Error"
+            
+
+async def ChatGPT_API_async(model, prompt, api_key=CHATGPT_API_KEY):
+    max_retries = 10
+    messages = [{"role": "user", "content": prompt}]
+    for i in range(max_retries):
+        try:
+            async with openai.AsyncOpenAI(api_key=api_key) as client:
+                response = await client.chat.completions.create(
+                    model=model,
+                    messages=messages,
+                    temperature=0,
+                )
+                return response.choices[0].message.content
+        except Exception as e:
+            print('************* Retrying *************')
+            logging.error(f"Error: {e}")
+            if i < max_retries - 1:
+                await asyncio.sleep(1)  # Wait for 1s before retrying
+            else:
+                logging.error('Max retries reached for prompt: ' + prompt)
+                return "Error"  
+            
+            
+def get_json_content(response):
+    start_idx = response.find("```json")
+    if start_idx != -1:
+        start_idx += 7
+        response = response[start_idx:]
+        
+    end_idx = response.rfind("```")
+    if end_idx != -1:
+        response = response[:end_idx]
+    
+    json_content = response.strip()
+    return json_content
+         
+
+def extract_json(content):
+    try:
+        # First, try to extract JSON enclosed within ```json and ```
+        start_idx = content.find("```json")
+        if start_idx != -1:
+            start_idx += 7  # Adjust index to start after the delimiter
+            end_idx = content.rfind("```")
+            json_content = content[start_idx:end_idx].strip()
+        else:
+            # If no delimiters, assume entire content could be JSON
+            json_content = content.strip()
+
+        # Clean up common issues that might cause parsing errors
+        json_content = json_content.replace('None', 'null')  # Replace Python None with JSON null
+        json_content = json_content.replace('\n', ' ').replace('\r', ' ')  # Remove newlines
+        json_content = ' '.join(json_content.split())  # Normalize whitespace
+
+        # Attempt to parse and return the JSON object
+        return json.loads(json_content)
+    except json.JSONDecodeError as e:
+        logging.error(f"Failed to extract JSON: {e}")
+        # Try to clean up the content further if initial parsing fails
+        try:
+            # Remove any trailing commas before closing brackets/braces
+            json_content = json_content.replace(',]', ']').replace(',}', '}')
+            return json.loads(json_content)
+        except:
+            logging.error("Failed to parse JSON even after cleanup")
+            return {}
+    except Exception as e:
+        logging.error(f"Unexpected error while extracting JSON: {e}")
+        return {}
+
+def write_node_id(data, node_id=0):
+    if isinstance(data, dict):
+        data['node_id'] = str(node_id).zfill(4)
+        node_id += 1
+        for key in list(data.keys()):
+            if 'nodes' in key:
+                node_id = write_node_id(data[key], node_id)
+    elif isinstance(data, list):
+        for index in range(len(data)):
+            node_id = write_node_id(data[index], node_id)
+    return node_id
+
+def get_nodes(structure):
+    if isinstance(structure, dict):
+        structure_node = copy.deepcopy(structure)
+        structure_node.pop('nodes', None)
+        nodes = [structure_node]
+        for key in list(structure.keys()):
+            if 'nodes' in key:
+                nodes.extend(get_nodes(structure[key]))
+        return nodes
+    elif isinstance(structure, list):
+        nodes = []
+        for item in structure:
+            nodes.extend(get_nodes(item))
+        return nodes
+    
+def structure_to_list(structure):
+    if isinstance(structure, dict):
+        nodes = []
+        nodes.append(structure)
+        if 'nodes' in structure:
+            nodes.extend(structure_to_list(structure['nodes']))
+        return nodes
+    elif isinstance(structure, list):
+        nodes = []
+        for item in structure:
+            nodes.extend(structure_to_list(item))
+        return nodes
+
+    
+def get_leaf_nodes(structure):
+    if isinstance(structure, dict):
+        if not structure['nodes']:
+            structure_node = copy.deepcopy(structure)
+            structure_node.pop('nodes', None)
+            return [structure_node]
+        else:
+            leaf_nodes = []
+            for key in list(structure.keys()):
+                if 'nodes' in key:
+                    leaf_nodes.extend(get_leaf_nodes(structure[key]))
+            return leaf_nodes
+    elif isinstance(structure, list):
+        leaf_nodes = []
+        for item in structure:
+            leaf_nodes.extend(get_leaf_nodes(item))
+        return leaf_nodes
+
+def is_leaf_node(data, node_id):
+    # Helper function to find the node by its node_id
+    def find_node(data, node_id):
+        if isinstance(data, dict):
+            if data.get('node_id') == node_id:
+                return data
+            for key in data.keys():
+                if 'nodes' in key:
+                    result = find_node(data[key], node_id)
+                    if result:
+                        return result
+        elif isinstance(data, list):
+            for item in data:
+                result = find_node(item, node_id)
+                if result:
+                    return result
+        return None
+
+    # Find the node with the given node_id
+    node = find_node(data, node_id)
+
+    # Check if the node is a leaf node
+    if node and not node.get('nodes'):
+        return True
+    return False
+
+def get_last_node(structure):
+    return structure[-1]
+
+
+def extract_text_from_pdf(pdf_path):
+    pdf_reader = PyPDF2.PdfReader(pdf_path)
+    ###return text not list 
+    text=""
+    for page_num in range(len(pdf_reader.pages)):
+        page = pdf_reader.pages[page_num]
+        text+=page.extract_text()
+    return text
+
+def get_pdf_title(pdf_path):
+    pdf_reader = PyPDF2.PdfReader(pdf_path)
+    meta = pdf_reader.metadata
+    title = meta.title if meta and meta.title else 'Untitled'
+    return title
+
+def get_text_of_pages(pdf_path, start_page, end_page, tag=True):
+    pdf_reader = PyPDF2.PdfReader(pdf_path)
+    text = ""
+    for page_num in range(start_page-1, end_page):
+        page = pdf_reader.pages[page_num]
+        page_text = page.extract_text()
+        if tag:
+            text += f"<start_index_{page_num+1}>\n{page_text}\n<end_index_{page_num+1}>\n"
+        else:
+            text += page_text
+    return text
+
+def get_first_start_page_from_text(text):
+    start_page = -1
+    start_page_match = re.search(r'<start_index_(\d+)>', text)
+    if start_page_match:
+        start_page = int(start_page_match.group(1))
+    return start_page
+
+def get_last_start_page_from_text(text):
+    start_page = -1
+    # Find all matches of start_index tags
+    start_page_matches = re.finditer(r'<start_index_(\d+)>', text)
+    # Convert iterator to list and get the last match if any exist
+    matches_list = list(start_page_matches)
+    if matches_list:
+        start_page = int(matches_list[-1].group(1))
+    return start_page
+
+
+def sanitize_filename(filename, replacement='-'):
+    # In Linux, only '/' and '\0' (null) are invalid in filenames.
+    # Null can't be represented in strings, so we only handle '/'.
+    return filename.replace('/', replacement)
+
+def get_pdf_name(pdf_path):
+    # Extract PDF name
+    if isinstance(pdf_path, str):
+        pdf_name = os.path.basename(pdf_path)
+    elif isinstance(pdf_path, BytesIO):
+        pdf_reader = PyPDF2.PdfReader(pdf_path)
+        meta = pdf_reader.metadata
+        pdf_name = meta.title if meta and meta.title else 'Untitled'
+        pdf_name = sanitize_filename(pdf_name)
+    return pdf_name
+
+
+class JsonLogger:
+    def __init__(self, file_path):
+        # Extract PDF name for logger name
+        pdf_name = get_pdf_name(file_path)
+            
+        current_time = datetime.now().strftime("%Y%m%d_%H%M%S")
+        self.filename = f"{pdf_name}_{current_time}.json"
+        os.makedirs("./logs", exist_ok=True)
+        # Initialize empty list to store all messages
+        self.log_data = []
+
+    def log(self, level, message, **kwargs):
+        if isinstance(message, dict):
+            self.log_data.append(message)
+        else:
+            self.log_data.append({'message': message})
+        # Add new message to the log data
+        
+        # Write entire log data to file
+        with open(self._filepath(), "w") as f:
+            json.dump(self.log_data, f, indent=2)
+
+    def info(self, message, **kwargs):
+        self.log("INFO", message, **kwargs)
+
+    def error(self, message, **kwargs):
+        self.log("ERROR", message, **kwargs)
+
+    def debug(self, message, **kwargs):
+        self.log("DEBUG", message, **kwargs)
+
+    def exception(self, message, **kwargs):
+        kwargs["exception"] = True
+        self.log("ERROR", message, **kwargs)
+
+    def _filepath(self):
+        return os.path.join("logs", self.filename)
+    
+
+
+
+def list_to_tree(data):
+    def get_parent_structure(structure):
+        """Helper function to get the parent structure code"""
+        if not structure:
+            return None
+        parts = str(structure).split('.')
+        return '.'.join(parts[:-1]) if len(parts) > 1 else None
+    
+    # First pass: Create nodes and track parent-child relationships
+    nodes = {}
+    root_nodes = []
+    
+    for item in data:
+        structure = item.get('structure')
+        node = {
+            'title': item.get('title'),
+            'start_index': item.get('start_index'),
+            'end_index': item.get('end_index'),
+            'nodes': []
+        }
+        
+        nodes[structure] = node
+        
+        # Find parent
+        parent_structure = get_parent_structure(structure)
+        
+        if parent_structure:
+            # Add as child to parent if parent exists
+            if parent_structure in nodes:
+                nodes[parent_structure]['nodes'].append(node)
+            else:
+                root_nodes.append(node)
+        else:
+            # No parent, this is a root node
+            root_nodes.append(node)
+    
+    # Helper function to clean empty children arrays
+    def clean_node(node):
+        if not node['nodes']:
+            del node['nodes']
+        else:
+            for child in node['nodes']:
+                clean_node(child)
+        return node
+    
+    # Clean and return the tree
+    return [clean_node(node) for node in root_nodes]
+
+def add_preface_if_needed(data):
+    if not isinstance(data, list) or not data:
+        return data
+
+    if data[0]['physical_index'] is not None and data[0]['physical_index'] > 1:
+        preface_node = {
+            "structure": "0",
+            "title": "Preface",
+            "physical_index": 1,
+        }
+        data.insert(0, preface_node)
+    return data
+
+
+
+def get_page_tokens(pdf_path, model="gpt-4o-2024-11-20", pdf_parser="PyPDF2"):
+    enc = tiktoken.encoding_for_model(model)
+    if pdf_parser == "PyPDF2":
+        pdf_reader = PyPDF2.PdfReader(pdf_path)
+        page_list = []
+        for page_num in range(len(pdf_reader.pages)):
+            page = pdf_reader.pages[page_num]
+            page_text = page.extract_text()
+            token_length = len(enc.encode(page_text))
+            page_list.append((page_text, token_length))
+        return page_list
+    elif pdf_parser == "PyMuPDF":
+        if isinstance(pdf_path, BytesIO):
+            pdf_stream = pdf_path
+            doc = pymupdf.open(stream=pdf_stream, filetype="pdf")
+        elif isinstance(pdf_path, str) and os.path.isfile(pdf_path) and pdf_path.lower().endswith(".pdf"):
+            doc = pymupdf.open(pdf_path)
+        page_list = []
+        for page in doc:
+            page_text = page.get_text()
+            token_length = len(enc.encode(page_text))
+            page_list.append((page_text, token_length))
+        return page_list
+    else:
+        raise ValueError(f"Unsupported PDF parser: {pdf_parser}")
+
+        
+
+def get_text_of_pdf_pages(pdf_pages, start_page, end_page):
+    text = ""
+    for page_num in range(start_page-1, end_page):
+        text += pdf_pages[page_num][0]
+    return text
+
+def get_text_of_pdf_pages_with_labels(pdf_pages, start_page, end_page):
+    text = ""
+    for page_num in range(start_page-1, end_page):
+        text += f"<physical_index_{page_num+1}>\n{pdf_pages[page_num][0]}\n<physical_index_{page_num+1}>\n"
+    return text
+
+def get_number_of_pages(pdf_path):
+    pdf_reader = PyPDF2.PdfReader(pdf_path)
+    num = len(pdf_reader.pages)
+    return num
+
+
+
+def post_processing(structure, end_physical_index):
+    # First convert page_number to start_index in flat list
+    for i, item in enumerate(structure):
+        item['start_index'] = item.get('physical_index')
+        if i < len(structure) - 1:
+            if structure[i + 1].get('appear_start') == 'yes':
+                item['end_index'] = structure[i + 1]['physical_index']-1
+            else:
+                item['end_index'] = structure[i + 1]['physical_index']
+        else:
+            item['end_index'] = end_physical_index
+    tree = list_to_tree(structure)
+    if len(tree)!=0:
+        return tree
+    else:
+        ### remove appear_start 
+        for node in structure:
+            node.pop('appear_start', None)
+            node.pop('physical_index', None)
+        return structure
+
+def clean_structure_post(data):
+    if isinstance(data, dict):
+        data.pop('page_number', None)
+        data.pop('start_index', None)
+        data.pop('end_index', None)
+        if 'nodes' in data:
+            clean_structure_post(data['nodes'])
+    elif isinstance(data, list):
+        for section in data:
+            clean_structure_post(section)
+    return data
+
+def remove_fields(data, fields=['text']):
+    if isinstance(data, dict):
+        return {k: remove_fields(v, fields)
+            for k, v in data.items() if k not in fields}
+    elif isinstance(data, list):
+        return [remove_fields(item, fields) for item in data]
+    return data
+
+def print_toc(tree, indent=0):
+    for node in tree:
+        print('  ' * indent + node['title'])
+        if node.get('nodes'):
+            print_toc(node['nodes'], indent + 1)
+
+def print_json(data, max_len=40, indent=2):
+    def simplify_data(obj):
+        if isinstance(obj, dict):
+            return {k: simplify_data(v) for k, v in obj.items()}
+        elif isinstance(obj, list):
+            return [simplify_data(item) for item in obj]
+        elif isinstance(obj, str) and len(obj) > max_len:
+            return obj[:max_len] + '...'
+        else:
+            return obj
+    
+    simplified = simplify_data(data)
+    print(json.dumps(simplified, indent=indent, ensure_ascii=False))
+
+
+def remove_structure_text(data):
+    if isinstance(data, dict):
+        data.pop('text', None)
+        if 'nodes' in data:
+            remove_structure_text(data['nodes'])
+    elif isinstance(data, list):
+        for item in data:
+            remove_structure_text(item)
+    return data
+
+
+def check_token_limit(structure, limit=110000):
+    list = structure_to_list(structure)
+    for node in list:
+        num_tokens = count_tokens(node['text'], model='gpt-4o')
+        if num_tokens > limit:
+            print(f"Node ID: {node['node_id']} has {num_tokens} tokens")
+            print("Start Index:", node['start_index'])
+            print("End Index:", node['end_index'])
+            print("Title:", node['title'])
+            print("\n")
+
+
+def convert_physical_index_to_int(data):
+    if isinstance(data, list):
+        for i in range(len(data)):
+            # Check if item is a dictionary and has 'physical_index' key
+            if isinstance(data[i], dict) and 'physical_index' in data[i]:
+                if isinstance(data[i]['physical_index'], str):
+                    if data[i]['physical_index'].startswith('<physical_index_'):
+                        data[i]['physical_index'] = int(data[i]['physical_index'].split('_')[-1].rstrip('>').strip())
+                    elif data[i]['physical_index'].startswith('physical_index_'):
+                        data[i]['physical_index'] = int(data[i]['physical_index'].split('_')[-1].strip())
+    elif isinstance(data, str):
+        if data.startswith('<physical_index_'):
+            data = int(data.split('_')[-1].rstrip('>').strip())
+        elif data.startswith('physical_index_'):
+            data = int(data.split('_')[-1].strip())
+        # Check data is int
+        if isinstance(data, int):
+            return data
+        else:
+            return None
+    return data
+
+
+def convert_page_to_int(data):
+    for item in data:
+        if 'page' in item and isinstance(item['page'], str):
+            try:
+                item['page'] = int(item['page'])
+            except ValueError:
+                # Keep original value if conversion fails
+                pass
+    return data
+
+
+def add_node_text(node, pdf_pages):
+    if isinstance(node, dict):
+        start_page = node.get('start_index')
+        end_page = node.get('end_index')
+        node['text'] = get_text_of_pdf_pages(pdf_pages, start_page, end_page)
+        if 'nodes' in node:
+            add_node_text(node['nodes'], pdf_pages)
+    elif isinstance(node, list):
+        for index in range(len(node)):
+            add_node_text(node[index], pdf_pages)
+    return
+
+
+def add_node_text_with_labels(node, pdf_pages):
+    if isinstance(node, dict):
+        start_page = node.get('start_index')
+        end_page = node.get('end_index')
+        node['text'] = get_text_of_pdf_pages_with_labels(pdf_pages, start_page, end_page)
+        if 'nodes' in node:
+            add_node_text_with_labels(node['nodes'], pdf_pages)
+    elif isinstance(node, list):
+        for index in range(len(node)):
+            add_node_text_with_labels(node[index], pdf_pages)
+    return
+
+
+async def generate_node_summary(node, model=None):
+    prompt = f"""You are given a part of a document, your task is to generate a description of the partial document about what are main points covered in the partial document.
+
+    Partial Document Text: {node['text']}
+    
+    Directly return the description, do not include any other text.
+    """
+    response = await ChatGPT_API_async(model, prompt)
+    return response
+
+
+async def generate_summaries_for_structure(structure, model=None):
+    nodes = structure_to_list(structure)
+    tasks = [generate_node_summary(node, model=model) for node in nodes]
+    summaries = await asyncio.gather(*tasks)
+    
+    for node, summary in zip(nodes, summaries):
+        node['summary'] = summary
+    return structure
+
+
+def create_clean_structure_for_description(structure):
+    """
+    Create a clean structure for document description generation,
+    excluding unnecessary fields like 'text'.
+    """
+    if isinstance(structure, dict):
+        clean_node = {}
+        # Only include essential fields for description
+        for key in ['title', 'node_id', 'summary', 'prefix_summary']:
+            if key in structure:
+                clean_node[key] = structure[key]
+        
+        # Recursively process child nodes
+        if 'nodes' in structure and structure['nodes']:
+            clean_node['nodes'] = create_clean_structure_for_description(structure['nodes'])
+        
+        return clean_node
+    elif isinstance(structure, list):
+        return [create_clean_structure_for_description(item) for item in structure]
+    else:
+        return structure
+
+
+def generate_doc_description(structure, model=None):
+    prompt = f"""Your are an expert in generating descriptions for a document.
+    You are given a structure of a document. Your task is to generate a one-sentence description for the document, which makes it easy to distinguish the document from other documents.
+        
+    Document Structure: {structure}
+    
+    Directly return the description, do not include any other text.
+    """
+    response = ChatGPT_API(model, prompt)
+    return response
+
+
+def reorder_dict(data, key_order):
+    if not key_order:
+        return data
+    return {key: data[key] for key in key_order if key in data}
+
+
+def format_structure(structure, order=None):
+    if not order:
+        return structure
+    if isinstance(structure, dict):
+        if 'nodes' in structure:
+            structure['nodes'] = format_structure(structure['nodes'], order)
+        if not structure.get('nodes'):
+            structure.pop('nodes', None)
+        structure = reorder_dict(structure, order)
+    elif isinstance(structure, list):
+        structure = [format_structure(item, order) for item in structure]
+    return structure
+
+
+class ConfigLoader:
+    def __init__(self, default_path: str = None):
+        if default_path is None:
+            default_path = Path(__file__).parent / "config.yaml"
+        self._default_dict = self._load_yaml(default_path)
+
+    @staticmethod
+    def _load_yaml(path):
+        with open(path, "r", encoding="utf-8") as f:
+            return yaml.safe_load(f) or {}
+
+    def _validate_keys(self, user_dict):
+        unknown_keys = set(user_dict) - set(self._default_dict)
+        if unknown_keys:
+            raise ValueError(f"Unknown config keys: {unknown_keys}")
+
+    def load(self, user_opt=None) -> config:
+        """
+        Load the configuration, merging user options with default values.
+        """
+        if user_opt is None:
+            user_dict = {}
+        elif isinstance(user_opt, config):
+            user_dict = vars(user_opt)
+        elif isinstance(user_opt, dict):
+            user_dict = user_opt
+        else:
+            raise TypeError("user_opt must be dict, config(SimpleNamespace) or None")
+
+        self._validate_keys(user_dict)
+        merged = {**self._default_dict, **user_dict}
+        return config(**merged)

requirements.txt

@@ -0,0 +1,8 @@
+gradio
+openai
+pydantic
+python-dotenv
+pymupdf==1.26.4
+PyPDF2==3.0.1
+tiktoken==0.11.0
+pyyaml==6.0.2

run_pageindex.py

@@ -0,0 +1,133 @@
+import argparse
+import os
+import json
+from pageindex import *
+from pageindex.page_index_md import md_to_tree
+
+if __name__ == "__main__":
+    # Set up argument parser
+    parser = argparse.ArgumentParser(description='Process PDF or Markdown document and generate structure')
+    parser.add_argument('--pdf_path', type=str, help='Path to the PDF file')
+    parser.add_argument('--md_path', type=str, help='Path to the Markdown file')
+
+    parser.add_argument('--model', type=str, default='gpt-4o-2024-11-20', help='Model to use')
+
+    parser.add_argument('--toc-check-pages', type=int, default=20, 
+                      help='Number of pages to check for table of contents (PDF only)')
+    parser.add_argument('--max-pages-per-node', type=int, default=10,
+                      help='Maximum number of pages per node (PDF only)')
+    parser.add_argument('--max-tokens-per-node', type=int, default=20000,
+                      help='Maximum number of tokens per node (PDF only)')
+
+    parser.add_argument('--if-add-node-id', type=str, default='yes',
+                      help='Whether to add node id to the node')
+    parser.add_argument('--if-add-node-summary', type=str, default='yes',
+                      help='Whether to add summary to the node')
+    parser.add_argument('--if-add-doc-description', type=str, default='no',
+                      help='Whether to add doc description to the doc')
+    parser.add_argument('--if-add-node-text', type=str, default='no',
+                      help='Whether to add text to the node')
+                      
+    # Markdown specific arguments
+    parser.add_argument('--if-thinning', type=str, default='no',
+                      help='Whether to apply tree thinning for markdown (markdown only)')
+    parser.add_argument('--thinning-threshold', type=int, default=5000,
+                      help='Minimum token threshold for thinning (markdown only)')
+    parser.add_argument('--summary-token-threshold', type=int, default=200,
+                      help='Token threshold for generating summaries (markdown only)')
+    args = parser.parse_args()
+    
+    # Validate that exactly one file type is specified
+    if not args.pdf_path and not args.md_path:
+        raise ValueError("Either --pdf_path or --md_path must be specified")
+    if args.pdf_path and args.md_path:
+        raise ValueError("Only one of --pdf_path or --md_path can be specified")
+    
+    if args.pdf_path:
+        # Validate PDF file
+        if not args.pdf_path.lower().endswith('.pdf'):
+            raise ValueError("PDF file must have .pdf extension")
+        if not os.path.isfile(args.pdf_path):
+            raise ValueError(f"PDF file not found: {args.pdf_path}")
+            
+        # Process PDF file
+        # Configure options
+        opt = config(
+            model=args.model,
+            toc_check_page_num=args.toc_check_pages,
+            max_page_num_each_node=args.max_pages_per_node,
+            max_token_num_each_node=args.max_tokens_per_node,
+            if_add_node_id=args.if_add_node_id,
+            if_add_node_summary=args.if_add_node_summary,
+            if_add_doc_description=args.if_add_doc_description,
+            if_add_node_text=args.if_add_node_text
+        )
+
+        # Process the PDF
+        toc_with_page_number = page_index_main(args.pdf_path, opt)
+        print('Parsing done, saving to file...')
+        
+        # Save results
+        pdf_name = os.path.splitext(os.path.basename(args.pdf_path))[0]    
+        output_dir = './results'
+        output_file = f'{output_dir}/{pdf_name}_structure.json'
+        os.makedirs(output_dir, exist_ok=True)
+        
+        with open(output_file, 'w', encoding='utf-8') as f:
+            json.dump(toc_with_page_number, f, indent=2)
+        
+        print(f'Tree structure saved to: {output_file}')
+            
+    elif args.md_path:
+        # Validate Markdown file
+        if not args.md_path.lower().endswith(('.md', '.markdown')):
+            raise ValueError("Markdown file must have .md or .markdown extension")
+        if not os.path.isfile(args.md_path):
+            raise ValueError(f"Markdown file not found: {args.md_path}")
+            
+        # Process markdown file
+        print('Processing markdown file...')
+        
+        # Process the markdown
+        import asyncio
+        
+        # Use ConfigLoader to get consistent defaults (matching PDF behavior)
+        from pageindex.utils import ConfigLoader
+        config_loader = ConfigLoader()
+        
+        # Create options dict with user args
+        user_opt = {
+            'model': args.model,
+            'if_add_node_summary': args.if_add_node_summary,
+            'if_add_doc_description': args.if_add_doc_description,
+            'if_add_node_text': args.if_add_node_text,
+            'if_add_node_id': args.if_add_node_id
+        }
+        
+        # Load config with defaults from config.yaml
+        opt = config_loader.load(user_opt)
+        
+        toc_with_page_number = asyncio.run(md_to_tree(
+            md_path=args.md_path,
+            if_thinning=args.if_thinning.lower() == 'yes',
+            min_token_threshold=args.thinning_threshold,
+            if_add_node_summary=opt.if_add_node_summary,
+            summary_token_threshold=args.summary_token_threshold,
+            model=opt.model,
+            if_add_doc_description=opt.if_add_doc_description,
+            if_add_node_text=opt.if_add_node_text,
+            if_add_node_id=opt.if_add_node_id
+        ))
+        
+        print('Parsing done, saving to file...')
+        
+        # Save results
+        md_name = os.path.splitext(os.path.basename(args.md_path))[0]    
+        output_dir = './results'
+        output_file = f'{output_dir}/{md_name}_structure.json'
+        os.makedirs(output_dir, exist_ok=True)
+        
+        with open(output_file, 'w', encoding='utf-8') as f:
+            json.dump(toc_with_page_number, f, indent=2, ensure_ascii=False)
+        
+        print(f'Tree structure saved to: {output_file}')

tests/pdfs/2023-annual-report-truncated.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e1702e593dcc2b55f1f121422b6787fcde37c387cc6228ba783fe79825d0214
+size 1463411

tests/pdfs/2023-annual-report.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aacf44d05ebbc21f5e78d029864dc69871f88364b61a2ae405ad4ca43143f666
+size 2255266

tests/pdfs/PRML.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ee767e0a6b04fa05ba7e599e9dbb4637a94a4407ccedf0b4d316b1fd7c8ec64
+size 18090775

tests/pdfs/Regulation Best Interest_Interpretive release.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b1f896c166dace5e1314af16b5c4709a16a2aa6dccd6955d2779ceba11113d15
+size 357253

tests/pdfs/Regulation Best Interest_proposed rule.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8611d968d8ba9542ec775f43b80953b365758599701fd9c438383a6b82cd58f
+size 1803860

tests/pdfs/earthmover.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b28ea18276ce6ebb0ba1298813ce96fa812bb70ec8851933f07d18424813d74d
+size 1036803

tests/pdfs/four-lectures.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37bf130021e972b4203038e81d88298f13dca7f6fae95a1f3dfdec58935c4e11
+size 310124

tests/pdfs/q1-fy25-earnings.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:edb5d4c78cc2c403f2387a7b8aa7134c1a96018acb489722d13aa6a968ae93bb
+size 101417

tests/results/2023-annual-report-truncated_structure.json

@@ -0,0 +1,83 @@
+{
+  "doc_name": "2023-annual-report-truncated.pdf",
+  "structure": [
+    {
+      "title": "Preface",
+      "start_index": 1,
+      "end_index": 4,
+      "node_id": "0000"
+    },
+    {
+      "title": "About the Federal Reserve",
+      "start_index": 5,
+      "end_index": 7,
+      "node_id": "0001"
+    },
+    {
+      "title": "Overview",
+      "start_index": 7,
+      "end_index": 8,
+      "node_id": "0002"
+    },
+    {
+      "title": "Monetary Policy and Economic Developments",
+      "start_index": 9,
+      "end_index": 9,
+      "nodes": [
+        {
+          "title": "March 2024 Summary",
+          "start_index": 9,
+          "end_index": 14,
+          "node_id": "0004"
+        },
+        {
+          "title": "June 2023 Summary",
+          "start_index": 15,
+          "end_index": 20,
+          "node_id": "0005"
+        }
+      ],
+      "node_id": "0003"
+    },
+    {
+      "title": "Financial Stability",
+      "start_index": 21,
+      "end_index": 21,
+      "nodes": [
+        {
+          "title": "Monitoring Financial Vulnerabilities",
+          "start_index": 22,
+          "end_index": 28,
+          "node_id": "0007"
+        },
+        {
+          "title": "Domestic and International Cooperation and Coordination",
+          "start_index": 28,
+          "end_index": 30,
+          "node_id": "0008"
+        }
+      ],
+      "node_id": "0006"
+    },
+    {
+      "title": "Supervision and Regulation",
+      "start_index": 31,
+      "end_index": 32,
+      "nodes": [
+        {
+          "title": "Supervised and Regulated Institutions",
+          "start_index": 32,
+          "end_index": 35,
+          "node_id": "0010"
+        },
+        {
+          "title": "Supervisory Developments",
+          "start_index": 35,
+          "end_index": 50,
+          "node_id": "0011"
+        }
+      ],
+      "node_id": "0009"
+    }
+  ]
+}

tests/results/2023-annual-report_structure.json

@@ -0,0 +1,493 @@
+{
+  "doc_name": "2023-annual-report.pdf",
+  "structure": [
+    {
+      "title": "Preface",
+      "start_index": 1,
+      "end_index": 4,
+      "node_id": "0000"
+    },
+    {
+      "title": "About the Federal Reserve",
+      "start_index": 5,
+      "end_index": 6,
+      "node_id": "0001"
+    },
+    {
+      "title": "Overview",
+      "start_index": 7,
+      "end_index": 8,
+      "node_id": "0002"
+    },
+    {
+      "title": "Monetary Policy and Economic Developments",
+      "start_index": 9,
+      "end_index": 9,
+      "nodes": [
+        {
+          "title": "March 2024 Summary",
+          "start_index": 9,
+          "end_index": 14,
+          "node_id": "0004"
+        },
+        {
+          "title": "June 2023 Summary",
+          "start_index": 15,
+          "end_index": 20,
+          "node_id": "0005"
+        }
+      ],
+      "node_id": "0003"
+    },
+    {
+      "title": "Financial Stability",
+      "start_index": 21,
+      "end_index": 21,
+      "nodes": [
+        {
+          "title": "Monitoring Financial Vulnerabilities",
+          "start_index": 22,
+          "end_index": 28,
+          "node_id": "0007"
+        },
+        {
+          "title": "Domestic and International Cooperation and Coordination",
+          "start_index": 28,
+          "end_index": 31,
+          "node_id": "0008"
+        }
+      ],
+      "node_id": "0006"
+    },
+    {
+      "title": "Supervision and Regulation",
+      "start_index": 31,
+      "end_index": 31,
+      "nodes": [
+        {
+          "title": "Supervised and Regulated Institutions",
+          "start_index": 32,
+          "end_index": 35,
+          "node_id": "0010"
+        },
+        {
+          "title": "Supervisory Developments",
+          "start_index": 35,
+          "end_index": 54,
+          "node_id": "0011"
+        },
+        {
+          "title": "Regulatory Developments",
+          "start_index": 55,
+          "end_index": 59,
+          "node_id": "0012"
+        }
+      ],
+      "node_id": "0009"
+    },
+    {
+      "title": "Payment System and Reserve Bank Oversight",
+      "start_index": 59,
+      "end_index": 59,
+      "nodes": [
+        {
+          "title": "Payment Services to Depository and Other Institutions",
+          "start_index": 60,
+          "end_index": 65,
+          "node_id": "0014"
+        },
+        {
+          "title": "Currency and Coin",
+          "start_index": 66,
+          "end_index": 68,
+          "node_id": "0015"
+        },
+        {
+          "title": "Fiscal Agency and Government Depository Services",
+          "start_index": 69,
+          "end_index": 72,
+          "node_id": "0016"
+        },
+        {
+          "title": "Evolutions and Improvements to the System",
+          "start_index": 72,
+          "end_index": 75,
+          "node_id": "0017"
+        },
+        {
+          "title": "Oversight of Federal Reserve Banks",
+          "start_index": 75,
+          "end_index": 81,
+          "node_id": "0018"
+        },
+        {
+          "title": "Pro Forma Financial Statements for Federal Reserve Priced Services",
+          "start_index": 82,
+          "end_index": 88,
+          "node_id": "0019"
+        }
+      ],
+      "node_id": "0013"
+    },
+    {
+      "title": "Consumer and Community Affairs",
+      "start_index": 89,
+      "end_index": 89,
+      "nodes": [
+        {
+          "title": "Consumer Compliance Supervision",
+          "start_index": 89,
+          "end_index": 101,
+          "node_id": "0021"
+        },
+        {
+          "title": "Consumer Laws and Regulations",
+          "start_index": 101,
+          "end_index": 102,
+          "node_id": "0022"
+        },
+        {
+          "title": "Consumer Research and Analysis of Emerging Issues and Policy",
+          "start_index": 102,
+          "end_index": 105,
+          "node_id": "0023"
+        },
+        {
+          "title": "Community Development",
+          "start_index": 105,
+          "end_index": 106,
+          "node_id": "0024"
+        }
+      ],
+      "node_id": "0020"
+    },
+    {
+      "title": "Appendixes",
+      "start_index": 107,
+      "end_index": 109,
+      "node_id": "0025"
+    },
+    {
+      "title": "Federal Reserve System Organization",
+      "start_index": 109,
+      "end_index": 109,
+      "nodes": [
+        {
+          "title": "Board of Governors",
+          "start_index": 109,
+          "end_index": 116,
+          "node_id": "0027"
+        },
+        {
+          "title": "Federal Open Market Committee",
+          "start_index": 117,
+          "end_index": 118,
+          "node_id": "0028"
+        },
+        {
+          "title": "Board of Governors Advisory Councils",
+          "start_index": 119,
+          "end_index": 122,
+          "node_id": "0029"
+        },
+        {
+          "title": "Federal Reserve Banks and Branches",
+          "start_index": 123,
+          "end_index": 146,
+          "node_id": "0030"
+        }
+      ],
+      "node_id": "0026"
+    },
+    {
+      "title": "Minutes of Federal Open Market Committee Meetings",
+      "start_index": 147,
+      "end_index": 147,
+      "nodes": [
+        {
+          "title": "Meeting Minutes",
+          "start_index": 147,
+          "end_index": 149,
+          "node_id": "0032"
+        }
+      ],
+      "node_id": "0031"
+    },
+    {
+      "title": "Federal Reserve System Audits",
+      "start_index": 149,
+      "end_index": 149,
+      "nodes": [
+        {
+          "title": "Office of Inspector General Activities",
+          "start_index": 149,
+          "end_index": 151,
+          "node_id": "0034"
+        },
+        {
+          "title": "Government Accountability Office Reviews",
+          "start_index": 151,
+          "end_index": 152,
+          "node_id": "0035"
+        }
+      ],
+      "node_id": "0033"
+    },
+    {
+      "title": "Federal Reserve System Budgets",
+      "start_index": 153,
+      "end_index": 153,
+      "nodes": [
+        {
+          "title": "System Budgets Overview",
+          "start_index": 153,
+          "end_index": 157,
+          "node_id": "0037"
+        },
+        {
+          "title": "Board of Governors Budgets",
+          "start_index": 157,
+          "end_index": 163,
+          "node_id": "0038"
+        },
+        {
+          "title": "Federal Reserve Banks Budgets",
+          "start_index": 163,
+          "end_index": 169,
+          "node_id": "0039"
+        },
+        {
+          "title": "Currency Budget",
+          "start_index": 169,
+          "end_index": 174,
+          "node_id": "0040"
+        }
+      ],
+      "node_id": "0036"
+    },
+    {
+      "title": "Record of Policy Actions of the Board of Governors",
+      "start_index": 175,
+      "end_index": 175,
+      "nodes": [
+        {
+          "title": "Rules and Regulations",
+          "start_index": 175,
+          "end_index": 176,
+          "node_id": "0042"
+        },
+        {
+          "title": "Policy Statements and Other Actions",
+          "start_index": 177,
+          "end_index": 181,
+          "node_id": "0043"
+        },
+        {
+          "title": "Discount Rates for Depository Institutions in 2023",
+          "start_index": 181,
+          "end_index": 183,
+          "node_id": "0044"
+        },
+        {
+          "title": "The Board of Governors and the Government Performance and Results Act",
+          "start_index": 184,
+          "end_index": 184,
+          "node_id": "0045"
+        }
+      ],
+      "node_id": "0041"
+    },
+    {
+      "title": "Litigation",
+      "start_index": 185,
+      "end_index": 185,
+      "nodes": [
+        {
+          "title": "Pending",
+          "start_index": 185,
+          "end_index": 186,
+          "node_id": "0047"
+        },
+        {
+          "title": "Resolved",
+          "start_index": 186,
+          "end_index": 187,
+          "node_id": "0048"
+        }
+      ],
+      "node_id": "0046"
+    },
+    {
+      "title": "Statistical Tables",
+      "start_index": 187,
+      "end_index": 187,
+      "nodes": [
+        {
+          "title": "Federal Reserve open market transactions, 2023",
+          "start_index": 187,
+          "end_index": 187,
+          "nodes": [
+            {
+              "title": "Type of security and transaction",
+              "start_index": 187,
+              "end_index": 188,
+              "node_id": "0051"
+            },
+            {
+              "title": "Federal agency obligations",
+              "start_index": 188,
+              "end_index": 188,
+              "node_id": "0052"
+            },
+            {
+              "title": "Mortgage-backed securities",
+              "start_index": 188,
+              "end_index": 188,
+              "node_id": "0053"
+            },
+            {
+              "title": "Temporary transactions",
+              "start_index": 188,
+              "end_index": 188,
+              "node_id": "0054"
+            }
+          ],
+          "node_id": "0050"
+        },
+        {
+          "title": "Federal Reserve Bank holdings of U.S. Treasury and federal agency securities, December 31, 2021\u201323",
+          "start_index": 189,
+          "end_index": 189,
+          "nodes": [
+            {
+              "title": "By remaining maturity",
+              "start_index": 189,
+              "end_index": 189,
+              "node_id": "0056"
+            },
+            {
+              "title": "By type",
+              "start_index": 189,
+              "end_index": 190,
+              "node_id": "0057"
+            },
+            {
+              "title": "By issuer",
+              "start_index": 190,
+              "end_index": 190,
+              "node_id": "0058"
+            }
+          ],
+          "node_id": "0055"
+        },
+        {
+          "title": "Reserve requirements of depository institutions, December 31, 2023",
+          "start_index": 191,
+          "end_index": 191,
+          "node_id": "0059"
+        },
+        {
+          "title": "Banking offices and banks affiliated with bank holding companies in the United States, December 31, 2022 and 2023",
+          "start_index": 192,
+          "end_index": 192,
+          "node_id": "0060"
+        },
+        {
+          "title": "Reserves of depository institutions, Federal Reserve Bank credit, and related items, year-end 1984\u20132023 and month-end 2023",
+          "start_index": 193,
+          "end_index": 196,
+          "node_id": "0061"
+        },
+        {
+          "title": "Reserves of depository institutions, Federal Reserve Bank credit, and related items, year-end 1918\u20131983",
+          "start_index": 197,
+          "end_index": 200,
+          "node_id": "0062"
+        },
+        {
+          "title": "Principal assets and liabilities of insured commercial banks, by class of bank, June 30, 2023 and 2022",
+          "start_index": 201,
+          "end_index": 201,
+          "node_id": "0063"
+        },
+        {
+          "title": "Initial margin requirements under Regulations T, U, and X",
+          "start_index": 202,
+          "end_index": 203,
+          "node_id": "0064"
+        },
+        {
+          "title": "Statement of condition of the Federal Reserve Banks, by Bank, December 31, 2023 and 2022",
+          "start_index": 203,
+          "end_index": 209,
+          "node_id": "0065"
+        },
+        {
+          "title": "Statement of condition of the Federal Reserve Banks, December 31, 2023 and 2022",
+          "start_index": 209,
+          "end_index": 210,
+          "node_id": "0066"
+        },
+        {
+          "title": "Income and expenses of the Federal Reserve Banks, by Bank, 2023",
+          "start_index": 210,
+          "end_index": 212,
+          "nodes": [
+            {
+              "title": "Income and expenses of the Federal Reserve Banks, by Bank, 2023\u2014continued",
+              "start_index": 212,
+              "end_index": 214,
+              "node_id": "0068"
+            }
+          ],
+          "node_id": "0067"
+        },
+        {
+          "title": "Income and expenses of the Federal Reserve Banks, 1914\u20132023",
+          "start_index": 214,
+          "end_index": 215,
+          "nodes": [
+            {
+              "title": "Income and expenses of the Federal Reserve Banks, 1914\u20132023\u2014continued",
+              "start_index": 215,
+              "end_index": 216,
+              "node_id": "0070"
+            },
+            {
+              "title": "Income and expenses of the Federal Reserve Banks, 1914\u20132023\u2014continued",
+              "start_index": 216,
+              "end_index": 217,
+              "node_id": "0071"
+            },
+            {
+              "title": "Income and expenses of the Federal Reserve Banks, 1914\u20132023\u2014continued",
+              "start_index": 217,
+              "end_index": 217,
+              "node_id": "0072"
+            }
+          ],
+          "node_id": "0069"
+        },
+        {
+          "title": "Operations in principal departments of the Federal Reserve Banks, 2020\u201323",
+          "start_index": 218,
+          "end_index": 218,
+          "node_id": "0073"
+        },
+        {
+          "title": "Number and annual salaries of officers and employees of the Federal Reserve Banks, December 31, 2023",
+          "start_index": 219,
+          "end_index": 220,
+          "node_id": "0074"
+        },
+        {
+          "title": "Acquisition costs and net book value of the premises of the Federal Reserve Banks and Branches, December 31, 2023",
+          "start_index": 220,
+          "end_index": 222,
+          "node_id": "0075"
+        }
+      ],
+      "node_id": "0049"
+    }
+  ]
+}

tests/results/PRML_structure.json

@@ -0,0 +1,1847 @@
+{
+  "doc_name": "PRML.pdf",
+  "structure": [
+    {
+      "title": "Preface",
+      "start_index": 1,
+      "end_index": 6,
+      "node_id": "0000"
+    },
+    {
+      "title": "Preface",
+      "start_index": 7,
+      "end_index": 10,
+      "node_id": "0001"
+    },
+    {
+      "title": "Mathematical notation",
+      "start_index": 11,
+      "end_index": 13,
+      "node_id": "0002"
+    },
+    {
+      "title": "Contents",
+      "start_index": 13,
+      "end_index": 20,
+      "node_id": "0003"
+    },
+    {
+      "title": "Introduction",
+      "start_index": 21,
+      "end_index": 24,
+      "nodes": [
+        {
+          "title": "Example: Polynomial Curve Fitting",
+          "start_index": 24,
+          "end_index": 32,
+          "node_id": "0005"
+        },
+        {
+          "title": "Probability Theory",
+          "start_index": 32,
+          "end_index": 37,
+          "nodes": [
+            {
+              "title": "Probability densities",
+              "start_index": 37,
+              "end_index": 39,
+              "node_id": "0007"
+            },
+            {
+              "title": "Expectations and covariances",
+              "start_index": 39,
+              "end_index": 41,
+              "node_id": "0008"
+            },
+            {
+              "title": "Bayesian probabilities",
+              "start_index": 41,
+              "end_index": 44,
+              "node_id": "0009"
+            },
+            {
+              "title": "The Gaussian distribution",
+              "start_index": 44,
+              "end_index": 48,
+              "node_id": "0010"
+            },
+            {
+              "title": "Curve fitting re-visited",
+              "start_index": 48,
+              "end_index": 50,
+              "node_id": "0011"
+            },
+            {
+              "title": "Bayesian curve fitting",
+              "start_index": 50,
+              "end_index": 52,
+              "node_id": "0012"
+            }
+          ],
+          "node_id": "0006"
+        },
+        {
+          "title": "Model Selection",
+          "start_index": 52,
+          "end_index": 53,
+          "node_id": "0013"
+        },
+        {
+          "title": "The Curse of Dimensionality",
+          "start_index": 53,
+          "end_index": 58,
+          "node_id": "0014"
+        },
+        {
+          "title": "Decision Theory",
+          "start_index": 58,
+          "end_index": 59,
+          "nodes": [
+            {
+              "title": "Minimizing the misclassification rate",
+              "start_index": 59,
+              "end_index": 61,
+              "node_id": "0016"
+            },
+            {
+              "title": "Minimizing the expected loss",
+              "start_index": 61,
+              "end_index": 62,
+              "node_id": "0017"
+            },
+            {
+              "title": "The reject option",
+              "start_index": 62,
+              "end_index": 62,
+              "node_id": "0018"
+            },
+            {
+              "title": "Inference and decision",
+              "start_index": 62,
+              "end_index": 66,
+              "node_id": "0019"
+            },
+            {
+              "title": "Loss functions for regression",
+              "start_index": 66,
+              "end_index": 68,
+              "node_id": "0020"
+            }
+          ],
+          "node_id": "0015"
+        },
+        {
+          "title": "Information Theory",
+          "start_index": 68,
+          "end_index": 75,
+          "nodes": [
+            {
+              "title": "Relative entropy and mutual information",
+              "start_index": 75,
+              "end_index": 78,
+              "node_id": "0022"
+            }
+          ],
+          "node_id": "0021"
+        }
+      ],
+      "node_id": "0004"
+    },
+    {
+      "title": "Exercises",
+      "start_index": 78,
+      "end_index": 87,
+      "node_id": "0023"
+    },
+    {
+      "title": "Probability Distributions",
+      "start_index": 87,
+      "end_index": 88,
+      "nodes": [
+        {
+          "title": "Binary Variables",
+          "start_index": 88,
+          "end_index": 91,
+          "nodes": [
+            {
+              "title": "The beta distribution",
+              "start_index": 91,
+              "end_index": 94,
+              "node_id": "0026"
+            }
+          ],
+          "node_id": "0025"
+        },
+        {
+          "title": "Multinomial Variables",
+          "start_index": 94,
+          "end_index": 96,
+          "nodes": [
+            {
+              "title": "The Dirichlet distribution",
+              "start_index": 96,
+              "end_index": 98,
+              "node_id": "0028"
+            }
+          ],
+          "node_id": "0027"
+        },
+        {
+          "title": "The Gaussian Distribution",
+          "start_index": 98,
+          "end_index": 105,
+          "nodes": [
+            {
+              "title": "Conditional Gaussian distributions",
+              "start_index": 105,
+              "end_index": 108,
+              "node_id": "0030"
+            },
+            {
+              "title": "Marginal Gaussian distributions",
+              "start_index": 108,
+              "end_index": 110,
+              "node_id": "0031"
+            },
+            {
+              "title": "Bayes\u2019 theorem for Gaussian variables",
+              "start_index": 110,
+              "end_index": 113,
+              "node_id": "0032"
+            },
+            {
+              "title": "Maximum likelihood for the Gaussian",
+              "start_index": 113,
+              "end_index": 114,
+              "node_id": "0033"
+            },
+            {
+              "title": "Sequential estimation",
+              "start_index": 114,
+              "end_index": 117,
+              "node_id": "0034"
+            },
+            {
+              "title": "Bayesian inference for the Gaussian",
+              "start_index": 117,
+              "end_index": 122,
+              "node_id": "0035"
+            },
+            {
+              "title": "Student\u2019s t-distribution",
+              "start_index": 122,
+              "end_index": 125,
+              "node_id": "0036"
+            },
+            {
+              "title": "Periodic variables",
+              "start_index": 125,
+              "end_index": 130,
+              "node_id": "0037"
+            },
+            {
+              "title": "Mixtures of Gaussians",
+              "start_index": 130,
+              "end_index": 133,
+              "node_id": "0038"
+            }
+          ],
+          "node_id": "0029"
+        },
+        {
+          "title": "The Exponential Family",
+          "start_index": 133,
+          "end_index": 136,
+          "nodes": [
+            {
+              "title": "Maximum likelihood and sufficient statistics",
+              "start_index": 136,
+              "end_index": 137,
+              "node_id": "0040"
+            },
+            {
+              "title": "Conjugate priors",
+              "start_index": 137,
+              "end_index": 137,
+              "node_id": "0041"
+            },
+            {
+              "title": "Noninformative priors",
+              "start_index": 137,
+              "end_index": 140,
+              "node_id": "0042"
+            }
+          ],
+          "node_id": "0039"
+        },
+        {
+          "title": "Nonparametric Methods",
+          "start_index": 140,
+          "end_index": 142,
+          "nodes": [
+            {
+              "title": "Kernel density estimators",
+              "start_index": 142,
+              "end_index": 144,
+              "node_id": "0044"
+            },
+            {
+              "title": "Nearest-neighbour methods",
+              "start_index": 144,
+              "end_index": 147,
+              "node_id": "0045"
+            }
+          ],
+          "node_id": "0043"
+        }
+      ],
+      "node_id": "0024"
+    },
+    {
+      "title": "Exercises",
+      "start_index": 147,
+      "end_index": 156,
+      "node_id": "0046"
+    },
+    {
+      "title": "Linear Models for Regression",
+      "start_index": 157,
+      "end_index": 158,
+      "nodes": [
+        {
+          "title": "Linear Basis Function Models",
+          "start_index": 158,
+          "end_index": 160,
+          "nodes": [
+            {
+              "title": "Maximum likelihood and least squares",
+              "start_index": 160,
+              "end_index": 163,
+              "node_id": "0049"
+            },
+            {
+              "title": "Geometry of least squares",
+              "start_index": 163,
+              "end_index": 163,
+              "node_id": "0050"
+            },
+            {
+              "title": "Sequential learning",
+              "start_index": 163,
+              "end_index": 164,
+              "node_id": "0051"
+            },
+            {
+              "title": "Regularized least squares",
+              "start_index": 164,
+              "end_index": 166,
+              "node_id": "0052"
+            },
+            {
+              "title": "Multiple outputs",
+              "start_index": 166,
+              "end_index": 167,
+              "node_id": "0053"
+            }
+          ],
+          "node_id": "0048"
+        },
+        {
+          "title": "The Bias-Variance Decomposition",
+          "start_index": 167,
+          "end_index": 172,
+          "node_id": "0054"
+        },
+        {
+          "title": "Bayesian Linear Regression",
+          "start_index": 172,
+          "end_index": 172,
+          "nodes": [
+            {
+              "title": "Parameter distribution",
+              "start_index": 172,
+              "end_index": 176,
+              "node_id": "0056"
+            },
+            {
+              "title": "Predictive distribution",
+              "start_index": 176,
+              "end_index": 179,
+              "node_id": "0057"
+            },
+            {
+              "title": "Equivalent kernel",
+              "start_index": 179,
+              "end_index": 181,
+              "node_id": "0058"
+            }
+          ],
+          "node_id": "0055"
+        },
+        {
+          "title": "Bayesian Model Comparison",
+          "start_index": 181,
+          "end_index": 185,
+          "node_id": "0059"
+        },
+        {
+          "title": "The Evidence Approximation",
+          "start_index": 185,
+          "end_index": 186,
+          "nodes": [
+            {
+              "title": "Evaluation of the evidence function",
+              "start_index": 186,
+              "end_index": 188,
+              "node_id": "0061"
+            },
+            {
+              "title": "Maximizing the evidence function",
+              "start_index": 188,
+              "end_index": 190,
+              "node_id": "0062"
+            },
+            {
+              "title": "Effective number of parameters",
+              "start_index": 190,
+              "end_index": 192,
+              "node_id": "0063"
+            }
+          ],
+          "node_id": "0060"
+        },
+        {
+          "title": "Limitations of Fixed Basis Functions",
+          "start_index": 192,
+          "end_index": 193,
+          "node_id": "0064"
+        }
+      ],
+      "node_id": "0047"
+    },
+    {
+      "title": "Exercises",
+      "start_index": 193,
+      "end_index": 199,
+      "node_id": "0065"
+    },
+    {
+      "title": "Linear Models for Classification",
+      "start_index": 199,
+      "end_index": 201,
+      "nodes": [
+        {
+          "title": "Discriminant Functions",
+          "start_index": 201,
+          "end_index": 201,
+          "nodes": [
+            {
+              "title": "Two classes",
+              "start_index": 201,
+              "end_index": 202,
+              "node_id": "0068"
+            },
+            {
+              "title": "Multiple classes",
+              "start_index": 202,
+              "end_index": 204,
+              "node_id": "0069"
+            },
+            {
+              "title": "Least squares for classification",
+              "start_index": 204,
+              "end_index": 206,
+              "node_id": "0070"
+            },
+            {
+              "title": "Fisher\u2019s linear discriminant",
+              "start_index": 206,
+              "end_index": 209,
+              "node_id": "0071"
+            },
+            {
+              "title": "Relation to least squares",
+              "start_index": 209,
+              "end_index": 211,
+              "node_id": "0072"
+            },
+            {
+              "title": "Fisher\u2019s discriminant for multiple classes",
+              "start_index": 211,
+              "end_index": 212,
+              "node_id": "0073"
+            },
+            {
+              "title": "The perceptron algorithm",
+              "start_index": 212,
+              "end_index": 216,
+              "node_id": "0074"
+            }
+          ],
+          "node_id": "0067"
+        },
+        {
+          "title": "Probabilistic Generative Models",
+          "start_index": 216,
+          "end_index": 218,
+          "nodes": [
+            {
+              "title": "Continuous inputs",
+              "start_index": 218,
+              "end_index": 220,
+              "node_id": "0076"
+            },
+            {
+              "title": "Maximum likelihood solution",
+              "start_index": 220,
+              "end_index": 222,
+              "node_id": "0077"
+            },
+            {
+              "title": "Discrete features",
+              "start_index": 222,
+              "end_index": 222,
+              "node_id": "0078"
+            },
+            {
+              "title": "Exponential family",
+              "start_index": 222,
+              "end_index": 223,
+              "node_id": "0079"
+            }
+          ],
+          "node_id": "0075"
+        },
+        {
+          "title": "Probabilistic Discriminative Models",
+          "start_index": 223,
+          "end_index": 224,
+          "nodes": [
+            {
+              "title": "Fixed basis functions",
+              "start_index": 224,
+              "end_index": 225,
+              "node_id": "0081"
+            },
+            {
+              "title": "Logistic regression",
+              "start_index": 225,
+              "end_index": 227,
+              "node_id": "0082"
+            },
+            {
+              "title": "Iterative reweighted least squares",
+              "start_index": 227,
+              "end_index": 229,
+              "node_id": "0083"
+            },
+            {
+              "title": "Multiclass logistic regression",
+              "start_index": 229,
+              "end_index": 230,
+              "node_id": "0084"
+            },
+            {
+              "title": "Probit regression",
+              "start_index": 230,
+              "end_index": 232,
+              "node_id": "0085"
+            },
+            {
+              "title": "Canonical link functions",
+              "start_index": 232,
+              "end_index": 232,
+              "node_id": "0086"
+            }
+          ],
+          "node_id": "0080"
+        },
+        {
+          "title": "The Laplace Approximation",
+          "start_index": 233,
+          "end_index": 236,
+          "nodes": [
+            {
+              "title": "Model comparison and BIC",
+              "start_index": 236,
+              "end_index": 237,
+              "node_id": "0088"
+            }
+          ],
+          "node_id": "0087"
+        },
+        {
+          "title": "Bayesian Logistic Regression",
+          "start_index": 237,
+          "end_index": 237,
+          "nodes": [
+            {
+              "title": "Laplace approximation",
+              "start_index": 237,
+              "end_index": 238,
+              "node_id": "0090"
+            },
+            {
+              "title": "Predictive distribution",
+              "start_index": 238,
+              "end_index": 240,
+              "node_id": "0091"
+            }
+          ],
+          "node_id": "0089"
+        }
+      ],
+      "node_id": "0066"
+    },
+    {
+      "title": "Exercises",
+      "start_index": 240,
+      "end_index": 245,
+      "node_id": "0092"
+    },
+    {
+      "title": "Neural Networks",
+      "start_index": 245,
+      "end_index": 247,
+      "nodes": [
+        {
+          "title": "Feed-forward Network Functions",
+          "start_index": 247,
+          "end_index": 251,
+          "nodes": [
+            {
+              "title": "Weight-space symmetries",
+              "start_index": 251,
+              "end_index": 252,
+              "node_id": "0095"
+            }
+          ],
+          "node_id": "0094"
+        },
+        {
+          "title": "Network Training",
+          "start_index": 252,
+          "end_index": 256,
+          "nodes": [
+            {
+              "title": "Parameter optimization",
+              "start_index": 256,
+              "end_index": 257,
+              "node_id": "0097"
+            },
+            {
+              "title": "Local quadratic approximation",
+              "start_index": 257,
+              "end_index": 259,
+              "node_id": "0098"
+            },
+            {
+              "title": "Use of gradient information",
+              "start_index": 259,
+              "end_index": 260,
+              "node_id": "0099"
+            },
+            {
+              "title": "Gradient descent optimization",
+              "start_index": 260,
+              "end_index": 261,
+              "node_id": "0100"
+            }
+          ],
+          "node_id": "0096"
+        },
+        {
+          "title": "Error Backpropagation",
+          "start_index": 261,
+          "end_index": 262,
+          "nodes": [
+            {
+              "title": "Evaluation of error-function derivatives",
+              "start_index": 262,
+              "end_index": 265,
+              "node_id": "0102"
+            },
+            {
+              "title": "A simple example",
+              "start_index": 265,
+              "end_index": 266,
+              "node_id": "0103"
+            },
+            {
+              "title": "Efficiency of backpropagation",
+              "start_index": 266,
+              "end_index": 267,
+              "node_id": "0104"
+            },
+            {
+              "title": "The Jacobian matrix",
+              "start_index": 267,
+              "end_index": 269,
+              "node_id": "0105"
+            }
+          ],
+          "node_id": "0101"
+        },
+        {
+          "title": "The Hessian Matrix",
+          "start_index": 269,
+          "end_index": 270,
+          "nodes": [
+            {
+              "title": "Diagonal approximation",
+              "start_index": 270,
+              "end_index": 271,
+              "node_id": "0107"
+            },
+            {
+              "title": "Outer product approximation",
+              "start_index": 271,
+              "end_index": 272,
+              "node_id": "0108"
+            },
+            {
+              "title": "Inverse Hessian",
+              "start_index": 272,
+              "end_index": 272,
+              "node_id": "0109"
+            },
+            {
+              "title": "Finite differences",
+              "start_index": 272,
+              "end_index": 273,
+              "node_id": "0110"
+            },
+            {
+              "title": "Exact evaluation of the Hessian",
+              "start_index": 273,
+              "end_index": 274,
+              "node_id": "0111"
+            },
+            {
+              "title": "Fast multiplication by the Hessian",
+              "start_index": 274,
+              "end_index": 276,
+              "node_id": "0112"
+            }
+          ],
+          "node_id": "0106"
+        },
+        {
+          "title": "Regularization in Neural Networks",
+          "start_index": 276,
+          "end_index": 277,
+          "nodes": [
+            {
+              "title": "Consistent Gaussian priors",
+              "start_index": 277,
+              "end_index": 279,
+              "node_id": "0114"
+            },
+            {
+              "title": "Early stopping",
+              "start_index": 279,
+              "end_index": 281,
+              "node_id": "0115"
+            },
+            {
+              "title": "Invariances",
+              "start_index": 281,
+              "end_index": 283,
+              "node_id": "0116"
+            },
+            {
+              "title": "Tangent propagation",
+              "start_index": 283,
+              "end_index": 285,
+              "node_id": "0117"
+            },
+            {
+              "title": "Training with transformed data",
+              "start_index": 285,
+              "end_index": 287,
+              "node_id": "0118"
+            },
+            {
+              "title": "Convolutional networks",
+              "start_index": 287,
+              "end_index": 289,
+              "node_id": "0119"
+            },
+            {
+              "title": "Soft weight sharing",
+              "start_index": 289,
+              "end_index": 292,
+              "node_id": "0120"
+            }
+          ],
+          "node_id": "0113"
+        },
+        {
+          "title": "Mixture Density Networks",
+          "start_index": 292,
+          "end_index": 297,
+          "node_id": "0121"
+        },
+        {
+          "title": "Bayesian Neural Networks",
+          "start_index": 297,
+          "end_index": 298,
+          "nodes": [
+            {
+              "title": "Posterior parameter distribution",
+              "start_index": 298,
+              "end_index": 300,
+              "node_id": "0123"
+            },
+            {
+              "title": "Hyperparameter optimization",
+              "start_index": 300,
+              "end_index": 301,
+              "node_id": "0124"
+            },
+            {
+              "title": "Bayesian neural networks for classification",
+              "start_index": 301,
+              "end_index": 304,
+              "node_id": "0125"
+            }
+          ],
+          "node_id": "0122"
+        }
+      ],
+      "node_id": "0093"
+    },
+    {
+      "title": "Exercises",
+      "start_index": 304,
+      "end_index": 311,
+      "node_id": "0126"
+    },
+    {
+      "title": "Kernel Methods",
+      "start_index": 311,
+      "end_index": 313,
+      "nodes": [
+        {
+          "title": "Dual Representations",
+          "start_index": 313,
+          "end_index": 314,
+          "node_id": "0128"
+        },
+        {
+          "title": "Constructing Kernels",
+          "start_index": 314,
+          "end_index": 319,
+          "node_id": "0129"
+        },
+        {
+          "title": "Radial Basis Function Networks",
+          "start_index": 319,
+          "end_index": 321,
+          "nodes": [
+            {
+              "title": "Nadaraya-Watson model",
+              "start_index": 321,
+              "end_index": 323,
+              "node_id": "0131"
+            }
+          ],
+          "node_id": "0130"
+        },
+        {
+          "title": "Gaussian Processes",
+          "start_index": 323,
+          "end_index": 324,
+          "nodes": [
+            {
+              "title": "Linear regression revisited",
+              "start_index": 324,
+              "end_index": 326,
+              "node_id": "0133"
+            },
+            {
+              "title": "Gaussian processes for regression",
+              "start_index": 326,
+              "end_index": 331,
+              "node_id": "0134"
+            },
+            {
+              "title": "Learning the hyperparameters",
+              "start_index": 331,
+              "end_index": 332,
+              "node_id": "0135"
+            },
+            {
+              "title": "Automatic relevance determination",
+              "start_index": 332,
+              "end_index": 333,
+              "node_id": "0136"
+            },
+            {
+              "title": "Gaussian processes for classification",
+              "start_index": 333,
+              "end_index": 335,
+              "node_id": "0137"
+            },
+            {
+              "title": "Laplace approximation",
+              "start_index": 335,
+              "end_index": 339,
+              "node_id": "0138"
+            },
+            {
+              "title": "Connection to neural networks",
+              "start_index": 339,
+              "end_index": 340,
+              "node_id": "0139"
+            }
+          ],
+          "node_id": "0132"
+        }
+      ],
+      "node_id": "0127"
+    },
+    {
+      "title": "Exercises",
+      "start_index": 340,
+      "end_index": 344,
+      "node_id": "0140"
+    },
+    {
+      "title": "Sparse Kernel Machines",
+      "start_index": 345,
+      "end_index": 346,
+      "nodes": [
+        {
+          "title": "Maximum Margin Classifiers",
+          "start_index": 346,
+          "end_index": 351,
+          "nodes": [
+            {
+              "title": "Overlapping class distributions",
+              "start_index": 351,
+              "end_index": 356,
+              "node_id": "0143"
+            },
+            {
+              "title": "Relation to logistic regression",
+              "start_index": 356,
+              "end_index": 358,
+              "node_id": "0144"
+            },
+            {
+              "title": "Multiclass SVMs",
+              "start_index": 358,
+              "end_index": 359,
+              "node_id": "0145"
+            },
+            {
+              "title": "SVMs for regression",
+              "start_index": 359,
+              "end_index": 364,
+              "node_id": "0146"
+            },
+            {
+              "title": "Computational learning theory",
+              "start_index": 364,
+              "end_index": 365,
+              "node_id": "0147"
+            }
+          ],
+          "node_id": "0142"
+        },
+        {
+          "title": "Relevance Vector Machines",
+          "start_index": 365,
+          "end_index": 365,
+          "nodes": [
+            {
+              "title": "RVM for regression",
+              "start_index": 365,
+              "end_index": 369,
+              "node_id": "0149"
+            },
+            {
+              "title": "Analysis of sparsity",
+              "start_index": 369,
+              "end_index": 373,
+              "node_id": "0150"
+            },
+            {
+              "title": "RVM for classification",
+              "start_index": 373,
+              "end_index": 377,
+              "node_id": "0151"
+            }
+          ],
+          "node_id": "0148"
+        }
+      ],
+      "node_id": "0141"
+    },
+    {
+      "title": "Exercises",
+      "start_index": 377,
+      "end_index": 379,
+      "node_id": "0152"
+    },
+    {
+      "title": "Graphical Models",
+      "start_index": 379,
+      "end_index": 380,
+      "nodes": [
+        {
+          "title": "Bayesian Networks",
+          "start_index": 380,
+          "end_index": 382,
+          "nodes": [
+            {
+              "title": "Example: Polynomial regression",
+              "start_index": 382,
+              "end_index": 385,
+              "node_id": "0155"
+            },
+            {
+              "title": "Generative models",
+              "start_index": 385,
+              "end_index": 386,
+              "node_id": "0156"
+            },
+            {
+              "title": "Discrete variables",
+              "start_index": 386,
+              "end_index": 390,
+              "node_id": "0157"
+            },
+            {
+              "title": "Linear-Gaussian models",
+              "start_index": 390,
+              "end_index": 392,
+              "node_id": "0158"
+            }
+          ],
+          "node_id": "0154"
+        },
+        {
+          "title": "Conditional Independence",
+          "start_index": 392,
+          "end_index": 393,
+          "nodes": [
+            {
+              "title": "Three example graphs",
+              "start_index": 393,
+              "end_index": 398,
+              "node_id": "0160"
+            },
+            {
+              "title": "D-separation",
+              "start_index": 398,
+              "end_index": 403,
+              "node_id": "0161"
+            }
+          ],
+          "node_id": "0159"
+        },
+        {
+          "title": "Markov Random Fields",
+          "start_index": 403,
+          "end_index": 403,
+          "nodes": [
+            {
+              "title": "Conditional independence properties",
+              "start_index": 403,
+              "end_index": 404,
+              "node_id": "0163"
+            },
+            {
+              "title": "Factorization properties",
+              "start_index": 404,
+              "end_index": 407,
+              "node_id": "0164"
+            },
+            {
+              "title": "Illustration: Image de-noising",
+              "start_index": 407,
+              "end_index": 410,
+              "node_id": "0165"
+            },
+            {
+              "title": "Relation to directed graphs",
+              "start_index": 410,
+              "end_index": 413,
+              "node_id": "0166"
+            }
+          ],
+          "node_id": "0162"
+        },
+        {
+          "title": "Inference in Graphical Models",
+          "start_index": 413,
+          "end_index": 414,
+          "nodes": [
+            {
+              "title": "Inference on a chain",
+              "start_index": 414,
+              "end_index": 418,
+              "node_id": "0168"
+            },
+            {
+              "title": "Trees",
+              "start_index": 418,
+              "end_index": 419,
+              "node_id": "0169"
+            },
+            {
+              "title": "Factor graphs",
+              "start_index": 419,
+              "end_index": 422,
+              "node_id": "0170"
+            },
+            {
+              "title": "The sum-product algorithm",
+              "start_index": 422,
+              "end_index": 431,
+              "node_id": "0171"
+            },
+            {
+              "title": "The max-sum algorithm",
+              "start_index": 431,
+              "end_index": 436,
+              "node_id": "0172"
+            },
+            {
+              "title": "Exact inference in general graphs",
+              "start_index": 436,
+              "end_index": 437,
+              "node_id": "0173"
+            },
+            {
+              "title": "Loopy belief propagation",
+              "start_index": 437,
+              "end_index": 438,
+              "node_id": "0174"
+            },
+            {
+              "title": "Learning the graph structure",
+              "start_index": 438,
+              "end_index": 438,
+              "node_id": "0175"
+            }
+          ],
+          "node_id": "0167"
+        }
+      ],
+      "node_id": "0153"
+    },
+    {
+      "title": "Exercises",
+      "start_index": 438,
+      "end_index": 443,
+      "node_id": "0176"
+    },
+    {
+      "title": "Mixture Models and EM",
+      "start_index": 443,
+      "end_index": 444,
+      "nodes": [
+        {
+          "title": "K-means Clustering",
+          "start_index": 444,
+          "end_index": 448,
+          "nodes": [
+            {
+              "title": "Image segmentation and compression",
+              "start_index": 448,
+              "end_index": 450,
+              "node_id": "0179"
+            }
+          ],
+          "node_id": "0178"
+        },
+        {
+          "title": "Mixtures of Gaussians",
+          "start_index": 450,
+          "end_index": 452,
+          "nodes": [
+            {
+              "title": "Maximum likelihood",
+              "start_index": 452,
+              "end_index": 455,
+              "node_id": "0181"
+            },
+            {
+              "title": "EM for Gaussian mixtures",
+              "start_index": 455,
+              "end_index": 459,
+              "node_id": "0182"
+            }
+          ],
+          "node_id": "0180"
+        },
+        {
+          "title": "An Alternative View of EM",
+          "start_index": 459,
+          "end_index": 461,
+          "nodes": [
+            {
+              "title": "Gaussian mixtures revisited",
+              "start_index": 461,
+              "end_index": 463,
+              "node_id": "0184"
+            },
+            {
+              "title": "Relation to K-means",
+              "start_index": 463,
+              "end_index": 464,
+              "node_id": "0185"
+            },
+            {
+              "title": "Mixtures of Bernoulli distributions",
+              "start_index": 464,
+              "end_index": 468,
+              "node_id": "0186"
+            },
+            {
+              "title": "EM for Bayesian linear regression",
+              "start_index": 468,
+              "end_index": 470,
+              "node_id": "0187"
+            }
+          ],
+          "node_id": "0183"
+        },
+        {
+          "title": "The EM Algorithm in General",
+          "start_index": 470,
+          "end_index": 475,
+          "node_id": "0188"
+        }
+      ],
+      "node_id": "0177"
+    },
+    {
+      "title": "Exercises",
+      "start_index": 475,
+      "end_index": 480,
+      "node_id": "0189"
+    },
+    {
+      "title": "Approximate Inference",
+      "start_index": 481,
+      "end_index": 482,
+      "nodes": [
+        {
+          "title": "Variational Inference",
+          "start_index": 482,
+          "end_index": 484,
+          "nodes": [
+            {
+              "title": "Factorized distributions",
+              "start_index": 484,
+              "end_index": 486,
+              "node_id": "0192"
+            },
+            {
+              "title": "Properties of factorized approximations",
+              "start_index": 486,
+              "end_index": 490,
+              "node_id": "0193"
+            },
+            {
+              "title": "Example: The univariate Gaussian",
+              "start_index": 490,
+              "end_index": 493,
+              "node_id": "0194"
+            },
+            {
+              "title": "Model comparison",
+              "start_index": 493,
+              "end_index": 494,
+              "node_id": "0195"
+            }
+          ],
+          "node_id": "0191"
+        },
+        {
+          "title": "Illustration: Variational Mixture of Gaussians",
+          "start_index": 494,
+          "end_index": 495,
+          "nodes": [
+            {
+              "title": "Variational distribution",
+              "start_index": 495,
+              "end_index": 501,
+              "node_id": "0197"
+            },
+            {
+              "title": "Variational lower bound",
+              "start_index": 501,
+              "end_index": 502,
+              "node_id": "0198"
+            },
+            {
+              "title": "Predictive density",
+              "start_index": 502,
+              "end_index": 503,
+              "node_id": "0199"
+            },
+            {
+              "title": "Determining the number of components",
+              "start_index": 503,
+              "end_index": 505,
+              "node_id": "0200"
+            },
+            {
+              "title": "Induced factorizations",
+              "start_index": 505,
+              "end_index": 506,
+              "node_id": "0201"
+            }
+          ],
+          "node_id": "0196"
+        },
+        {
+          "title": "Variational Linear Regression",
+          "start_index": 506,
+          "end_index": 506,
+          "nodes": [
+            {
+              "title": "Variational distribution",
+              "start_index": 506,
+              "end_index": 508,
+              "node_id": "0203"
+            },
+            {
+              "title": "Predictive distribution",
+              "start_index": 508,
+              "end_index": 509,
+              "node_id": "0204"
+            },
+            {
+              "title": "Lower bound",
+              "start_index": 509,
+              "end_index": 510,
+              "node_id": "0205"
+            }
+          ],
+          "node_id": "0202"
+        },
+        {
+          "title": "Exponential Family Distributions",
+          "start_index": 510,
+          "end_index": 511,
+          "nodes": [
+            {
+              "title": "Variational message passing",
+              "start_index": 511,
+              "end_index": 512,
+              "node_id": "0207"
+            }
+          ],
+          "node_id": "0206"
+        },
+        {
+          "title": "Local Variational Methods",
+          "start_index": 513,
+          "end_index": 518,
+          "node_id": "0208"
+        },
+        {
+          "title": "Variational Logistic Regression",
+          "start_index": 518,
+          "end_index": 518,
+          "nodes": [
+            {
+              "title": "Variational posterior distribution",
+              "start_index": 518,
+              "end_index": 520,
+              "node_id": "0210"
+            },
+            {
+              "title": "Optimizing the variational parameters",
+              "start_index": 520,
+              "end_index": 522,
+              "node_id": "0211"
+            },
+            {
+              "title": "Inference of hyperparameters",
+              "start_index": 522,
+              "end_index": 525,
+              "node_id": "0212"
+            }
+          ],
+          "node_id": "0209"
+        },
+        {
+          "title": "Expectation Propagation",
+          "start_index": 525,
+          "end_index": 531,
+          "nodes": [
+            {
+              "title": "Example: The clutter problem",
+              "start_index": 531,
+              "end_index": 533,
+              "node_id": "0214"
+            },
+            {
+              "title": "Expectation propagation on graphs",
+              "start_index": 533,
+              "end_index": 537,
+              "node_id": "0215"
+            }
+          ],
+          "node_id": "0213"
+        }
+      ],
+      "node_id": "0190"
+    },
+    {
+      "title": "Exercises",
+      "start_index": 537,
+      "end_index": 542,
+      "node_id": "0216"
+    },
+    {
+      "title": "Sampling Methods",
+      "start_index": 543,
+      "end_index": 546,
+      "nodes": [
+        {
+          "title": "Basic Sampling Algorithms",
+          "start_index": 546,
+          "end_index": 546,
+          "nodes": [
+            {
+              "title": "Standard distributions",
+              "start_index": 546,
+              "end_index": 548,
+              "node_id": "0219"
+            },
+            {
+              "title": "Rejection sampling",
+              "start_index": 548,
+              "end_index": 550,
+              "node_id": "0220"
+            },
+            {
+              "title": "Adaptive rejection sampling",
+              "start_index": 550,
+              "end_index": 552,
+              "node_id": "0221"
+            },
+            {
+              "title": "Importance sampling",
+              "start_index": 552,
+              "end_index": 554,
+              "node_id": "0222"
+            },
+            {
+              "title": "Sampling-importance-resampling",
+              "start_index": 554,
+              "end_index": 556,
+              "node_id": "0223"
+            },
+            {
+              "title": "Sampling and the EM algorithm",
+              "start_index": 556,
+              "end_index": 556,
+              "node_id": "0224"
+            }
+          ],
+          "node_id": "0218"
+        },
+        {
+          "title": "Markov Chain Monte Carlo",
+          "start_index": 557,
+          "end_index": 559,
+          "nodes": [
+            {
+              "title": "Markov chains",
+              "start_index": 559,
+              "end_index": 561,
+              "node_id": "0226"
+            },
+            {
+              "title": "The Metropolis-Hastings algorithm",
+              "start_index": 561,
+              "end_index": 562,
+              "node_id": "0227"
+            }
+          ],
+          "node_id": "0225"
+        },
+        {
+          "title": "Gibbs Sampling",
+          "start_index": 562,
+          "end_index": 566,
+          "node_id": "0228"
+        },
+        {
+          "title": "Slice Sampling",
+          "start_index": 566,
+          "end_index": 568,
+          "node_id": "0229"
+        },
+        {
+          "title": "The Hybrid Monte Carlo Algorithm",
+          "start_index": 568,
+          "end_index": 568,
+          "nodes": [
+            {
+              "title": "Dynamical systems",
+              "start_index": 568,
+              "end_index": 572,
+              "node_id": "0231"
+            },
+            {
+              "title": "Hybrid Monte Carlo",
+              "start_index": 572,
+              "end_index": 574,
+              "node_id": "0232"
+            }
+          ],
+          "node_id": "0230"
+        },
+        {
+          "title": "Estimating the Partition Function",
+          "start_index": 574,
+          "end_index": 576,
+          "node_id": "0233"
+        }
+      ],
+      "node_id": "0217"
+    },
+    {
+      "title": "Exercises",
+      "start_index": 576,
+      "end_index": 579,
+      "node_id": "0234"
+    },
+    {
+      "title": "Continuous Latent Variables",
+      "start_index": 579,
+      "end_index": 581,
+      "nodes": [
+        {
+          "title": "Principal Component Analysis",
+          "start_index": 581,
+          "end_index": 581,
+          "nodes": [
+            {
+              "title": "Maximum variance formulation",
+              "start_index": 581,
+              "end_index": 583,
+              "node_id": "0237"
+            },
+            {
+              "title": "Minimum-error formulation",
+              "start_index": 583,
+              "end_index": 585,
+              "node_id": "0238"
+            },
+            {
+              "title": "Applications of PCA",
+              "start_index": 585,
+              "end_index": 589,
+              "node_id": "0239"
+            },
+            {
+              "title": "PCA for high-dimensional data",
+              "start_index": 589,
+              "end_index": 590,
+              "node_id": "0240"
+            }
+          ],
+          "node_id": "0236"
+        },
+        {
+          "title": "Probabilistic PCA",
+          "start_index": 590,
+          "end_index": 594,
+          "nodes": [
+            {
+              "title": "Maximum likelihood PCA",
+              "start_index": 594,
+              "end_index": 597,
+              "node_id": "0242"
+            },
+            {
+              "title": "EM algorithm for PCA",
+              "start_index": 597,
+              "end_index": 600,
+              "node_id": "0243"
+            },
+            {
+              "title": "Bayesian PCA",
+              "start_index": 600,
+              "end_index": 603,
+              "node_id": "0244"
+            },
+            {
+              "title": "Factor analysis",
+              "start_index": 603,
+              "end_index": 606,
+              "node_id": "0245"
+            }
+          ],
+          "node_id": "0241"
+        },
+        {
+          "title": "Kernel PCA",
+          "start_index": 606,
+          "end_index": 610,
+          "node_id": "0246"
+        },
+        {
+          "title": "Nonlinear Latent Variable Models",
+          "start_index": 611,
+          "end_index": 611,
+          "nodes": [
+            {
+              "title": "Independent component analysis",
+              "start_index": 611,
+              "end_index": 612,
+              "node_id": "0248"
+            },
+            {
+              "title": "Autoassociative neural networks",
+              "start_index": 612,
+              "end_index": 615,
+              "node_id": "0249"
+            },
+            {
+              "title": "Modelling nonlinear manifolds",
+              "start_index": 615,
+              "end_index": 619,
+              "node_id": "0250"
+            }
+          ],
+          "node_id": "0247"
+        }
+      ],
+      "node_id": "0235"
+    },
+    {
+      "title": "Exercises",
+      "start_index": 619,
+      "end_index": 624,
+      "node_id": "0251"
+    },
+    {
+      "title": "Sequential Data",
+      "start_index": 625,
+      "end_index": 627,
+      "nodes": [
+        {
+          "title": "Markov Models",
+          "start_index": 627,
+          "end_index": 630,
+          "node_id": "0253"
+        },
+        {
+          "title": "Hidden Markov Models",
+          "start_index": 630,
+          "end_index": 635,
+          "nodes": [
+            {
+              "title": "Maximum likelihood for the HMM",
+              "start_index": 635,
+              "end_index": 638,
+              "node_id": "0255"
+            },
+            {
+              "title": "The forward-backward algorithm",
+              "start_index": 638,
+              "end_index": 645,
+              "node_id": "0256"
+            },
+            {
+              "title": "The sum-product algorithm for the HMM",
+              "start_index": 645,
+              "end_index": 647,
+              "node_id": "0257"
+            },
+            {
+              "title": "Scaling factors",
+              "start_index": 647,
+              "end_index": 649,
+              "node_id": "0258"
+            },
+            {
+              "title": "The Viterbi algorithm",
+              "start_index": 649,
+              "end_index": 651,
+              "node_id": "0259"
+            },
+            {
+              "title": "Extensions of the hidden Markov model",
+              "start_index": 651,
+              "end_index": 655,
+              "node_id": "0260"
+            }
+          ],
+          "node_id": "0254"
+        },
+        {
+          "title": "Linear Dynamical Systems",
+          "start_index": 655,
+          "end_index": 658,
+          "nodes": [
+            {
+              "title": "Inference in LDS",
+              "start_index": 658,
+              "end_index": 662,
+              "node_id": "0262"
+            },
+            {
+              "title": "Learning in LDS",
+              "start_index": 662,
+              "end_index": 664,
+              "node_id": "0263"
+            },
+            {
+              "title": "Extensions of LDS",
+              "start_index": 664,
+              "end_index": 665,
+              "node_id": "0264"
+            },
+            {
+              "title": "Particle filters",
+              "start_index": 665,
+              "end_index": 666,
+              "node_id": "0265"
+            }
+          ],
+          "node_id": "0261"
+        }
+      ],
+      "node_id": "0252"
+    },
+    {
+      "title": "Exercises",
+      "start_index": 666,
+      "end_index": 672,
+      "node_id": "0266"
+    },
+    {
+      "title": "Combining Models",
+      "start_index": 673,
+      "end_index": 674,
+      "nodes": [
+        {
+          "title": "Bayesian Model Averaging",
+          "start_index": 674,
+          "end_index": 675,
+          "node_id": "0268"
+        },
+        {
+          "title": "Committees",
+          "start_index": 675,
+          "end_index": 677,
+          "node_id": "0269"
+        },
+        {
+          "title": "Boosting",
+          "start_index": 677,
+          "end_index": 679,
+          "nodes": [
+            {
+              "title": "Minimizing exponential error",
+              "start_index": 679,
+              "end_index": 681,
+              "node_id": "0271"
+            },
+            {
+              "title": "Error functions for boosting",
+              "start_index": 681,
+              "end_index": 683,
+              "node_id": "0272"
+            }
+          ],
+          "node_id": "0270"
+        },
+        {
+          "title": "Tree-based Models",
+          "start_index": 683,
+          "end_index": 686,
+          "node_id": "0273"
+        },
+        {
+          "title": "Conditional Mixture Models",
+          "start_index": 686,
+          "end_index": 687,
+          "nodes": [
+            {
+              "title": "Mixtures of linear regression models",
+              "start_index": 687,
+              "end_index": 690,
+              "node_id": "0275"
+            },
+            {
+              "title": "Mixtures of logistic models",
+              "start_index": 690,
+              "end_index": 692,
+              "node_id": "0276"
+            },
+            {
+              "title": "Mixtures of experts",
+              "start_index": 692,
+              "end_index": 694,
+              "node_id": "0277"
+            }
+          ],
+          "node_id": "0274"
+        }
+      ],
+      "node_id": "0267"
+    },
+    {
+      "title": "Exercises",
+      "start_index": 694,
+      "end_index": 696,
+      "node_id": "0278"
+    },
+    {
+      "title": "Appendix A Data Sets",
+      "start_index": 697,
+      "end_index": 704,
+      "node_id": "0279"
+    },
+    {
+      "title": "Appendix B Probability Distributions",
+      "start_index": 705,
+      "end_index": 714,
+      "node_id": "0280"
+    },
+    {
+      "title": "Appendix C Properties of Matrices",
+      "start_index": 715,
+      "end_index": 722,
+      "node_id": "0281"
+    },
+    {
+      "title": "Appendix D Calculus of Variations",
+      "start_index": 723,
+      "end_index": 726,
+      "node_id": "0282"
+    },
+    {
+      "title": "Appendix E Lagrange Multipliers",
+      "start_index": 727,
+      "end_index": 730,
+      "node_id": "0283"
+    },
+    {
+      "title": "References",
+      "start_index": 731,
+      "end_index": 749,
+      "node_id": "0284"
+    },
+    {
+      "title": "Index",
+      "start_index": 749,
+      "end_index": 758,
+      "node_id": "0285"
+    }
+  ]
+}

tests/results/Regulation Best Interest_Interpretive release_structure.json

@@ -0,0 +1,73 @@
+{
+  "doc_name": "Regulation Best Interest_Interpretive release.pdf",
+  "doc_description": "A detailed analysis of the SEC's interpretation of the \"solely incidental\" prong of the broker-dealer exclusion under the Investment Advisers Act of 1940, including its historical context, application guidance, economic implications, and regulatory considerations.",
+  "structure": [
+    {
+      "title": "Preface",
+      "start_index": 1,
+      "end_index": 2,
+      "node_id": "0000",
+      "summary": "The partial document outlines an interpretation by the Securities and Exchange Commission (SEC) regarding the \"solely incidental\" prong of the broker-dealer exclusion under the Investment Advisers Act of 1940. It clarifies that brokers or dealers providing advisory services that are incidental to their primary business and for which they receive no special compensation are excluded from the definition of \"investment adviser\" under the Act. The document includes a historical and legislative context, the scope of the \"solely incidental\" prong, guidance on its application, and economic considerations related to the interpretation. It also provides contact information for further inquiries and specifies the effective date of the interpretation as July 12, 2019."
+    },
+    {
+      "title": "Introduction",
+      "start_index": 2,
+      "end_index": 6,
+      "node_id": "0001",
+      "summary": "The partial document discusses the regulation of investment advisers under the Advisers Act, specifically focusing on the \"broker-dealer exclusion,\" which exempts brokers and dealers from being classified as investment advisers under certain conditions. Key points include:\n\n1. **Introduction to the Advisers Act**: Overview of the regulation of investment advisers and the broker-dealer exclusion, which applies when advisory services are \"solely incidental\" to brokerage business and no special compensation is received.\n\n2. **Historical Context and Legislative History**: Examination of the historical practices of broker-dealers providing investment advice, distinguishing between auxiliary advice as part of brokerage services and separate advisory services.\n\n3. **Interpretation of the Solely Incidental Prong**: Clarification of the \"solely incidental\" condition of the broker-dealer exclusion, including its application to activities like investment discretion and account monitoring.\n\n4. **Economic Considerations**: Discussion of the potential economic effects of the interpretation and application of the broker-dealer exclusion.\n\n5. **Regulatory Developments**: Reference to the Commission's 2018 proposals, including Regulation Best Interest (Reg. BI), the Proposed Fiduciary Interpretation, and the Relationship Summary Proposal, aimed at enhancing standards of conduct and investor understanding.\n\n6. **Public Comments and Feedback**: Summary of public comments on the scope and interpretation of the broker-dealer exclusion, highlighting disagreements and requests for clarification on the \"solely incidental\" prong.\n\n7. **Adoption of Interpretation**: The Commission's adoption of an interpretation to confirm and clarify its position on the \"solely incidental\" prong, complementing related rules and forms to improve investor understanding of broker-dealer and adviser relationships."
+    },
+    {
+      "title": "Interpretation and Application",
+      "start_index": 6,
+      "end_index": 8,
+      "nodes": [
+        {
+          "title": "Historical Context and Legislative History",
+          "start_index": 8,
+          "end_index": 10,
+          "node_id": "0003",
+          "summary": "The partial document discusses the historical context and legislative development of the Investment Advisers Act of 1940. It highlights the findings of a congressional study conducted by the SEC between 1935 and 1939, which identified issues with distinguishing legitimate investment counselors from unregulated \"tipster\" organizations and problems in the organization and operation of investment counsel institutions. The document explains how these findings led to the passage of the Advisers Act, which broadly defined \"investment adviser\" and established regulatory oversight for those providing investment advice for compensation. It also addresses the exclusion of certain professionals, such as broker-dealers, from the definition of \"investment adviser\" if their advice is incidental to their primary business and not specially compensated. Additionally, the document explores the scope of the \"solely incidental\" prong of the broker-dealer exclusion, referencing interpretations and rules by the SEC, including a 2005 rule regarding fee-based brokerage accounts."
+        },
+        {
+          "title": "Scope of the Solely Incidental Prong of the Broker-Dealer Exclusion",
+          "start_index": 10,
+          "end_index": 14,
+          "node_id": "0004",
+          "summary": "The partial document discusses the \"broker-dealer exclusion\" under the Investment Advisers Act, specifically focusing on the \"solely incidental\" prong. It examines the scope of this exclusion, emphasizing that investment advice provided by broker-dealers is considered \"solely incidental\" if it is connected to and reasonably related to their primary business of effecting securities transactions. The document references historical interpretations, court rulings (e.g., Financial Planning Association v. SEC and Thomas v. Metropolitan Life Insurance Company), and legislative history to clarify this standard. It highlights that the frequency or importance of advice does not determine whether it meets the \"solely incidental\" standard, but rather its relationship to the broker-dealer's primary business. The document also provides guidance on applying this interpretation to specific practices, such as exercising investment discretion and account monitoring, noting that certain discretionary activities may fall outside the scope of the exclusion."
+        },
+        {
+          "title": "Guidance on Applying the Interpretation of the Solely Incidental Prong",
+          "start_index": 14,
+          "end_index": 22,
+          "node_id": "0005",
+          "summary": "The partial document provides guidance on the application of the \"solely incidental\" prong of the broker-dealer exclusion under the Advisers Act. It focuses on two key areas: (1) the exercise of investment discretion by broker-dealers over customer accounts and (2) account monitoring. The document discusses the Commission's interpretation that unlimited investment discretion is not \"solely incidental\" to a broker-dealer's business, as it indicates a primarily advisory relationship. However, temporary or limited discretion in specific scenarios (e.g., cash management, tax-loss sales, or margin requirements) may be consistent with the \"solely incidental\" prong. It also addresses account monitoring, stating that agreed-upon periodic monitoring for buy, sell, or hold recommendations may align with the broker-dealer exclusion, while continuous monitoring or advisory-like services would not. The document includes examples, refinements to prior interpretations, and considerations for broker-dealers to adopt policies ensuring compliance. It concludes with economic considerations, highlighting the potential impact on broker-dealers, customers, and the financial advice market."
+        }
+      ],
+      "node_id": "0002",
+      "summary": "The partial document discusses the historical context and legislative history of the Advisers Act of 1940, focusing on the roles of broker-dealers in providing investment advice. It highlights two distinct ways broker-dealers offered advice: as part of traditional brokerage services with fixed commissions and as separate advisory services for a fee. The document examines the concept of \"brokerage house advice,\" detailing the types of information and services provided, such as market analyses, tax information, and investment recommendations. It also references a congressional study conducted between 1935 and 1939, which identified issues with distinguishing legitimate investment counselors from \"tipster\" organizations and problems in the organization and operation of investment counsel institutions. These findings led to the enactment of the Advisers Act, which broadly defined \"investment adviser\" to regulate those providing investment advice for compensation. The document also references various reports, hearings, and literature that informed the development of the Act."
+    },
+    {
+      "title": "Economic Considerations",
+      "start_index": 22,
+      "end_index": 22,
+      "nodes": [
+        {
+          "title": "Background",
+          "start_index": 22,
+          "end_index": 23,
+          "node_id": "0007",
+          "summary": "The partial document discusses the U.S. Securities and Exchange Commission's (SEC) interpretation of the \"solely incidental\" prong of the broker-dealer exclusion, clarifying its understanding without creating new legal obligations. It examines the potential economic effects of this interpretation on broker-dealers, their associated persons, customers, and the broader financial advice market. The document provides background data on broker-dealers, including their assets, customer accounts, and dual registration as investment advisers. It highlights compliance costs for broker-dealers to align with the interpretation and notes the limited circumstances under which broker-dealers exercise temporary or limited investment discretion. The document also references the lack of data received during the Reg. BI Proposal to analyze the economic impact further."
+        },
+        {
+          "title": "Potential Economic Effects",
+          "start_index": 23,
+          "end_index": 28,
+          "node_id": "0008",
+          "summary": "The partial document discusses the economic effects and regulatory implications of the SEC's interpretation of the \"solely incidental\" prong of the broker-dealer exclusion from the definition of an investment adviser. Key points include:\n\n1. **Compliance Costs**: Broker-dealers currently incur costs to align their practices with the \"solely incidental\" prong, and the interpretation may lead to additional costs for evaluating and adjusting practices.\n\n2. **Impact on Broker-Dealer Practices**: Broker-dealers providing advisory services beyond the scope of the interpretation may need to adjust their practices, potentially resulting in reduced services, loss of customers, or a shift to advisory accounts.\n\n3. **Market Effects**: The interpretation could lead to decreased competition, increased fees, and a diminished number of broker-dealers offering commission-based services. It may also shift demand from broker-dealers to investment advisers.\n\n4. **Regulatory Adjustments**: Broker-dealers may choose to register as investment advisers, incurring new compliance costs, or migrate customers to advisory accounts of affiliates.\n\n5. **Potential Benefits**: Some broker-dealers may expand limited discretionary services or monitoring activities, benefiting investors with more efficient access to these services.\n\n6. **Regulatory Arbitrage Risks**: The interpretation raises concerns about regulatory arbitrage, though these risks may be mitigated by enhanced standards of conduct for broker-dealers.\n\n7. **Amendments to Regulations**: The document includes amendments to the Code of Federal Regulations, adding an interpretive release regarding the \"solely incidental\" prong, dated June 5, 2019."
+        }
+      ],
+      "node_id": "0006",
+      "summary": "The partial document discusses the SEC's interpretation of the \"solely incidental\" prong of the broker-dealer exclusion, clarifying that it does not impose new legal obligations but may have economic implications if broker-dealer practices deviate from this interpretation. It provides background on the potential effects on broker-dealers, their associated persons, customers, and the broader financial advice market. The document includes data on the number of registered broker-dealers, their customer accounts, total assets, and the prevalence of dual registrants (firms registered as both broker-dealers and investment advisers) as of December 2018."
+    }
+  ]
+}

tests/results/earthmover_structure.json

@@ -0,0 +1,137 @@
+{
+  "doc_name": "earthmover.pdf",
+  "structure": [
+    {
+      "title": "Earth Mover\u2019s Distance based Similarity Search at Scale",
+      "start_index": 1,
+      "end_index": 1,
+      "node_id": "0000"
+    },
+    {
+      "title": "ABSTRACT",
+      "start_index": 1,
+      "end_index": 1,
+      "node_id": "0001"
+    },
+    {
+      "title": "INTRODUCTION",
+      "start_index": 1,
+      "end_index": 2,
+      "node_id": "0002"
+    },
+    {
+      "title": "PRELIMINARIES",
+      "start_index": 2,
+      "end_index": 2,
+      "nodes": [
+        {
+          "title": "Computing the EMD",
+          "start_index": 3,
+          "end_index": 3,
+          "node_id": "0004"
+        },
+        {
+          "title": "Filter-and-Refinement Framework",
+          "start_index": 3,
+          "end_index": 4,
+          "node_id": "0005"
+        }
+      ],
+      "node_id": "0003"
+    },
+    {
+      "title": "SCALING UP SSP",
+      "start_index": 4,
+      "end_index": 5,
+      "node_id": "0006"
+    },
+    {
+      "title": "BOOSTING THE REFINEMENT PHASE",
+      "start_index": 5,
+      "end_index": 5,
+      "nodes": [
+        {
+          "title": "Analysis of EMD Calculation",
+          "start_index": 5,
+          "end_index": 6,
+          "node_id": "0008"
+        },
+        {
+          "title": "Progressive Bounding",
+          "start_index": 6,
+          "end_index": 6,
+          "node_id": "0009"
+        },
+        {
+          "title": "Sensitivity to Refinement Order",
+          "start_index": 6,
+          "end_index": 7,
+          "node_id": "0010"
+        },
+        {
+          "title": "Dynamic Refinement Ordering",
+          "start_index": 7,
+          "end_index": 8,
+          "node_id": "0011"
+        },
+        {
+          "title": "Running Upper Bound",
+          "start_index": 8,
+          "end_index": 8,
+          "node_id": "0012"
+        }
+      ],
+      "node_id": "0007"
+    },
+    {
+      "title": "EXPERIMENTAL EVALUATION",
+      "start_index": 8,
+      "end_index": 9,
+      "nodes": [
+        {
+          "title": "Performance Improvement",
+          "start_index": 9,
+          "end_index": 10,
+          "node_id": "0014"
+        },
+        {
+          "title": "Scalability Experiments",
+          "start_index": 10,
+          "end_index": 11,
+          "node_id": "0015"
+        },
+        {
+          "title": "Parameter Tuning in DRO",
+          "start_index": 11,
+          "end_index": 12,
+          "node_id": "0016"
+        }
+      ],
+      "node_id": "0013"
+    },
+    {
+      "title": "RELATED WORK",
+      "start_index": 12,
+      "end_index": 12,
+      "node_id": "0017"
+    },
+    {
+      "title": "CONCLUSION",
+      "start_index": 12,
+      "end_index": 12,
+      "node_id": "0018"
+    },
+    {
+      "title": "ACKNOWLEDGMENT",
+      "start_index": 12,
+      "end_index": 12,
+      "node_id": "0019"
+    },
+    {
+      "title": "REFERENCES",
+      "start_index": 12,
+      "end_index": 12,
+      "node_id": "0020"
+    }
+  ]
+}

tests/results/four-lectures_structure.json

@@ -0,0 +1,333 @@
+{
+  "doc_name": "four-lectures.pdf",
+  "structure": [
+    {
+      "title": "Preface",
+      "start_index": 1,
+      "end_index": 1,
+      "node_id": "0000"
+    },
+    {
+      "title": "ML at a Glance",
+      "start_index": 2,
+      "end_index": 2,
+      "nodes": [
+        {
+          "title": "An ML session",
+          "start_index": 2,
+          "end_index": 3,
+          "node_id": "0002"
+        },
+        {
+          "title": "Types and Values",
+          "start_index": 3,
+          "end_index": 4,
+          "node_id": "0003"
+        },
+        {
+          "title": "Recursive Functions",
+          "start_index": 4,
+          "end_index": 4,
+          "node_id": "0004"
+        },
+        {
+          "title": "Raising Exceptions",
+          "start_index": 4,
+          "end_index": 5,
+          "node_id": "0005"
+        },
+        {
+          "title": "Structures",
+          "start_index": 5,
+          "end_index": 6,
+          "node_id": "0006"
+        },
+        {
+          "title": "Signatures",
+          "start_index": 6,
+          "end_index": 7,
+          "node_id": "0007"
+        },
+        {
+          "title": "Coercive Signature Matching",
+          "start_index": 7,
+          "end_index": 8,
+          "node_id": "0008"
+        },
+        {
+          "title": "Functor Declaration",
+          "start_index": 8,
+          "end_index": 9,
+          "node_id": "0009"
+        },
+        {
+          "title": "Functor Application",
+          "start_index": 9,
+          "end_index": 9,
+          "node_id": "0010"
+        },
+        {
+          "title": "Summary",
+          "start_index": 9,
+          "end_index": 9,
+          "node_id": "0011"
+        }
+      ],
+      "node_id": "0001"
+    },
+    {
+      "title": "Programming with ML Modules",
+      "start_index": 10,
+      "end_index": 10,
+      "nodes": [
+        {
+          "title": "Introduction",
+          "start_index": 10,
+          "end_index": 11,
+          "node_id": "0013"
+        },
+        {
+          "title": "Signatures",
+          "start_index": 11,
+          "end_index": 12,
+          "node_id": "0014"
+        },
+        {
+          "title": "Structures",
+          "start_index": 12,
+          "end_index": 13,
+          "node_id": "0015"
+        },
+        {
+          "title": "Functors",
+          "start_index": 13,
+          "end_index": 14,
+          "node_id": "0016"
+        },
+        {
+          "title": "Substructures",
+          "start_index": 14,
+          "end_index": 15,
+          "node_id": "0017"
+        },
+        {
+          "title": "Sharing",
+          "start_index": 15,
+          "end_index": 16,
+          "node_id": "0018"
+        },
+        {
+          "title": "Building the System",
+          "start_index": 16,
+          "end_index": 17,
+          "node_id": "0019"
+        },
+        {
+          "title": "Separate Compilation",
+          "start_index": 17,
+          "end_index": 18,
+          "node_id": "0020"
+        },
+        {
+          "title": "Good Style",
+          "start_index": 18,
+          "end_index": 18,
+          "node_id": "0021"
+        },
+        {
+          "title": "Bad Style",
+          "start_index": 18,
+          "end_index": 19,
+          "node_id": "0022"
+        }
+      ],
+      "node_id": "0012"
+    },
+    {
+      "title": "The Static Semantics of Modules",
+      "start_index": 20,
+      "end_index": 20,
+      "nodes": [
+        {
+          "title": "Elaboration",
+          "start_index": 20,
+          "end_index": 21,
+          "node_id": "0024"
+        },
+        {
+          "title": "Names",
+          "start_index": 21,
+          "end_index": 21,
+          "node_id": "0025"
+        },
+        {
+          "title": "Decorating Structures",
+          "start_index": 21,
+          "end_index": 21,
+          "node_id": "0026"
+        },
+        {
+          "title": "Decorating Signatures",
+          "start_index": 22,
+          "end_index": 23,
+          "node_id": "0027"
+        },
+        {
+          "title": "Signature Instantiation",
+          "start_index": 23,
+          "end_index": 24,
+          "node_id": "0028"
+        },
+        {
+          "title": "Signature Matching",
+          "start_index": 24,
+          "end_index": 25,
+          "node_id": "0029"
+        },
+        {
+          "title": "Signature Constraints",
+          "start_index": 25,
+          "end_index": 25,
+          "node_id": "0030"
+        },
+        {
+          "title": "Decorating Functors",
+          "start_index": 26,
+          "end_index": 26,
+          "node_id": "0031"
+        },
+        {
+          "title": "External Sharing",
+          "start_index": 26,
+          "end_index": 27,
+          "node_id": "0032"
+        },
+        {
+          "title": "Functors with Arguments",
+          "start_index": 27,
+          "end_index": 28,
+          "node_id": "0033"
+        },
+        {
+          "title": "Sharing Between Argument and Result",
+          "start_index": 28,
+          "end_index": 28,
+          "node_id": "0034"
+        },
+        {
+          "title": "Explicit Result Signatures",
+          "start_index": 28,
+          "end_index": 29,
+          "node_id": "0035"
+        }
+      ],
+      "node_id": "0023"
+    },
+    {
+      "title": "Implementing an Interpreter in ML",
+      "start_index": 30,
+      "end_index": 32,
+      "nodes": [
+        {
+          "title": "Version 1: The Bare Typechecker",
+          "start_index": 32,
+          "end_index": 33,
+          "node_id": "0037"
+        },
+        {
+          "title": "Version 2: Adding Lists and Polymorphism",
+          "start_index": 33,
+          "end_index": 37,
+          "node_id": "0038"
+        },
+        {
+          "title": "Version 3: A Different Implementation of Types",
+          "start_index": 37,
+          "end_index": 39,
+          "node_id": "0039"
+        },
+        {
+          "title": "Version 4: Introducing Variables and Let",
+          "start_index": 39,
+          "end_index": 43,
+          "node_id": "0040"
+        },
+        {
+          "title": "Acknowledgement",
+          "start_index": 43,
+          "end_index": 43,
+          "node_id": "0041"
+        }
+      ],
+      "node_id": "0036"
+    },
+    {
+      "title": "Appendix A: The Bare Interpreter",
+      "start_index": 44,
+      "end_index": 44,
+      "nodes": [
+        {
+          "title": "Syntax",
+          "start_index": 44,
+          "end_index": 44,
+          "node_id": "0043"
+        },
+        {
+          "title": "Parsing",
+          "start_index": 44,
+          "end_index": 45,
+          "node_id": "0044"
+        },
+        {
+          "title": "Environments",
+          "start_index": 45,
+          "end_index": 45,
+          "node_id": "0045"
+        },
+        {
+          "title": "Evaluation",
+          "start_index": 45,
+          "end_index": 46,
+          "node_id": "0046"
+        },
+        {
+          "title": "Type Checking",
+          "start_index": 46,
+          "end_index": 46,
+          "node_id": "0047"
+        },
+        {
+          "title": "The Interpreter",
+          "start_index": 46,
+          "end_index": 47,
+          "node_id": "0048"
+        },
+        {
+          "title": "The Evaluator",
+          "start_index": 47,
+          "end_index": 48,
+          "node_id": "0049"
+        },
+        {
+          "title": "The Typechecker",
+          "start_index": 48,
+          "end_index": 49,
+          "node_id": "0050"
+        },
+        {
+          "title": "The Basics",
+          "start_index": 50,
+          "end_index": 52,
+          "node_id": "0051"
+        }
+      ],
+      "node_id": "0042"
+    },
+    {
+      "title": "Appendix B: Files",
+      "start_index": 53,
+      "end_index": 53,
+      "node_id": "0052"
+    }
+  ]
+}

tests/results/q1-fy25-earnings_structure.json

@@ -0,0 +1,311 @@
+{
+  "doc_name": "q1-fy25-earnings.pdf",
+  "doc_description": "A comprehensive financial report detailing The Walt Disney Company's first-quarter fiscal 2025 performance, including revenue growth, segment highlights, guidance for fiscal 2025, and key financial metrics such as adjusted EPS, operating income, and cash flow.",
+  "structure": [
+    {
+      "title": "THE WALT DISNEY COMPANY REPORTS FIRST QUARTER EARNINGS FOR FISCAL 2025",
+      "start_index": 1,
+      "end_index": 1,
+      "nodes": [
+        {
+          "title": "Financial Results for the Quarter",
+          "start_index": 1,
+          "end_index": 1,
+          "nodes": [
+            {
+              "title": "Key Points",
+              "start_index": 1,
+              "end_index": 1,
+              "node_id": "0002",
+              "summary": "The partial document outlines The Walt Disney Company's financial performance for the first fiscal quarter of 2025, ending December 28, 2024. Key points include:\n\n1. **Financial Results**: \n   - Revenue increased by 5% to $24.7 billion.\n   - Income before taxes rose by 27% to $3.7 billion.\n   - Diluted EPS grew by 35% to $1.40.\n   - Total segment operating income increased by 31% to $5.1 billion, with adjusted EPS up 44% to $1.76.\n\n2. **Entertainment Segment**:\n   - Operating income increased by $0.8 billion to $1.7 billion.\n   - Direct-to-Consumer operating income rose by $431 million to $293 million, with advertising revenue (excluding Disney+ Hotstar in India) up 16%.\n   - Disney+ and Hulu subscriptions increased by 0.9 million, while Disney+ subscribers decreased by 0.7 million.\n   - Content sales/licensing income grew by $536 million, driven by the success of *Moana 2*.\n\n3. **Sports Segment**:\n   - Operating income increased by $350 million to $247 million.\n   - Domestic ESPN advertising revenue grew by 15%.\n\n4. **Experiences Segment**:\n   - Operating income remained at $3.1 billion, with a 6 percentage-point adverse impact due to Hurricanes Milton and Helene and pre-opening expenses for the Disney Treasure.\n   - Domestic Parks & Experiences income declined by 5%, while International Parks & Experiences income increased by 28%."
+            }
+          ],
+          "node_id": "0001",
+          "summary": "The partial document is a report from The Walt Disney Company detailing its financial performance for the first fiscal quarter of 2025, ending December 28, 2024. Key points include:\n\n1. **Financial Performance**:\n   - Revenue increased by 5% to $24.7 billion.\n   - Income before taxes rose by 27% to $3.7 billion.\n   - Diluted EPS grew by 35% to $1.40.\n   - Total segment operating income increased by 31% to $5.1 billion, with adjusted EPS up 44% to $1.76.\n\n2. **Segment Highlights**:\n   - **Entertainment**: Operating income increased by $0.8 billion to $1.7 billion. Direct-to-Consumer income rose by $431 million, though advertising revenue declined 2% (up 16% excluding Disney+ Hotstar in India). Disney+ and Hulu subscriptions increased slightly, while Disney+ subscribers decreased by 0.7 million. Content sales/licensing income grew, driven by the success of *Moana 2*.\n   - **Sports**: Operating income increased by $350 million to $247 million, with ESPN domestic advertising revenue up 15%.\n   - **Experiences**: Operating income remained at $3.1 billion, with adverse impacts from hurricanes and pre-opening expenses for the Disney Treasure. Domestic Parks & Experiences income declined by 5%, while International Parks & Experiences income rose by 28%.\n\n3. **Additional Notes**:\n   - Non-GAAP financial measures are used for certain metrics.\n   - Disney+ Hotstar in India saw a significant decline in advertising revenue compared to the previous year."
+        },
+        {
+          "title": "Guidance and Outlook",
+          "start_index": 2,
+          "end_index": 2,
+          "nodes": [
+            {
+              "title": "Star India deconsolidated in Q1",
+              "start_index": 2,
+              "end_index": 2,
+              "node_id": "0004",
+              "summary": "The partial document outlines Disney's financial guidance and outlook for fiscal 2025, including the deconsolidation of Star India and its impact on operating income for the Entertainment and Sports segments. It highlights expectations for Q2 fiscal 2025, such as a modest decline in Disney+ subscribers, adverse impacts on Sports segment income, and pre-opening expenses for Disney Cruise Line. For fiscal 2025, the company projects high-single-digit adjusted EPS growth, $15 billion in cash from operations, and segment operating income growth across Entertainment, Sports, and Experiences. The CEO emphasizes strong Q1 results, including box office success, improved profitability in streaming, advancements in ESPN\u2019s digital strategy, and continued investments in the Experiences segment, expressing confidence in Disney's growth strategy."
+            },
+            {
+              "title": "Q2 Fiscal 2025",
+              "start_index": 2,
+              "end_index": 2,
+              "node_id": "0005",
+              "summary": "The partial document outlines Disney's financial guidance and outlook for fiscal 2025, including the deconsolidation of Star India and its impact on operating income for the Entertainment and Sports segments. It highlights expectations for Q2 fiscal 2025, such as a modest decline in Disney+ subscribers, adverse impacts on Sports segment income, and pre-opening expenses for Disney Cruise Line. For the full fiscal year 2025, it projects high-single-digit adjusted EPS growth, $15 billion in cash from operations, and segment operating income growth across Entertainment, Sports, and Experiences. The CEO emphasizes Disney's strong start to the fiscal year, citing achievements in box office performance, improved streaming profitability, ESPN's digital strategy, and the enduring appeal of the Experiences segment."
+            },
+            {
+              "title": "Fiscal Year 2025",
+              "start_index": 2,
+              "end_index": 2,
+              "node_id": "0006",
+              "summary": "The partial document outlines Disney's financial guidance and outlook for fiscal 2025, including the deconsolidation of Star India and its impact on operating income for the Entertainment and Sports segments. It highlights expectations for Q2 fiscal 2025, such as a modest decline in Disney+ subscribers, adverse impacts on Sports segment income, and pre-opening expenses for Disney Cruise Line. For the full fiscal year 2025, it projects high-single-digit adjusted EPS growth, $15 billion in cash from operations, and segment operating income growth across Entertainment, Sports, and Experiences. The CEO emphasizes Disney's creative and financial strength, strong box office performance, improved streaming profitability, advancements in ESPN's digital strategy, and continued global investments in the Experiences segment."
+            }
+          ],
+          "node_id": "0003",
+          "summary": "The partial document outlines Disney's financial guidance and outlook for fiscal 2025, including the deconsolidation of Star India and its impact on operating income for the Entertainment and Sports segments. It highlights expectations for Q2 fiscal 2025, such as a modest decline in Disney+ subscribers, adverse impacts on Sports segment income, and pre-opening expenses for Disney Cruise Line. For the full fiscal year 2025, it projects high-single-digit adjusted EPS growth, $15 billion in cash from operations, and segment operating income growth across Entertainment, Sports, and Experiences. The CEO emphasizes strong Q1 results, including box office success, improved profitability in streaming, advancements in ESPN\u2019s digital strategy, and continued investment in global experiences."
+        },
+        {
+          "title": "Message From Our CEO",
+          "start_index": 2,
+          "end_index": 2,
+          "node_id": "0007",
+          "summary": "The partial document outlines Disney's financial guidance and outlook for fiscal 2025, including the deconsolidation of Star India and its impact on operating income for the Entertainment and Sports segments. It highlights expectations for Q2 fiscal 2025, such as a modest decline in Disney+ subscribers, adverse impacts on Sports segment income, and pre-opening expenses for Disney Cruise Line. For the full fiscal year 2025, it projects high-single-digit adjusted EPS growth, $15 billion in cash from operations, and segment operating income growth across Entertainment, Sports, and Experiences. The CEO emphasizes strong Q1 results, including box office success, improved profitability in streaming, advancements in ESPN\u2019s digital strategy, and continued investment in global experiences."
+        }
+      ],
+      "node_id": "0000",
+      "summary": "The partial document is a report from The Walt Disney Company detailing its financial performance for the first fiscal quarter of 2025, ending December 28, 2024. Key points include:\n\n1. **Financial Results**:  \n   - Revenue increased by 5% to $24.7 billion.  \n   - Income before taxes rose by 27% to $3.7 billion.  \n   - Diluted EPS grew by 35% to $1.40.  \n   - Total segment operating income increased by 31% to $5.1 billion, and adjusted EPS rose by 44% to $1.76.  \n\n2. **Entertainment Segment**:  \n   - Operating income increased by $0.8 billion to $1.7 billion.  \n   - Direct-to-Consumer operating income rose by $431 million to $293 million, with advertising revenue up 16% (excluding Disney+ Hotstar in India).  \n   - Disney+ and Hulu subscriptions increased by 0.9 million, while Disney+ subscribers decreased by 0.7 million.  \n   - Content sales/licensing income grew by $536 million, driven by the success of *Moana 2*.  \n\n3. **Sports Segment**:  \n   - Operating income increased by $350 million to $247 million.  \n   - Domestic ESPN advertising revenue grew by 15%.  \n\n4. **Experiences Segment**:  \n   - Operating income remained at $3.1 billion, with a 6 percentage-point adverse impact due to Hurricanes Milton and Helene and pre-opening expenses for the Disney Treasure.  \n   - Domestic Parks & Experiences income declined by 5%, while International Parks & Experiences income increased by 28%.  \n\nThe report also includes non-GAAP financial measures and notes the impact of Disney+ Hotstar's advertising revenue in India."
+    },
+    {
+      "title": "SUMMARIZED FINANCIAL RESULTS",
+      "start_index": 3,
+      "end_index": 3,
+      "nodes": [
+        {
+          "title": "SUMMARIZED SEGMENT FINANCIAL RESULTS",
+          "start_index": 3,
+          "end_index": 3,
+          "node_id": "0009",
+          "summary": "The partial document provides a summarized overview of financial results for the first quarter of fiscal years 2025 and 2024. Key points include:\n\n1. **Overall Financial Performance**:\n   - Revenues increased by 5% from $23,549 million in 2024 to $24,690 million in 2025.\n   - Income before income taxes rose by 27%.\n   - Total segment operating income grew by 31%.\n   - Diluted EPS increased by 35%, and diluted EPS excluding certain items rose by 44%.\n   - Cash provided by operations increased by 47%, while free cash flow decreased by 17%.\n\n2. **Segment Financial Results**:\n   - Revenue growth was observed in the Entertainment segment (9%) and Experiences segment (3%), while Sports revenue remained flat.\n   - Segment operating income for Entertainment increased significantly by 95%, while Sports shifted from a loss to a positive income. Experiences segment operating income remained stable.\n\n3. **Non-GAAP Measures**:\n   - The document highlights the use of non-GAAP financial measures such as total segment operating income, diluted EPS excluding certain items, and free cash flow, with references to further details and reconciliations provided elsewhere in the report."
+        }
+      ],
+      "node_id": "0008",
+      "summary": "The partial document provides a summarized overview of financial results for the first quarter of fiscal years 2025 and 2024. Key points include:\n\n1. **Overall Financial Performance**:\n   - Revenues increased by 5% from $23,549 million in 2024 to $24,690 million in 2025.\n   - Income before income taxes rose by 27%.\n   - Total segment operating income grew by 31%.\n   - Diluted EPS increased by 35%, and diluted EPS excluding certain items rose by 44%.\n   - Cash provided by operations increased by 47%, while free cash flow decreased by 17%.\n\n2. **Segment Financial Results**:\n   - Revenue growth was observed in the Entertainment segment (9%) and Experiences segment (3%), while Sports revenue remained flat.\n   - Segment operating income for Entertainment increased significantly by 95%, while Sports shifted from a loss to a positive income. Experiences segment operating income remained stable.\n\n3. **Non-GAAP Measures**:\n   - The document highlights the use of non-GAAP financial measures such as total segment operating income, diluted EPS excluding certain items, and free cash flow, with references to further details and reconciliations provided in later sections."
+    },
+    {
+      "title": "DISCUSSION OF FIRST QUARTER SEGMENT RESULTS",
+      "start_index": 4,
+      "end_index": 4,
+      "nodes": [
+        {
+          "title": "Star India",
+          "start_index": 4,
+          "end_index": 4,
+          "node_id": "0011",
+          "summary": "The partial document discusses the first-quarter segment results, focusing on the Star India joint venture formed between the Company and Reliance Industries Limited (RIL) on November 14, 2024. The joint venture combines Star-branded entertainment and sports television channels, Disney+ Hotstar, and certain RIL-controlled media businesses, with RIL holding a 56% controlling interest, the Company holding 37%, and a third-party investment company holding 7%. The Company now recognizes its 37% share of the joint venture\u2019s results under \"Equity in the income of investees.\" Additionally, the document provides financial results for the Entertainment segment, showing a 9% increase in total revenues and a 95% increase in operating income compared to the prior-year quarter. The growth in operating income is attributed to improved results in Content Sales/Licensing and Direct-to-Consumer, partially offset by a decline in Linear Networks."
+        },
+        {
+          "title": "Entertainment",
+          "start_index": 4,
+          "end_index": 4,
+          "nodes": [
+            {
+              "title": "Linear Networks",
+              "start_index": 5,
+              "end_index": 5,
+              "node_id": "0013",
+              "summary": "The partial document provides financial performance details for Linear Networks and Direct-to-Consumer segments for the quarters ending December 28, 2024, and December 30, 2023. Key points include:\n\n1. **Linear Networks**:\n   - Revenue decreased by 7%, with domestic revenue remaining flat and international revenue declining by 31%.\n   - Operating income decreased by 11%, with domestic income stable and international income dropping by 39%.\n   - Domestic operating income was impacted by higher programming costs (due to the 2023 guild strikes), lower affiliate revenue (fewer subscribers), lower technology costs, and higher advertising revenue (driven by political advertising but offset by lower viewership).\n   - International operating income decline was attributed to the Star India Transaction.\n   - Equity income from investees decreased due to lower income from A+E Television Networks, reduced advertising and affiliate revenue, and the absence of a prior-year gain from an investment sale.\n\n2. **Direct-to-Consumer**:\n   - Revenue increased by 9%, driven by higher subscription revenue due to increased pricing and more subscribers, partially offset by unfavorable foreign exchange impacts.\n   - Operating income improved significantly, moving from a loss in the prior year to a profit, reflecting subscription revenue growth."
+            },
+            {
+              "title": "Direct-to-Consumer",
+              "start_index": 5,
+              "end_index": 7,
+              "node_id": "0014",
+              "summary": "The partial document provides a financial performance overview of various segments for the quarter ended December 28, 2024, compared to the prior-year quarter. Key points include:\n\n1. **Linear Networks**:\n   - Revenue decreased by 7%, with domestic revenue flat and international revenue down 31%.\n   - Operating income decreased by 11%, with domestic income flat and international income down 39%, primarily due to the Star India transaction.\n   - Equity income from investees declined by 29%, driven by lower income from A+E Television Networks and the absence of a prior-year gain on an investment sale.\n\n2. **Direct-to-Consumer (DTC)**:\n   - Revenue increased by 9%, and operating income improved significantly from a loss of $138 million to a profit of $293 million.\n   - Growth was driven by higher subscription revenue due to pricing increases and more subscribers, partially offset by higher costs and lower advertising revenue.\n   - Key metrics showed slight changes in Disney+ and Hulu subscriber numbers, with increases in average monthly revenue per paid subscriber due to pricing adjustments.\n\n3. **Content Sales/Licensing and Other**:\n   - Revenue increased by 34%, and operating income improved significantly, driven by strong theatrical performance, particularly from \"Moana 2,\" and contributions from \"Mufasa: The Lion King.\"\n\n4. **Sports**:\n   - ESPN revenue grew by 8%, with domestic and international segments showing increases, while Star India revenue dropped by 90%.\n   - Operating income for ESPN improved by 15%, while Star India shifted from a loss to a small profit.\n\nThe document highlights revenue trends, operating income changes, and key drivers for each segment, including programming costs, subscriber growth, pricing adjustments, and content performance."
+            },
+            {
+              "title": "Content Sales/Licensing and Other",
+              "start_index": 7,
+              "end_index": 7,
+              "node_id": "0015",
+              "summary": "The partial document discusses the financial performance of Disney's streaming services, content sales, and sports segment. Key points include:\n\n1. **Disney+ Revenue**: Domestic and international Disney+ average monthly revenue per paid subscriber increased due to pricing hikes, partially offset by promotional offerings. International revenue also benefited from higher advertising revenue.\n\n2. **Hulu Revenue**: Hulu SVOD Only revenue remained stable, with pricing increases offsetting lower advertising revenue. Hulu Live TV + SVOD revenue increased due to pricing hikes.\n\n3. **Content Sales/Licensing**: Revenue and operating income improved significantly, driven by strong theatrical distribution results, particularly from \"Moana 2,\" and contributions from \"Mufasa: The Lion King.\"\n\n4. **Sports Revenue**: ESPN domestic and international revenues grew, while Star India revenue declined sharply. Operating income for ESPN improved, with domestic income slightly down and international losses reduced. Star India showed a notable recovery in operating income."
+            }
+          ],
+          "node_id": "0012",
+          "summary": "The partial document discusses the first-quarter segment results, focusing on the Star India joint venture formed between the Company and Reliance Industries Limited (RIL) on November 14, 2024. The joint venture combines Star-branded entertainment and sports television channels and the Disney+ Hotstar service in India, with RIL holding a 56% controlling interest, the Company holding 37%, and a third-party investment company holding 7%. The Company now recognizes its 37% share of the joint venture\u2019s results under \u201cEquity in the income of investees.\u201d Additionally, the document provides financial results for the Entertainment segment, showing a 9% increase in total revenues compared to the prior year, driven by growth in Direct-to-Consumer and Content Sales/Licensing and Other, despite a decline in Linear Networks. Operating income increased by 95%, primarily due to improved results in Content Sales/Licensing and Other and Direct-to-Consumer, partially offset by a decrease in Linear Networks."
+        },
+        {
+          "title": "Sports",
+          "start_index": 7,
+          "end_index": 7,
+          "nodes": [
+            {
+              "title": "Domestic ESPN",
+              "start_index": 8,
+              "end_index": 8,
+              "node_id": "0017",
+              "summary": "The partial document discusses the financial performance of ESPN, including domestic and international operations, as well as Star India, for the current quarter compared to the prior-year quarter. Key points include:\n\n1. **Domestic ESPN**: \n   - Decrease in operating results due to higher programming and production costs, primarily from expanded college football programming rights and changes in the College Football Playoff (CFP) format.\n   - Increase in advertising revenue due to higher rates.\n   - Revenue from sub-licensing CFP programming rights.\n   - Affiliate revenue remained comparable, with rate increases offset by fewer subscribers.\n\n2. **International ESPN**: \n   - Decrease in operating loss driven by higher fees from the Entertainment segment for Disney+ sports content.\n   - Increased programming and production costs due to higher soccer rights costs.\n   - Lower affiliate revenue due to fewer subscribers.\n\n3. **Star India**: \n   - Improved operating results due to the absence of significant cricket events in the current quarter compared to the prior-year quarter, which included the ICC Cricket World Cup.\n\n4. **Key Metrics for ESPN+**:\n   - Paid subscribers decreased from 25.6 million to 24.9 million.\n   - Average monthly revenue per paid subscriber increased from $5.94 to $6.36, driven by pricing increases and higher advertising revenue."
+            },
+            {
+              "title": "International ESPN",
+              "start_index": 8,
+              "end_index": 8,
+              "node_id": "0018",
+              "summary": "The partial document discusses the financial performance of ESPN, including domestic and international operations, as well as Star India, for the current quarter compared to the prior-year quarter. Key points include:\n\n1. **Domestic ESPN**: \n   - Decrease in operating results due to higher programming and production costs, primarily from expanded college football programming rights and changes in the College Football Playoff (CFP) format.\n   - Increase in advertising revenue due to higher rates.\n   - Revenue from sub-licensing CFP programming rights.\n   - Affiliate revenue remained comparable, with rate increases offset by fewer subscribers.\n\n2. **International ESPN**: \n   - Decrease in operating loss driven by higher fees from the Entertainment segment for Disney+ sports content.\n   - Increased programming and production costs due to higher soccer rights costs.\n   - Lower affiliate revenue due to fewer subscribers.\n\n3. **Star India**: \n   - Improved operating results due to the absence of significant cricket events in the current quarter compared to the ICC Cricket World Cup in the prior-year quarter.\n\n4. **Key Metrics for ESPN+**:\n   - Paid subscribers decreased from 25.6 million to 24.9 million.\n   - Average monthly revenue per paid subscriber increased from $5.94 to $6.36, driven by pricing increases and higher advertising revenue."
+            },
+            {
+              "title": "Star India",
+              "start_index": 8,
+              "end_index": 8,
+              "node_id": "0019",
+              "summary": "The partial document discusses the financial performance of ESPN, including domestic and international operations, as well as Star India, for a specific quarter. Key points include:\n\n1. **Domestic ESPN**: \n   - Decrease in operating results due to higher programming and production costs, primarily from expanded college football programming rights, including additional College Football Playoff (CFP) games under a revised format.\n   - Increase in advertising revenue due to higher rates.\n   - Revenue from sub-licensing CFP programming rights.\n   - Affiliate revenue remained comparable to the prior year due to effective rate increases offset by fewer subscribers.\n\n2. **International ESPN**: \n   - Decrease in operating loss driven by higher fees from the Entertainment segment for sports content on Disney+.\n   - Increased programming and production costs due to higher soccer rights costs.\n   - Lower affiliate revenue due to fewer subscribers.\n\n3. **Star India**: \n   - Improvement in operating results due to the absence of significant cricket events in the current quarter compared to the prior year, which included the ICC Cricket World Cup.\n\n4. **Key Metrics for ESPN+**:\n   - Paid subscribers decreased from 25.6 million to 24.9 million.\n   - Average monthly revenue per paid subscriber increased from $5.94 to $6.36, driven by pricing increases and higher advertising revenue."
+            }
+          ],
+          "node_id": "0016",
+          "summary": "The partial document discusses the financial performance of Disney's streaming services, content sales, and sports segment. Key points include:\n\n1. **Disney+ Revenue**: Domestic and international Disney+ average monthly revenue per paid subscriber increased due to pricing hikes, partially offset by promotional offerings. International revenue also benefited from higher advertising revenue.\n\n2. **Hulu Revenue**: Hulu SVOD Only revenue remained stable, with pricing increases offsetting lower advertising revenue. Hulu Live TV + SVOD revenue increased due to pricing hikes.\n\n3. **Content Sales/Licensing**: Revenue and operating income improved significantly, driven by strong theatrical performance, particularly from \"Moana 2,\" and contributions from \"Mufasa: The Lion King.\"\n\n4. **Sports Revenue**: ESPN domestic and international revenues grew, while Star India revenue declined sharply. Operating income for ESPN improved, with domestic income slightly down and international income showing significant recovery. Star India showed a notable turnaround in operating income."
+        },
+        {
+          "title": "Experiences",
+          "start_index": 9,
+          "end_index": 9,
+          "node_id": "0020",
+          "summary": "The partial document provides financial performance details for the Parks & Experiences segment, including revenues and operating income for domestic and international operations, as well as consumer products. It highlights a 3% increase in total revenue and stable operating income compared to the prior year. Domestic parks and experiences were negatively impacted by hurricanes, leading to lower volumes and higher costs, despite increased guest spending. International parks and experiences saw growth in operating income due to higher guest spending, increased attendance, and new offerings. The document also notes increased corporate expenses due to a legal settlement and a $143 million loss related to the Star India Transaction."
+        }
+      ],
+      "node_id": "0010",
+      "summary": "The partial document discusses the first-quarter segment results, focusing on the Star India joint venture formed between the Company and Reliance Industries Limited (RIL) on November 14, 2024. The joint venture combines Star-branded entertainment and sports television channels, Disney+ Hotstar, and certain RIL-controlled media businesses, with RIL holding a 56% controlling interest, the Company holding 37%, and a third-party investment company holding 7%. The Company now recognizes its 37% share of the joint venture\u2019s results under \"Equity in the income of investees.\" Additionally, the document provides financial results for the Entertainment segment, showing a 9% increase in total revenues and a 95% increase in operating income compared to the prior-year quarter. The growth in operating income is attributed to improved results in Content Sales/Licensing and Direct-to-Consumer, partially offset by a decline in Linear Networks."
+    },
+    {
+      "title": "OTHER FINANCIAL INFORMATION",
+      "start_index": 9,
+      "end_index": 9,
+      "nodes": [
+        {
+          "title": "Corporate and Unallocated Shared Expenses",
+          "start_index": 9,
+          "end_index": 9,
+          "node_id": "0022",
+          "summary": "The partial document provides a financial overview of revenues and operating income for Parks & Experiences, including Domestic, International, and Consumer Products segments, comparing the quarters ending December 28, 2024, and December 30, 2023. It highlights a 3% increase in overall revenue and stable operating income. Domestic Parks and Experiences were negatively impacted by Hurricanes Milton and Helene, leading to closures, cancellations, higher costs, and lower attendance, despite increased guest spending. International Parks and Experiences saw growth in operating income due to higher guest spending, increased attendance, and new offerings, offset by higher costs. The document also notes a $152 million increase in corporate and unallocated shared expenses due to a legal settlement and a $143 million loss related to the Star India Transaction."
+        },
+        {
+          "title": "Restructuring and Impairment Charges",
+          "start_index": 9,
+          "end_index": 9,
+          "node_id": "0023",
+          "summary": "The partial document provides financial performance details for the Parks & Experiences segment, including revenues and operating income for domestic and international operations, as well as consumer products. It highlights a 3% increase in overall revenue and stable operating income compared to the prior year. Domestic parks and experiences were negatively impacted by hurricanes, leading to lower volumes and higher costs, despite increased guest spending. International parks and experiences saw growth in operating income due to higher guest spending, increased attendance, and new offerings, though costs also rose. Additionally, corporate and unallocated shared expenses increased due to a legal settlement, and a $143 million loss was recorded related to the Star India Transaction."
+        },
+        {
+          "title": "Interest Expense, net",
+          "start_index": 10,
+          "end_index": 10,
+          "node_id": "0024",
+          "summary": "The partial document provides a financial analysis of interest expense, net, equity in the income of investees, and income taxes for the quarters ending December 28, 2024, and December 30, 2023. Key points include:\n\n1. **Interest Expense, Net**: A decrease in interest expense due to lower average rates and debt balances, partially offset by reduced capitalized interest. Interest income and investment income declined due to lower cash balances, pension-related costs, and investment losses compared to prior-year gains.\n\n2. **Equity in the Income of Investees**: A $89 million decrease in income from investees, primarily due to lower income from A+E and losses from the India joint venture.\n\n3. **Income Taxes**: An increase in the effective income tax rate from 25.1% to 27.8%, driven by a non-cash tax charge related to the Star India Transaction, partially offset by favorable adjustments related to prior years, lower foreign tax rates, and a comparison to unfavorable prior-year effects of employee share-based awards."
+        },
+        {
+          "title": "Equity in the Income of Investees",
+          "start_index": 10,
+          "end_index": 10,
+          "node_id": "0025",
+          "summary": "The partial document provides a financial analysis of interest expense, net, equity in the income of investees, and income taxes for the quarters ended December 28, 2024, and December 30, 2023. It highlights a decrease in net interest expense due to lower average rates and debt balances, offset by reduced capitalized interest. Interest income and investment income declined due to lower cash balances, pension-related costs, and investment losses. Equity income from investees decreased significantly, driven by lower income from A+E and losses from the India joint venture. The effective income tax rate increased due to a non-cash tax charge related to the Star India Transaction, partially offset by favorable adjustments related to prior years, lower foreign tax rates, and a comparison to unfavorable prior-year effects."
+        },
+        {
+          "title": "Income Taxes",
+          "start_index": 10,
+          "end_index": 10,
+          "node_id": "0026",
+          "summary": "The partial document provides a financial analysis of interest expense, net, equity in the income of investees, and income taxes for the quarters ended December 28, 2024, and December 30, 2023. It highlights a decrease in net interest expense due to lower average rates and debt balances, offset by reduced capitalized interest. Interest income and investment income declined due to lower cash balances, pension-related costs, and investment losses. Equity income from investees dropped significantly, driven by lower income from A+E and losses from the India joint venture. The effective income tax rate increased due to a non-cash tax charge related to the Star India Transaction, partially offset by favorable adjustments related to prior years, lower foreign tax rates, and a comparison to unfavorable prior-year effects."
+        },
+        {
+          "title": "Noncontrolling Interests",
+          "start_index": 11,
+          "end_index": 11,
+          "node_id": "0027",
+          "summary": "The partial document covers two main points:\n\n1. **Noncontrolling Interests**: It discusses the net income attributable to noncontrolling interests, which decreased by 63% compared to the prior-year quarter. The decrease is attributed to the prior-year accretion of NBC Universal\u2019s interest in Hulu. The calculation of net income attributable to noncontrolling interests is based on income after royalties, management fees, financing costs, and income taxes.\n\n2. **Cash from Operations**: It details cash provided by operations and free cash flow, showing an increase in cash provided by operations by $1.0 billion to $3.2 billion in the current quarter. The increase is driven by lower tax payments, higher operating income at Entertainment, and higher film and television production spending, along with the timing of payments for sports rights. Free cash flow decreased by $147 million compared to the prior-year quarter."
+        },
+        {
+          "title": "Cash from Operations",
+          "start_index": 11,
+          "end_index": 11,
+          "node_id": "0028",
+          "summary": "The partial document covers two main points:\n\n1. **Noncontrolling Interests**: It discusses the net income attributable to noncontrolling interests, which decreased by 63% in the quarter ended December 28, 2024, compared to the prior-year quarter. The decrease is attributed to the prior-year accretion of NBC Universal\u2019s interest in Hulu. The calculation of net income attributable to noncontrolling interests includes royalties, management fees, financing costs, and income taxes.\n\n2. **Cash from Operations**: It details cash provided by operations and free cash flow for the quarter ended December 28, 2024, compared to the prior-year quarter. Cash provided by operations increased by $1.0 billion, driven by lower tax payments, higher operating income at Entertainment, and higher film and television production spending, along with the timing of payments for sports rights. Free cash flow decreased by $147 million due to increased investments in parks, resorts, and other property."
+        },
+        {
+          "title": "Capital Expenditures",
+          "start_index": 12,
+          "end_index": 12,
+          "node_id": "0029",
+          "summary": "The partial document provides details on capital expenditures and depreciation expenses for parks, resorts, and other properties. It highlights an increase in capital expenditures from $1.3 billion to $2.5 billion, primarily due to higher spending on cruise ship fleet expansion in the Experiences segment. The document also breaks down investments and depreciation expenses by category (Entertainment, Sports, Domestic and International Experiences, and Corporate) for the quarters ending December 28, 2024, and December 30, 2023. Depreciation expenses increased from $823 million to $909 million, with detailed figures provided for each segment."
+        },
+        {
+          "title": "Depreciation Expense",
+          "start_index": 12,
+          "end_index": 12,
+          "node_id": "0030",
+          "summary": "The partial document provides details on capital expenditures and depreciation expenses for parks, resorts, and other properties. It highlights an increase in capital expenditures from $1.3 billion to $2.5 billion, primarily due to higher spending on cruise ship fleet expansion in the Experiences segment. The breakdown of investments and depreciation expenses is provided for Entertainment, Sports, Domestic and International Experiences, and Corporate segments for the quarters ending December 28, 2024, and December 30, 2023. Depreciation expenses also increased from $823 million to $909 million, with detailed segment-wise allocations."
+        }
+      ],
+      "node_id": "0021",
+      "summary": "The partial document provides a financial overview of revenues and operating income for Parks & Experiences, including Domestic, International, and Consumer Products segments, comparing the quarters ending December 28, 2024, and December 30, 2023. It highlights a 3% increase in total revenue and stable operating income. Domestic Parks and Experiences were negatively impacted by Hurricanes Milton and Helene, leading to closures, cancellations, higher costs, and lower attendance, despite increased guest spending. International Parks and Experiences saw growth in operating income due to higher guest spending, increased attendance, and new offerings, offset by increased costs. The document also notes a rise in corporate and unallocated shared expenses due to a legal settlement and a $143 million loss related to the Star India Transaction."
+    },
+    {
+      "title": "THE WALT DISNEY COMPANY CONDENSED CONSOLIDATED STATEMENTS OF INCOME",
+      "start_index": 13,
+      "end_index": 13,
+      "node_id": "0031",
+      "summary": "The partial document provides a condensed consolidated statement of income for The Walt Disney Company for the quarters ended December 28, 2024, and December 30, 2023. It includes details on revenues, costs and expenses, restructuring and impairment charges, net interest expense, equity in the income of investees, income before income taxes, income taxes, and net income. It also breaks down net income attributable to noncontrolling interests and The Walt Disney Company. Additionally, it provides earnings per share (diluted and basic) and the weighted average number of shares outstanding (diluted and basic) for both periods."
+    },
+    {
+      "title": "THE WALT DISNEY COMPANY CONDENSED CONSOLIDATED BALANCE SHEETS",
+      "start_index": 14,
+      "end_index": 14,
+      "node_id": "0032",
+      "summary": "The partial document is a condensed consolidated balance sheet for The Walt Disney Company, comparing financial data as of December 28, 2024, and September 28, 2024. It details the company's assets, liabilities, and equity. Key points include:\n\n1. **Assets**: Breakdown of current assets (cash, receivables, inventories, content advances, and other assets), produced and licensed content costs, investments, property (attractions, buildings, equipment, projects in progress, and land), intangible assets, goodwill, and other assets. Total assets increased slightly from $196.2 billion to $197 billion.\n\n2. **Liabilities**: Includes current liabilities (accounts payable, borrowings, deferred revenue), long-term borrowings, deferred income taxes, and other long-term liabilities. Total liabilities remained relatively stable.\n\n3. **Equity**: Details Disney shareholders' equity, including common stock, retained earnings, accumulated other comprehensive loss, and treasury stock. Noncontrolling interests are also included. Total equity increased from $105.5 billion to $106.7 billion.\n\n4. **Overall Financial Position**: The balance sheet reflects a stable financial position with slight changes in assets, liabilities, and equity over the period."
+    },
+    {
+      "title": "THE WALT DISNEY COMPANY CONDENSED CONSOLIDATED STATEMENTS OF CASH FLOWS",
+      "start_index": 15,
+      "end_index": 15,
+      "node_id": "0033",
+      "summary": "The partial document provides a condensed consolidated statement of cash flows for The Walt Disney Company for the quarters ended December 28, 2024, and December 30, 2023. It details cash flow activities categorized into operating, investing, and financing activities. Key points include:\n\n1. **Operating Activities**: Net income increased from $2,151 million in 2023 to $2,644 million in 2024. Other significant changes include variations in depreciation, deferred taxes, equity income, content costs, and changes in operating assets and liabilities, resulting in cash provided by operations of $3,205 million in 2024 compared to $2,185 million in 2023.\n\n2. **Investing Activities**: Investments in parks, resorts, and other properties increased significantly in 2024 ($2,466 million) compared to 2023 ($1,299 million), leading to higher cash used in investing activities.\n\n3. **Financing Activities**: The company saw a net cash outflow in financing activities, including commercial paper borrowings, stock repurchases, and debt reduction. In 2024, cash used in financing activities was $997 million, a significant improvement from $8,006 million in 2023.\n\n4. **Exchange Rate Impact**: Exchange rates negatively impacted cash in 2024 by $153 million, compared to a positive impact of $79 million in 2023.\n\n5. **Overall Cash Position**: The company\u2019s cash, cash equivalents, and restricted cash decreased from $14,235 million at the beginning of the 2023 period to $5,582 million at the end of the 2024 period."
+    },
+    {
+      "title": "DTC PRODUCT DESCRIPTIONS AND KEY DEFINITIONS",
+      "start_index": 16,
+      "end_index": 16,
+      "node_id": "0034",
+      "summary": "The partial document provides an overview of Disney's Direct-to-Consumer (DTC) product offerings, key definitions, and metrics. It details the availability of Disney+, ESPN+, and Hulu as standalone services or bundled offerings in the U.S., including Hulu Live TV + SVOD, which incorporates Disney+ and ESPN+. It explains the global reach of Disney+ in over 150 countries and the various purchase channels, including websites, third-party platforms, and wholesale arrangements. The document defines \"paid subscribers\" as those generating subscription revenue, excluding extra member add-ons, and outlines how subscribers are counted for multi-product offerings. It also describes the calculation of average monthly revenue per paid subscriber for Hulu, ESPN+, and Disney+, including revenue components like subscription fees, advertising, and add-ons, while noting differences in revenue allocation and the impact of wholesale arrangements on average revenue."
+    },
+    {
+      "title": "NON-GAAP FINANCIAL MEASURES",
+      "start_index": 17,
+      "end_index": 17,
+      "nodes": [
+        {
+          "title": "Diluted EPS excluding certain items",
+          "start_index": 17,
+          "end_index": 18,
+          "node_id": "0036",
+          "summary": "The partial document discusses the use of non-GAAP financial measures, specifically diluted EPS excluding certain items (adjusted EPS), total segment operating income, and free cash flow. It explains that these measures are not defined by GAAP but are important for evaluating the company's performance. The document highlights that these measures should be reviewed alongside comparable GAAP measures and may not be directly comparable to similar measures from other companies. It provides details on the adjustments made to diluted EPS, including the exclusion of certain items affecting comparability and amortization of TFCF and Hulu intangible assets, to better reflect operational performance. The document also includes a reconciliation table comparing reported diluted EPS to adjusted EPS for specific quarters, showing the impact of excluded items such as restructuring charges and intangible asset amortization. Additionally, it notes the challenges in providing forward-looking GAAP measures due to unpredictable factors."
+        },
+        {
+          "title": "Total segment operating income",
+          "start_index": 19,
+          "end_index": 20,
+          "node_id": "0037",
+          "summary": "The partial document focuses on the evaluation of the company's performance through two key financial metrics: total segment operating income and free cash flow. It explains that total segment operating income is used to assess the performance of operating segments separately from non-operational factors, providing insights into operational results. A reconciliation table is provided, showing the calculation of total segment operating income for two quarters, highlighting changes in various components such as corporate expenses, restructuring charges, and interest expenses. Additionally, the document discusses free cash flow as a measure of cash available for purposes beyond capital expenditures, such as debt servicing, acquisitions, and shareholder returns. A summary of consolidated cash flows and a reconciliation of cash provided by operations to free cash flow are presented, comparing figures for two quarters and highlighting changes in cash flow components."
+        },
+        {
+          "title": "Free cash flow",
+          "start_index": 20,
+          "end_index": 20,
+          "node_id": "0038",
+          "summary": "The partial document provides a reconciliation of the company's consolidated cash provided by operations to free cash flow for the quarters ended December 28, 2024, and December 30, 2023. It highlights a $1,020 million increase in cash provided by operations, a $1,167 million increase in investments in parks, resorts, and other property, and a $147 million decrease in free cash flow."
+        }
+      ],
+      "node_id": "0035",
+      "summary": "The partial document discusses the use of non-GAAP financial measures by the company, including diluted EPS excluding certain items (adjusted EPS), total segment operating income, and free cash flow. It explains that these measures are not defined by GAAP but are important for evaluating the company's performance. The document emphasizes that these measures should be reviewed alongside comparable GAAP measures and may not be directly comparable to similar measures from other companies. It highlights the company's inability to provide forward-looking GAAP measures or reconciliations due to uncertainties in predicting significant items. Additionally, the document details the rationale for excluding certain items and amortization of TFCF and Hulu intangible assets from diluted EPS to enhance comparability and provide a clearer evaluation of operational performance, particularly given the significant impact of the 2019 TFCF and Hulu acquisition."
+    },
+    {
+      "title": "FORWARD-LOOKING STATEMENTS",
+      "start_index": 21,
+      "end_index": 21,
+      "node_id": "0039",
+      "summary": "The partial document outlines the inclusion of forward-looking statements in an earnings release, emphasizing that these statements are based on management's views and assumptions about future events and business performance. It highlights that actual results may differ materially due to various factors, including company actions (e.g., restructuring, strategic initiatives, cost rationalization), external developments (e.g., economic conditions, competition, consumer behavior, regulatory changes, technological advancements, labor market activities, and natural disasters), and their potential impacts on operations, profitability, content performance, advertising markets, and taxation. The document also references additional risk factors and analyses detailed in the company's filings with the SEC, such as annual and quarterly reports."
+    },
+    {
+      "title": "PREPARED EARNINGS REMARKS AND CONFERENCE CALL INFORMATION",
+      "start_index": 22,
+      "end_index": 22,
+      "node_id": "0040",
+      "summary": "The partial document provides information about The Walt Disney Company's prepared management remarks and a conference call scheduled for February 5, 2025, at 8:30 AM EST/5:30 AM PST, accessible via a live webcast on their investor website. It also mentions that a replay of the webcast will be available on the site. Additionally, contact details for Corporate Communications (David Jefferson) and Investor Relations (Carlos Gomez) are provided."
+    }
+  ]
+}

tutorials/doc-search/README.md

@@ -0,0 +1,17 @@
+
+
+## Document Search Examples
+
+
+PageIndex currently enables reasoning-based RAG within a single document by default.
+For users who need to search across multiple documents, we provide three best-practice workflows for different scenarios below.
+
+* [**Search by Metadata**:](metadata.md) for documents that can be distinguished by metadata.
+* [**Search by Semantics**:](semantics.md) for documents with different semantic content or cover diverse topics.
+* [**Search by Description**:](description.md) a lightweight strategy for a small number of documents.
+
+
+## 💬 Support
+
+* 🤝 [Join our Discord](https://discord.gg/VuXuf29EUj)
+* 📨 [Contact Us](https://ii2abc2jejf.typeform.com/to/meB40zV0)

tutorials/doc-search/description.md

@@ -0,0 +1,67 @@
+
+## Document Search by Description
+
+For documents that don't have metadata, you can use LLM-generated descriptions to help with document selection. This is a lightweight approach that works best with a small number of documents.
+
+
+### Example Pipeline
+
+
+#### PageIndex Tree Generation
+Upload all documents into PageIndex to get their `doc_id` and tree structure.
+
+#### Description Generation
+
+Generate a description for each document based on its PageIndex tree structure and node summaries.
+```python
+prompt = f"""
+You are given a table of contents structure of a document. 
+Your task is to generate a one-sentence description for the document that makes it easy to distinguish from other documents.
+    
+Document tree structure: {PageIndex_Tree}
+
+Directly return the description, do not include any other text.
+"""
+```
+
+#### Search with LLM
+
+Use an LLM to select relevant documents by comparing the user query against the generated descriptions.
+
+Below is a sample prompt for document selection based on their descriptions:
+
+```python
+prompt = f""" 
+You are given a list of documents with their IDs, file names, and descriptions. Your task is to select documents that may contain information relevant to answering the user query.
+
+Query: {query}
+
+Documents: [
+    {
+        "doc_id": "xxx",
+        "doc_name": "xxx",
+        "doc_description": "xxx"
+    }
+]
+
+Response Format:
+{{
+    "thinking": "<Your reasoning for document selection>",
+    "answer": <Python list of relevant doc_ids>, e.g. ['doc_id1', 'doc_id2']. Return [] if no documents are relevant.
+}}
+
+Return only the JSON structure, with no additional output.
+"""
+```
+
+#### Retrieve with PageIndex
+
+Use the PageIndex `doc_id` of the retrieved documents to perform further retrieval via the PageIndex retrieval API.
+
+
+
+## 💬 Help & Community
+Contact us if you need any advice on conducting document searches for your use case.
+
+- 🤝 [Join our Discord](https://discord.gg/VuXuf29EUj)  
+- 📨 [Leave us a message](https://ii2abc2jejf.typeform.com/to/meB40zV0)

tutorials/doc-search/metadata.md

@@ -0,0 +1,37 @@
+
+
+## Document Search by Metadata
+<callout>PageIndex with metadata support is in closed beta. Fill out this form to request early access to this feature.</callout>
+
+For documents that can be easily distinguished by metadata, we recommend using metadata to search the documents.
+This method is ideal for the following document types:
+- Financial reports categorized by company and time period
+- Legal documents categorized by case type
+- Medical records categorized by patient or condition
+- And many others
+
+In such cases, you can search documents by leveraging their metadata. A popular method is to use "Query to SQL" for document retrieval.
+
+
+### Example Pipeline
+
+#### PageIndex Tree Generation
+Upload all documents into PageIndex to get their `doc_id`.
+
+#### Set up SQL tables
+
+Store documents along with their metadata and the PageIndex `doc_id` in a database table.
+
+#### Query to SQL
+
+Use an LLM to transform a user’s retrieval request into a SQL query to fetch relevant documents.
+
+#### Retrieve with PageIndex
+
+Use the PageIndex `doc_id` of the retrieved documents to perform further retrieval via the PageIndex retrieval API.
+
+## 💬 Help & Community
+Contact us if you need any advice on conducting document searches for your use case.
+
+- 🤝 [Join our Discord](https://discord.gg/VuXuf29EUj)  
+- 📨 [Leave us a message](https://ii2abc2jejf.typeform.com/to/meB40zV0)

tutorials/doc-search/semantics.md

@@ -0,0 +1,41 @@
+## Document Search by Semantics
+
+For documents that cover diverse topics, one can also use vector-based semantic search to search the documents. The procedure is slightly different from the classic vector-search-based method.
+
+### Example Pipeline
+
+
+#### Chunking and Embedding
+Divide the documents into chunks, choose an embedding model to convert the chunks into vectors and store each vector with its corresponding `doc_id` in a vector database.
+
+
+#### Vector Search
+
+For each query, conduct a vector-based search to get top-K chunks with their corresponding documents. 
+
+#### Compute Document Score
+
+For each document, calculate a relevance score. Let N be the number of content chunks associated with each document, and let **ChunkScore**(n) be the relevance score of chunk n. The document score is computed as:
+
+
+$$
+\text{DocScore}=\frac{1}{\sqrt{N+1}}\sum_{n=1}^N \text{ChunkScore}(n)
+$$
+
+- The sum aggregates relevance from all related chunks.
+- The +1 inside the square root ensures the formula handles nodes with zero chunks.
+- Using the square root in the denominator allows the score to increase with the number of relevant chunks, but with diminishing returns. This rewards documents with more relevant chunks, while preventing large nodes from dominating due to quantity alone.
+- This scoring favors documents with fewer, highly relevant chunks over those with many weakly relevant ones.
+
+
+#### Retrieve with PageIndex
+
+Select the documents with the highest DocScore, then use their `doc_id` to perform further retrieval via the PageIndex retrieval API.
+
+
+
+## 💬 Help & Community
+Contact us if you need any advice on conducting document searches for your use case.
+
+- 🤝 [Join our Discord](https://discord.gg/VuXuf29EUj)  
+- 📨 [Leave us a message](https://ii2abc2jejf.typeform.com/to/meB40zV0)

tutorials/tree-search/README.md

@@ -0,0 +1,70 @@
+## Tree Search Examples
+This tutorial provides a basic example of how to perform retrieval using the PageIndex tree.
+
+### Basic LLM Tree Search Example
+A simple strategy is to use an LLM agent to conduct tree search. Here is a basic tree search prompt.
+
+```python
+prompt = f"""
+You are given a query and the tree structure of a document.
+You need to find all nodes that are likely to contain the answer.
+
+Query: {query}
+
+Document tree structure: {PageIndex_Tree}
+
+Reply in the following JSON format:
+{{
+  "thinking": <your reasoning about which nodes are relevant>,
+  "node_list": [node_id1, node_id2, ...]
+}}
+"""
+```
+<callout>
+In our dashboard and retrieval API, we use a combination of LLM tree search and value function-based Monte Carlo Tree Search ([MCTS](https://en.wikipedia.org/wiki/Monte_Carlo_tree_search)). More details will be released soon.
+</callout>
+
+### Integrating User Preference or Expert Knowledge
+Unlike vector-based RAG where integrating expert knowledge or user preference requires fine-tuning the embedding model, in PageIndex, you can incorporate user preferences or expert knowledge by simply adding knowledge to the LLM tree search prompt. Here is an example pipeline.
+
+
+#### 1. Preference Retrieval
+
+When a query is received, the system selects the most relevant user preference or expert knowledge snippets from a database or a set of domain-specific rules. This can be done using keyword matching, semantic similarity, or LLM-based relevance search.
+
+#### 2. Tree Search with Preference
+Integrating preference into the tree search prompt.
+
+**Enhanced Tree Search with Expert Preference Example**
+
+```python
+prompt = f"""
+You are given a question and a tree structure of a document.
+You need to find all nodes that are likely to contain the answer.
+
+Query: {query}
+
+Document tree structure:  {PageIndex_Tree}
+
+Expert Knowledge of relevant sections: {Preference}
+
+Reply in the following JSON format:
+{{
+  "thinking": <reasoning about which nodes are relevant>,
+  "node_list": [node_id1, node_id2, ...]
+}}
+"""
+```
+
+**Example Expert Preference**
+> If the query mentions EBITDA adjustments, prioritize Item 7 (MD&A) and footnotes in Item 8 (Financial Statements) in 10-K reports.
+
+
+
+By integrating user or expert preferences, node search becomes more targeted and effective, leveraging both the document structure and domain-specific insights.
+
+## 💬 Help & Community
+Contact us if you need any advice on conducting document searches for your use case.
+
+- 🤝 [Join our Discord](https://discord.gg/VuXuf29EUj)  
+- 📨 [Leave us a message](https://ii2abc2jejf.typeform.com/to/tK3AXl8T)