Local RAG Setup: TRIZ + Pelevin on Ollama

Two separate RAG projects, one shared architecture — setup, document pipeline, and the full script

Hardware

Future plan for the RTX 5060 Ti: fine-tune Llama 8B on the TRIZ dataset specifically, run RAG with Mistral 7B as an alternative base model, and explore quantized models for running larger LLMs than would otherwise fit.

1. Install Ollama & Python Packages

# Install Ollama
brew install ollama

# Pull the LLM model
ollama pull llama3.1

# Pull the embedding model (required!)
ollama pull nomic-embed-text

# Install Python packages
pip3 install llama-index llama-index-llms-ollama llama-index-embeddings-ollama

2. Test Ollama

# Start Ollama in background
ollama serve

# Open a new terminal tab (CMD+T) and test interactively
ollama run llama3.1

# Or pipe a question directly
echo "What are the 40 TRIZ principles?" | ollama run llama3.1

3. Project Folders

# TRIZ project
mkdir ~/triz-rag
cd ~/triz-rag
mkdir documents

# Pelevin project
mkdir ~/pelevin-rag
cd ~/pelevin-rag
mkdir documents

# Copy your PDF documents into the respective documents/ folder

4. Document Conversion Pipeline

The RAG script reads PDFs specifically, so anything else needs converting first.

EPUB / FB2 (via Calibre)

brew install calibre

# Single EPUB
ebook-convert yourbook.epub yourbook.pdf

# Single FB2
ebook-convert yourbook.fb2 yourbook.pdf

# Batch convert all EPUBs in a folder
for f in ~/books/*.epub; do
    ebook-convert "$f" "${f%.epub}.pdf"
    echo "Converted: $f"
done

# Batch convert all FB2s in a folder
for f in ~/books/*.fb2; do
    ebook-convert "$f" "${f%.fb2}.pdf"
    echo "Converted: $f"
done

DJVU (via djvulibre)

brew install djvulibre

# Single file
ddjvu -format=pdf yourbook.djvu yourbook.pdf

# Batch
for f in ~/books/*.djvu; do
    ddjvu -format=pdf "$f" "${f%.djvu}.pdf"
    echo "Converted: $f"
done

DOC / DOCX (via LibreOffice)

brew install libreoffice
libreoffice --convert-to pdf yourfile.doc

Audiobooks → Text (via Whisper)

# Install tools
pip3 install openai-whisper
brew install ffmpeg

# Single audiobook
whisper audiobook.mp3 --language Russian --output_format txt

# Batch processing
for f in ~/audiobooks/*.mp3; do
    whisper "$f" --language Russian --output_format txt
    echo "Done: $f"
done

5. The RAG Script — Hardened Version

Same script used across all six projects — only SYSTEM_PROMPT, DOCUMENTS_PATH, and the project name change between them. This version is a step up from the original: it checks that documents/ and storage/ exist (and creates them if not), tolerates a missing or corrupted tracking file instead of crashing, supports both .pdf and .txt input, and fails with an actual explanation instead of a stack trace when something's missing. The Dostoevsky prompt is shown active below (bilingual, since Dostoevsky and Pelevin are both run in Russian); swap in whichever author/topic prompt you need.

from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    StorageContext,
    load_index_from_storage
)
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.core import Settings
import os
import json

# ── Configuration ──────────────────────────────
DOCUMENTS_PATH = "./documents"
INDEX_PATH     = "./storage"
TRACKED_FILES  = "./storage/indexed_files.json"
MODEL_NAME     = "llama3.1"
EMBED_MODEL    = "nomic-embed-text"
TIMEOUT        = 600.0
DIVIDER        = "-" * 50
EXIT_COMMANDS  = ['exit', 'quit', 'q', 'bye', 'пока', 'выход']
# ───────────────────────────────────────────────

# ── Swap this prompt per project ───────────────
SYSTEM_PROMPT = """
You are a Fyodor Dostoevsky expert assistant with access to
his complete works — all novels, novellas, short stories,
essays, and journalism in Russian.

Answer in the same language the question is asked.
Be deep, philosophical, and compassionate — like Dostoevsky himself.
When quoting, mention which book it is from.

Always base answers on the provided Dostoevsky documents.
"""
# ───────────────────────────────────────────────

def setup():
    """Initialize models and ensure required directories exist"""
    print("Setting up models...")

    # Create documents directory if it doesn't exist
    if not os.path.exists(DOCUMENTS_PATH):
        os.makedirs(DOCUMENTS_PATH)
        print(f"Created documents directory: {DOCUMENTS_PATH}")
        print(f"Add your PDF/TXT files to {DOCUMENTS_PATH}, then run again.")
        exit(0)

    # Create storage directory if it doesn't exist
    if not os.path.exists(INDEX_PATH):
        os.makedirs(INDEX_PATH)
        print(f"Created storage directory: {INDEX_PATH}")

    Settings.llm = Ollama(
        model=MODEL_NAME,
        request_timeout=TIMEOUT,
        context_window=4096,
        system_prompt=SYSTEM_PROMPT
    )
    Settings.embed_model = OllamaEmbedding(
        model_name=EMBED_MODEL
    )
    Settings.chunk_size = 256      # smaller = more diverse retrieval
    Settings.chunk_overlap = 50    # more variety in chunks for context
    print("Models ready!")

# ── File Tracking ──────────────────────────────

def load_tracked_files():
    """Load list of already indexed files"""
    if not os.path.exists(TRACKED_FILES):
        os.makedirs(os.path.dirname(TRACKED_FILES), exist_ok=True)
        save_tracked_files(set())
        return set()

    try:
        with open(TRACKED_FILES, 'r') as f:
            return set(json.load(f))
    except (json.JSONDecodeError, FileNotFoundError):
        print("Tracked files file is corrupted. Starting fresh...")
        save_tracked_files(set())
        return set()

def save_tracked_files(tracked):
    """Save list of indexed files"""
    os.makedirs(os.path.dirname(TRACKED_FILES), exist_ok=True)
    with open(TRACKED_FILES, 'w') as f:
        json.dump(list(tracked), f, indent=2)

def get_new_files(tracked_files):
    """Find PDF/TXT files in documents folder not yet indexed"""
    if not os.path.exists(DOCUMENTS_PATH):
        os.makedirs(DOCUMENTS_PATH)
        print(f"Created documents directory: {DOCUMENTS_PATH}")
        return set()

    all_files = set(
        f for f in os.listdir(DOCUMENTS_PATH)
        if f.endswith('.pdf') or f.endswith('.txt')
    )

    if not all_files:
        print(f"No PDF or TXT files found in {DOCUMENTS_PATH}")
        return set()

    return all_files - tracked_files

# ── Index Management ───────────────────────────

def build_fresh_index():
    """Build index from scratch — first time only"""
    print("Loading all documents...")

    if not os.path.exists(DOCUMENTS_PATH) or not os.listdir(DOCUMENTS_PATH):
        print(f"No documents found in {DOCUMENTS_PATH}")
        return None

    try:
        documents = SimpleDirectoryReader(DOCUMENTS_PATH).load_data()
    except Exception as e:
        print(f"Error loading documents: {e}")
        return None

    if not documents:
        print(f"No documents could be loaded from {DOCUMENTS_PATH}")
        return None

    print(f"Loaded {len(documents)} document chunks")
    print("Building index (first time only)...")

    index = VectorStoreIndex.from_documents(documents, show_progress=True)

    os.makedirs(INDEX_PATH, exist_ok=True)
    index.storage_context.persist(persist_dir=INDEX_PATH)

    all_files = set(os.listdir(DOCUMENTS_PATH))
    save_tracked_files(all_files)
    print(f"Index built and saved! {len(all_files)} files tracked.")
    return index

def update_index(index, new_files):
    """Add only new documents to existing index"""
    print(f"\nFound {len(new_files)} new file(s) to index:")
    for f in new_files:
        print(f"  + {f}")

    try:
        new_docs = SimpleDirectoryReader(
            input_files=[os.path.join(DOCUMENTS_PATH, f) for f in new_files]
        ).load_data()
    except Exception as e:
        print(f"Error loading new documents: {e}")
        return index

    if not new_docs:
        print("No new documents could be loaded.")
        return index

    print(f"\nIndexing {len(new_docs)} new chunks...")
    for doc in new_docs:
        index.insert(doc)

    os.makedirs(INDEX_PATH, exist_ok=True)
    index.storage_context.persist(persist_dir=INDEX_PATH)

    tracked = load_tracked_files()
    tracked.update(new_files)
    save_tracked_files(tracked)

    print(f"Done! {len(new_files)} new files added to index.")
    return index

def load_or_build_index():
    """Smart loader — builds, updates, or loads index, never crashes silently"""

    if not os.path.exists(DOCUMENTS_PATH) or not os.listdir(DOCUMENTS_PATH):
        print(f"No documents found in {DOCUMENTS_PATH}. Add files first.")
        return None

    if not os.path.exists(INDEX_PATH):
        return build_fresh_index()

    print("Loading existing index...")
    try:
        storage_context = StorageContext.from_defaults(persist_dir=INDEX_PATH)
        index = load_index_from_storage(storage_context)
    except Exception as e:
        print(f"Could not load existing index: {e}")
        print("Building fresh index...")
        return build_fresh_index()

    tracked_files = load_tracked_files()
    new_files = get_new_files(tracked_files)

    if new_files:
        print("New files detected — updating index...")
        index = update_index(index, new_files)
    else:
        print(f"Index up to date! {len(tracked_files)} files loaded.")

    return index

# ── Chat ───────────────────────────────────────

def chat(query_engine):
    print("\nAssistant Ready!")
    print("Ask in Russian or English")
    print("Type 'exit' or 'выход' to quit")
    print(DIVIDER)

    while True:
        try:
            question = input("\nYour question: ").strip()
        except (KeyboardInterrupt, EOFError):
            print("\nGoodbye! / Пока!")
            break

        if question.lower() in EXIT_COMMANDS:
            print("Goodbye! / Пока!")
            break

        if not question:
            continue

        print("\nThinking...")
        try:
            response = query_engine.query(question)
            print(f"\nAnswer:\n{response}")
        except Exception as e:
            print(f"\nError: {e}")
            print("Try asking again.")

        print(DIVIDER)

def main():
    """Main entry point with proper error handling"""
    setup()
    index = load_or_build_index()

    if index is None:
        print("\nCould not initialize the index. Please make sure you have:")
        print(f"  1. Added PDF or TXT files to {DOCUMENTS_PATH}")
        print("  2. Ollama running with the required models")
        print("  3. Restarted the script")
        return

    query_engine = index.as_query_engine(similarity_top_k=8)
    chat(query_engine)

if __name__ == "__main__":
    main()

What changed from the first version: directory bootstrapping for both documents/ and storage/, a tracking file that rebuilds itself instead of crashing if corrupted or missing, .txt support alongside PDF, and every I/O step wrapped so a bad file or a half-finished index produces a message instead of a traceback. None of this changes behaviour when things go right — it only changes what happens when they don't.

6. How Incremental Indexing Works

First run:
└── No index found → builds from scratch → saves index
└── Saves list of indexed files to indexed_files.json

Next runs:
└── Loads existing index instantly (~5 seconds)
└── Checks documents folder for new files
└── Only indexes new files → inserts into existing index
└── Updates tracked files list

Adding a new book:
└── Copy new PDF to documents/
└── Run script → detects and indexes only the new file
└── Done in minutes instead of hours

Rebuild index from scratch

# Only needed if you want a completely fresh start
rm -rf ~/triz-rag/storage
python3 triz_rag.py

Adding new books

# Copy new book to documents folder
cp new_book.pdf ~/pelevin-rag/documents/

# Run — detects and indexes only the new book
python3 pelevin-rag.py

Expected Indexing Times

Key Settings Reference

Project File Structure

~/triz-rag/               ~/pelevin-rag/
├── documents/            ├── documents/
│   └── *.pdf             │   └── *.pdf
├── storage/              ├── storage/
│   ├── *.json            │   ├── *.json
│   └── indexed_files.json│   └── indexed_files.json
└── triz_rag.py           └── pelevin-rag.py

Extending to Other Collections

The same script works for any document collection — just change:

1. DOCUMENTS_PATH to point to your documents folder
2. SYSTEM_PROMPT to describe the new domain
3. Rename the script

Ideas on the list: Discworld, materials science papers, legal documents, medical textbooks, any research domain — the same architecture is genuinely domain-agnostic.