Calibrated Entropy-Weighted Hybrid Retrieval — Technical Guide & Operational Engine

Operational Engine

🎮 Operational Retrieval Engine & Live Queries

Test the actual hybrid retrieval pipeline I built across Computer Science & Biomedical corpora. If top candidates fall below minimum confidence (Score < 0.15), the system explicitly rejects the result to prevent hallucination.

Search Question / Claim

Retrieval Mode

Sample Queries:

Technology Stack

What I Built & Core Technologies

Every library, framework, and model I integrated into the system.

🐍

Python 3.11

Core runtime language, typing system, async context management, and fast scientific computation.

⚡

FastAPI & Uvicorn

High-performance web API framework with automatic Pydantic request validation and interactive Swagger UI docs.

🔍

BM25Okapi (rank_bm25)

Lexical keyword retriever calculating term frequency / inverse document frequency scores across tokenized corpora.

📐

FAISS & Sentence Transformers

Dense vector search using all-MiniLM-L6-v2 embeddings (384 dimensions) and IndexFlatL2 distance indexing.

🎯

Cross-Encoder Reranker

High-precision reranking using cross-encoder/ms-marco-MiniLM-L6-v2 (22M parameter pairwise cross-attention model).

📊

NumPy & SciPy

Empirical CDF quantile mapping via np.searchsorted, Pearson correlation r with p-values, and bootstrap resampling.

System Architecture

End-to-End Pipeline & Background Data Flow

Click any pipeline node to expand and inspect its detailed execution logic right beneath that step.

1

Query Receipt & Tokenization

User query hits POST /query endpoint

app/main.py

Step 1: User Query Input & Tokenization

The user submits a natural language search query (e.g. "Transformer models rely on multi-head self-attention mechanisms.") to the POST /query endpoint. The query is cleaned and tokenized separately for lexical and dense search streams.

2

Dual Parallel Retrieval (BM25 + FAISS)

Retrieves candidate_k=20 chunks from both engines

app/retriever.py

3

Corpus-Level CDF Quantile Calibration

Maps raw scores to corpus percentiles via np.searchsorted

app/calibration.py

4

Shannon Entropy Computation (H_sparse & H_dense)

Measures retrieval uncertainty across score distributions

app/calibration.py

5

Adaptive Dynamic Alpha Weight Calculation

Computes precision-weighted balance: α = H_dense / (H_dense + H_sparse + ε)

app/fusion.py

6

Weighted Linear Fusion

Score = α · BM25_cal + (1-α) · FAISS_cal

app/fusion.py

7

Cross-Encoder Pairwise Reranking

Full cross-attention scoring using ms-marco-MiniLM-L6-v2

app/reranker.py

8

Document ID Deduplication

Keeps top-scoring chunk per source document

scripts/evaluate.py

9

Final Output & Telemetry

Returns top-k snippets, sources, scores, latency & entropy metrics

app/main.py

Data Lifecycle

How Data Was Before, How It's Split, & Persistence

From raw text to FAISS vectors, BM25 inverted indices, and corpus CDF lookup tables.

📄 1. Input Data Structure

Raw Sources: Uploaded PDF files, plain text (.txt), markdown (.md), Computer Science documents, or the benchmark SciFact corpus (5,183 PubMed scientific abstracts).

Before Chunking: Long documents with full title, abstract, and body text. Raw scores across different length documents would introduce severe length bias without chunking.

✂️ 2. Recursive Chunking Strategy

Using RecursiveCharacterTextSplitter:

chunk_size = 800 characters
chunk_overlap = 120 characters

Metadata Attached: Each chunk is assigned a tuple key (source_doc_id, chunk_index), e.g. "CS-101:1".

🗂️ 3. FAISS Vector Store

Each chunk is embedded into a 384-dimensional vector using all-MiniLM-L6-v2 and saved to data/faiss_index/index.faiss.

📖 4. BM25 Tokenized Corpus

Chunks are lowercased, stripped of punctuation, whitespace-split, and pickled in bm25_index.pkl for BM25Okapi keyword matching.

📈 5. Corpus CDF Score Arrays

100 sample queries are scored against ALL 17,243 corpus chunks offline (1,724,300 scores per retriever). Sorted arrays saved to corpus_cdf_bm25.npy and corpus_cdf_dense.npy.

Mathematical Foundations

Exact Formulas & Algorithm Mechanics

Understanding score calibration, entropy, precision-weighting, and statistical testing.

1. BM25 Okapi Keyword Scoring

BM25 measures term frequency saturation and document length normalization:

$$\text{BM25}(q, D) = \sum_{i=1}^{n} \text{IDF}(q_i) \cdot \frac{f(q_i, D) \cdot (k_1 + 1)}{f(q_i, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)}$$

Where k₁=1.5 controls term frequency saturation, b=0.75 controls document length penalty, and IDF(q_i) = ln((N - n(q_i) + 0.5) / (n(q_i) + 0.5) + 1).

2. FAISS Dense L2 Distance to Similarity

FAISS IndexFlatL2 computes Euclidean distance d_L2. I convert distance to similarity score in (0, 1]:

$$S_{\text{dense}} = \frac{1}{1 + d_{\text{L2}}}$$

3. Corpus-Level Quantile CDF Calibration

Per-query min-max destroys global scale information. Instead, I map score s to its percentile rank in the 1.7M corpus score distribution:

$$S_{\text{cdf}}(s) = \frac{\text{searchsorted}(\text{Corpus\_CDF}, s)}{N_{\text{corpus}}}$$

Maps any raw score distribution to approximately uniform [0, 1] quantile space while preserving per-query confidence variation.

4. Shannon Entropy of Score Distributions

Normalizes scores into a probability mass p_i = S_i / Σ S_j, then calculates Shannon entropy (base 2):

$$H = -\sum_{i=1}^{k} p_i \log_2(p_i)$$

Note: For negative-valued z-scores, softmax is applied: p_i = e^{S_i - max(S)} / Σ e^{S_j - max(S)} to prevent affine collapse.

5. Bayesian Precision-Weighted Fusion Weight (α)

Per-query dynamic weight allocation based on retriever entropy (uncertainty):

$$\alpha = \frac{H_{\text{dense}}}{H_{\text{dense}} + H_{\text{sparse}} + \varepsilon}$$ $$\text{Fused Score} = \alpha \cdot S_{\text{sparse, cal}} + (1 - \alpha) \cdot S_{\text{dense, cal}}$$

If FAISS dense retrieval has high entropy (uncertainty), α $\to$ 1 (system relies on BM25). If BM25 has high entropy, α $\to$ 0 (system relies on FAISS dense vectors).

Interactive Playground

Live Entropy & Adaptive α Weight Simulator

Adjust dense and sparse entropy sliders to watch how the system dynamically shifts trust!

Dense Retriever Entropy (H_dense) 4.20

Higher value = FAISS vector search is uncertain/confused.

Sparse Retriever Entropy (H_sparse) 4.20

Higher value = BM25 keyword search is uncertain/confused.

Calculated Alpha (α)

0.500

BM25 Weight: 50.0% FAISS Weight: 50.0%

Balanced Confidence: Equal weighting between lexical keywords and vector semantics.

Codebase Directory

What Each File & Function Does

Exhaustive guide to all 9 source files and their underlying functions.

📄 app/main.py

FastAPI Endpoints & Lifespan

FastAPI application defining REST API endpoints and startup handlers.

lifespan(app): Async startup context manager that creates upload directory and triggers store.load().
GET /health: Returns index status, loaded files, and document counts.
GET /modes: Lists all 7 available retrieval modes with descriptions.
POST /upload: Accepts PDF, TXT, MD files, saves to data/uploads/, and indexes chunks.
POST /query: Main search endpoint accepting question, top_k, and retrieval mode.

📄 app/retriever.py

Pipeline Orchestrator & DocumentStore

Central document manager and mode router combining FAISS, BM25, and Cross-Encoder.

DocumentStore.load(): Loads FAISS vectorstore, BM25 pickle index, CDF arrays, and text lookup map from disk.
DocumentStore.add_file(path): Extracts text from PDF/TXT/MD, chunks text, and builds indices.
DocumentStore._search_dense(query, top_k): Runs FAISS similarity search and converts L2 distance to [0, 1] similarity.
DocumentStore._search_sparse(query, top_k): Runs BM25 tokenized keyword search.
DocumentStore.search(query, top_k, mode): Routes search to one of 7 modes and records latency telemetry.
extract_text(path): Extracts raw text from PDF using pypdf.PdfReader or UTF-8 text files.
chunk_text(text, source): Splits text with RecursiveCharacterTextSplitter(800, 120).

📄 app/calibration.py

Score Calibration, Entropy & Statistics

Research math module handling distribution transforms, Shannon entropy, QPP baselines, and bootstrap significance testing.

calibrate_raw(scores): Identity baseline.
calibrate_minmax(scores): Scales scores to [0, 1] per query.
calibrate_zscore(scores): Standardizes scores to mean=0, std=1.
calibrate_cdf(scores, corpus_cdf): Maps scores to corpus percentile ranks using np.searchsorted.
compute_entropy(scores): Computes Shannon entropy H = -Σ p log2 p with softmax handling for negative z-scores.
compute_alpha(h_dense, h_sparse): Calculates dynamic precision weight α = H_dense / (H_dense + H_sparse + ε).
build_corpus_cdfs(...): Scores sample queries against ALL corpus docs to create offline CDF arrays.
clarity_score(top_k_texts, corpus_freqs): Computes KL divergence of top-k language model vs corpus language model.
nqc(scores, corpus_mean): Normalized Query Commitment std(top_k) / |mean(corpus)|.
bootstrap_ci(a, b, n_resamples=1000): Performs paired bootstrap resampling for 95% confidence intervals and p-value significance.

📄 app/fusion.py

RRF & Linear Score Fusion

Implements reciprocal rank fusion, fixed linear fusion, and calibrated entropy-weighted fusion.

rrf_fuse(sparse, dense, k=60): Combines candidate ranks via RRF(d) = Σ 1 / (60 + r(d)).
linear_fuse(sparse, dense, alpha=0.5): Merges candidates with fixed α=0.5 linear combination.
entropy_fuse(sparse, dense, calibration="cdf"): Calibrates scores, computes Shannon entropy per retriever, calculates dynamic α, and merges candidates.

📄 app/sparse_retriever.py

BM25 Index & Search

Manages BM25Okapi index and full-corpus scoring.

BM25Index.tokenize(text): Lowercases, strips punctuation, and splits text into tokens.
BM25Index.add_documents(docs): Tokenizes documents and builds BM25Okapi model.
BM25Index.score_all(query): Returns BM25 scores across all corpus documents (used for offline CDF building).

📄 app/reranker.py

Cross-Encoder Transformer

Lazy-loads cross-encoder/ms-marco-MiniLM-L6-v2 for pairwise (query, document) scoring.

CrossEncoderReranker.rerank(query, candidates, top_k): Runs pairwise cross-attention prediction and sorts candidates descending by score.

📄 app/datasets.py

SciFact Dataset Loader

Loads BEIR SciFact benchmark corpus (5,183 PubMed abstracts), test claims (300 queries), and qrels relevancy labels.

load_scifact_corpus(): Parses corpus.jsonl into document records.
load_scifact_queries(): Parses queries.jsonl mapping query IDs to claim text.
load_scifact_qrels(): Loads ground-truth relevance pairs from qrels/test.tsv.

📄 scripts/build_index.py

Offline Index & CDF Generator

Offline script executing indexing step, generating FAISS vectors, BM25 indices, unigram language models, document lookup tables, and corpus-level CDF numpy files.

📄 scripts/evaluate.py

Scientific Evaluation & H1/H2 Benchmark

Evaluates 7 retrieval modes across 300 test claims, computes Pearson correlations for H1, runs 1,000 paired bootstrap resamples for H2, and deduplicates chunk results by document ID.

Scientific Evidence

BEIR SciFact Benchmark Results & Research Hypotheses

Evaluating 7 retrieval modes across 300 test claims (5,183 PubMed abstracts).

Mode	NDCG@10	MRR	P@3	P@5	R@5	p95 Latency
`dense`	0.6715	0.6329	0.2466	0.1647	0.7495	33 ms
`sparse`	0.6151	0.5804	0.2233	0.1520	0.7089	130 ms
`rrf`	0.7028	0.6713	0.2555	0.1680	0.7744	249 ms
`hybrid_fixed`	0.6829	0.6527	0.2522	0.1647	0.7594	232 ms
`hybrid_calibrated`	0.6981	0.6672	0.2489	0.1660	0.7661	241 ms
`hybrid_fixed_rerank`	0.7006	0.6653	0.2578	0.1700	0.7714	2432 ms
hybrid_calibrated_rerank	0.7071	0.6719	0.2622	0.1720	0.7838	2116 ms

🔬 H1: Entropy Correlation Analysis

Hypothesis: CDF entropy has the strongest negative correlation with retrieval quality.

❌ Verdict: Not Supported.

Z-score entropy turned out to be the strongest negative predictor (r = -0.4065, p < 10^-6) for dense retrieval on SciFact, whereas CDF entropy showed a weaker correlation (r = -0.1151).

📊 H2: Paired Bootstrap Significance

Hypothesis: Calibrated entropy fusion significantly outperforms fixed-alpha and RRF baselines.

⚠️ Verdict: Partially Supported.

CDF entropy fusion significantly beats fixed-alpha fusion (+0.0152 NDCG@10 gain, p = 0.012), but ties with RRF. With cross-encoder reranking, the full pipeline achieves the best overall NDCG@10 (0.7071).

Deployment & Setup Guide

How to Run & Deploy

Commands to build indices, run FastAPI, execute pytest, and evaluate performance.

1. Local Setup & Installation

# Create virtual environment
python -m venv venv

# Activate venv (Windows PowerShell)
.\venv\Scripts\Activate.ps1

# Install requirements
pip install -r requirements.txt

2. Build Retrieval Artifacts & Index SciFact

# Builds FAISS, BM25, Corpus CDFs, and Document Lookup tables
python scripts/build_index.py --dataset scifact

3. Start the FastAPI Server & Interactive Swagger Docs

# Launch server at http://127.0.0.1:8000
uvicorn app.main:app --reload

# Open Swagger Docs in your browser: http://127.0.0.1:8000/docs

4. Run Pytest Test Suite

# Executes all 9 test cases covering API endpoints, telemetry, and calibration
pytest tests/ -v

5. Run Benchmark Evaluation & Significance Tests

# Evaluates all 7 modes on SciFact test claims & runs bootstrap tests
python scripts/evaluate.py