Please enable JavaScript.
Coggle requires JavaScript to display documents.
Modern IR System Architecture - Coggle Diagram
Modern IR System Architecture
Ingestion & Document Pipeline
Distributed Web Crawling (Politeness, robots.txt, URL frontier)
Text Normalization & Tokenization (Stemming, stop-words, Unicode)
Near-Duplicate Detection (SimHash fingerprinting, MinHash LSH)
Document Parsers & Metadata (HTML/JSON stripping, tag weighting)
Indexing Engine & Storage
Inverted Index Architecture (Term dictionaries, posting lists)
Index Compression (Elias-Fano encoding, Variable Byte packing)
Dynamic Index Updates (LSM-trees, forward index buffers)
Tiered Sharding & Partitioning (Document vs. Term sharding)
Query Processing Layer
Query Parsing & Intent Detection (Syntax trees, entity recognition)
Query Expansion & Reformulation (Word embeddings, pseudo-relevance feedback)
Error Correction Pipelines (SymSpell, noisy channel models)
Security & Guardrails (Sanitization, injection blocking, rate limits)
Multi-Stage Scoring & Ranking
L1 Retrieval Baseline (Fast sparse retrieval, BM25, Lucene)
L2 Learning to Rank (LTR) (XGBoost/LightGBM, pairwise/listwise losses)
Dense Features Integration (Semantic vector distances + lexical scores)
Signal Aggregation (PageRank, click-through history, recency boosts)
Serving Infrastructure & Low Latency
Multi-Level Caching (Result caching, posting list caching)
Early Termination Algorithms (WAND, Block-Max WAND dynamic skipping)
Scatter-Gather Coordination (Broker node routing, parallel leaf execution)
Telemetry & Evaluation Logging (Real-time click logs, latency tracing)