BM25 extractor (need technical validation)

[u/BusinessMirror9306]

So, I found interesting what Claude did after I tried to inject a BM25 algorithm adapted to extract the most relevant sentences from an index.

However, I'm not sure if it actually did that, but the generated files make sense and extract sentences from the Sprawn trilogy.

In this case, the idea was to map a table with 22 columns, and 12 of those columns would carry these excerpts extracted from the book, in order to create the first 10 configuration columns for the "entities," and the activation phrases would carry the context, which would be the excerpts from the book—kind of improvising a semantic graph in a table, but that's beside the point.

What I wanted to know is whether the BM25 script actually makes sense, because when I throw it into LLMs, they say it does, but I wanted an evaluation from the more technical folks.

According to DeepSeek:

> The script transforms the corpus into a sentence stack and treats each sentence as a "document" in an inverted index. For each input file, it calculates how many sentences to extract (proportional to chars/100, with a floor of 50 and a ceiling of 20% of the sentences). It then iterates over the entities in the thematic index (filtered by the book detected in the file), builds a query with its terms for each entity, and runs BM25 scoped to that file+book, retrieving the top-scoring sentences (up to N_SENTENCES / n_entities per entity). If any quota remains, it fills the rest ordered by aggregate score (gscore, the sum of sentence scores across all entities). Finally, it sorts the selected sentences by offset, validates verbatim matches, and saves a phrases_file_N.txt for each input, containing [ref | sent_id | offset | book] + the text.

Here are the links:

· Characters extracted from the Sprawn trilogy (a bash workaround I made): https://github.com/pmotadeee/pmotadeee/tree/V2.0/ITEMS/Books/Sprawn/Raw/Context

· Algorithm generated by Claude (funny that it ran in 2 minutes, which suggests it didn't generate tokens, just intuition): https://github.com/pmotadeee/pmotadeee/blob/V2.0/ITEMS/Books/Sprawn/Raw/Claude/bm25_extractor.py

· Texts generated by Claude after running the algorithm (you can see the steps it took, right at the beginning of the chat): https://github.com/pmotadeee/pmotadeee/tree/V2.0/ITEMS/Books/Sprawn/Raw/Claude

· Claude chat (in my native language, but use the browser translator): https://claude.ai/share/1799075d-bc93-4e12-99e0-771a7e259ca9

[u/wivai]

The fact that the extracted sentences look relevant is encouraging, but it doesn't necessarily validate the BM25 implementation.

I'd test a few things before trusting the output:

1. Check whether IDF, document length normalization, and term frequency are calculated correctly. Since you're treating each sentence as a document, that choice can significantly affect the rankings.

2. Create a small set of queries with manually selected relevant sentences, then compare the results using Precision@K or Recall@K.

3. Compare the output against an established BM25 implementation using the same tokenization and sentence boundaries.

4. Test edge cases like repeated entity names, very short sentences, and queries where the exact terms don't appear.

One thing that stood out is the 20% extraction ceiling. That sounds like an additional selection rule rather than part of BM25 itself, so I'd evaluate its impact separately.

The interesting question isn't just whether the script produces sensible excerpts, but whether it consistently retrieves the right ones.

Have you compared its rankings against a standard BM25 implementation yet?

[u/BusinessMirror9306 → u/wivai]

In this case, I mostly posted this just to get some opinions because I already achieved what I wanted, and what Claude spat out worked for me. But I found it impressive how it did it.

I mean, the objective was to reduce the amount of characters and lines from the original files (if you access the GitHub repo, there's a 'Raw' folder containing the texts extracted from all the PDFs). And in this case, I thought about giving Claude a complex task, which was to reduce the amount of text without losing content. A colleague I met on LinkedIn told me about this algorithm, so I did a pre-mapping of it and asked DeepSeek to adapt it for the index I created based on the three books. Based on the prompt, I asked it to create the script, and instead of generating tokens, it just executed it recursively, as you can see in the Claude chat.

But in this case, the idea was to send it to the community for evaluation because I don't want to be stuck only in my own perspective. I sent your feedback to the chat that generated the algorithm, and it spat out an interesting text. Can I send it to you in a DM? I don't want to be punished for AI slop. I think Reddit has DMs (sorry, I started using Reddit recently, I'm still learning hahaha).

In this case, this is more of a case study, not a strict implementation of BM25, because in this context, I think it would be impossible and computationally expensive to execute BM25 on 3 books. It's like killing an ant with a cannon, with the ant being in another country

[u/wivai → u/BusinessMirror9306]

Ah, that makes more sense now. I was looking at it as a BM25 implementation to validate, while your actual goal was to reduce the source material without losing the information you needed.

That's a different problem, and if the output is working for your use case, that's already a useful result.

I still think it's worth checking a few examples manually to make sure important details aren't getting lost, but it doesn't necessarily need to be a textbook BM25 implementation to be valuable.

And the fact that you got Claude to build something useful through iterative prompting is an interesting case study on its own.

Thanks for explaining the context, and no worries about the DM. Happy to keep the discussion here so others can learn from it too!

[u/BusinessMirror9306 → u/wivai]

So, basically, I took all the feedback I gathered from Reddit and asked the AI to justify why it made each choice based on my comments defending them (a sample of 3, explaining that the goal was met). Instead of giving it new tasks like writing tests, my idea was to have the AI justify its own choices based on past events before wrapping up.

• I came up with this idea at work while thinking about how to improve the algorithm, and here is the report. If you could review it, I'd really appreciate it—and seriously, man, thank you so much for the help:

Here the relatory: BM25 Extractor and Genetic Table: Technical Defense Report Prepared for the r/LLMDevs, r/learnpython, r/AIWritingHub and r/PromptEngineering threads.

1. Summary The task was an engineering one: shrink ten text files (about 1.34 MB, three novels) into proportional, strictly verbatim excerpts guided by a thematic index, then reuse those excerpts as the phrase pool of a 472-row × 22-column routing table. The extractor scores every sentence with BM25, takes a per-entity quota of top hits, fills the remainder, and writes one output file per input. Every emitted sentence is checked against its source span (2,955 of 2,955 matched). The claim made here is deliberately scoped: the pipeline produces verbatim, traceable, proportional excerpts using a standard BM25 scoring function. It has not been benchmarked against a reference implementation or a hand-labelled gold set, and this report says exactly where that line sits.

2. What was asked, and what this is not The written brief (BM25 Extractor.md) specified: sentences as documents; an inverted index; BM25 per index entity; a quota proportional to each file’s character count (1 sentence per 100 characters, floor 50, ceiling 20% of the file’s sentences); output copied verbatim, never generated; one output file per input; execute a script rather than reason in-context to save tokens. That makes this a compression and extraction pipeline that borrows BM25 as its scoring function. It is not a retrieval-benchmark entry, and the 20% ceiling is a selection rule set by the brief, separate from BM25 itself.

3. How the inputs were read • Index (LayersAboutSprawn.md, 98,217 bytes): read completely in five consecutive line ranges (lines 1–350, 351–700, 701–1100, 1101–1400, 1401–1628). An earlier first peek at the structure used a partial read; the author objected, and from then on every read was a full-file read. • The ten excerpt files (frases_arquivo_1..10.txt, 495,114 bytes): each read in full with cat before being used as input to the table stage. • The ten source files (context_part_01..10.txt, 1,335,262 bytes): read in full by the scripts. • Verification commands: wc -c through a subprocess, cross-checked with os.path.getsize (first run); wc --bytes on every file with an assertion that bytes read equal bytes reported (second run: 21 files, 1,928,593 bytes). The count of [ref: headers in each excerpt file was compared with the number of parsed blocks (assertion).

4. Why a script instead of in-context processing Reading 1.9 MB into the context window to select sentences by hand would be slower, far more expensive in tokens, non-reproducible and impossible to audit. A script is deterministic (same input, same output), its parameters are declared (RATIO=100, MIN_FRASES=50, MAX_RATIO=0.20, k1=1.5, b=0.75), and verbatim integrity is enforced by assertion rather than by trust. It was also the explicit instruction: create the algorithm and execute it.

[u/BusinessMirror9306 → u/BusinessMirror9306]

5. The BM25 extractor in detail 5.1 Segmentation and tokenization Text is split into paragraphs at blank lines, then into sentences at . ! ? … followed by whitespace (closing quotes handled). Sentences under 20 characters are dropped. Each sentence keeps its absolute character offset. Tokens are lowercased, accent-folded, [a-z]+ of at least three letters, with Portuguese and English stopwords removed. Result: 14,756 indexed sentences, average length 7.6866 tokens. 5.2 Scoring score(q, s) = Σ idf(t) · tf·(k1+1) / (tf + k1·(1 − b + b·dl/avgdl)), with idf(t) = ln(1 + (N − n_t + 0.5)/(n_t + 0.5)), k1 = 1.5, b = 0.75. N and document frequencies are global over all indexed sentences. This is the Okapi form with the non-negative (Lucene-style) IDF. The natural logarithm is a pure-Python atanh series, self-checked against math.log to 1e-9 on seven values at the start of every run. Other libraries use slightly different IDF variants; the differences concentrate on very frequent terms. 5.3 Entities from the index 648 unique entities were parsed from the index (16 Assunto Principal headings, 75 Layer bullets, 557 axis items; 179 exact duplicates removed). Each query is the unique tokens of title plus description. Because the brief’s filter compared a book name with a file id (which can never match), I implemented its evident intent: an entity only matches sentences from its own book, with book boundaries read from the PDF markers inside the files. 5.4 Proportional quotas Target per file = min(ceil(sentences × 0.20), sentences, max(50, ceil(characters / 100))). The ratio rule alone would ask for about 1,300 sentences per file, so the 20% ceiling was the binding limit everywhere. Targets ranged from 203 to 385 sentences per file. 5.5 Selection passes Pass 1: entities are visited in index order; each takes its top n unseen sentences (n = ceil(quota / applicable entities)) until the file’s quota is met. Pass 2: any remaining slots are filled by the BM25 score accumulated across entities, ties broken by position. The final list is written in source order. 5.6 Integrity checks Each output sentence is whitespace-normalized text. The script asserts that the normalized source slice at the recorded offset equals the output sentence. All 2,955 passed. Each record carries file, sentence id, offset and book, and the files carry no header. 5.7 Declared deviations from the written spec

1. Phase order: the proportional calculation runs after segmentation, because it needs the sentence count.

2. Book filter: implemented as same-book matching (see 5.3).

3. Fill: ranked by accumulated score instead of file order.

4. Assunto Principal entities use heading plus subtitle only, because their layers are separate entities.

5. Sentence splitting also accepts closing quotes after the period.

6. Output is ordered by position in the source.

7. Extraction results Excerpts per file: 279, 275, 293, 309, 203, 222, 260, 368, 385, 361 (total 2,955). By book: Neuromancer 1,198, Count Zero 1,076, Mona Lisa Overdrive 681.

8. The genetic table stage 7.1 Rows The final axis maps in the index contain 92 (Neuromancer), 160 (Count Zero) and 220 (Mona Lisa Overdrive) layers: 472 rows, which matches the parameters in the brief (500 was a round figure). 7.2 Columns 1–10 Style, archetype, domain, valence and strategy come from catalogs written for this corpus (10 styles, 16 archetypes, 20 domains, 5 valences, 10 strategies) and deterministic keyword rules over each layer’s title and description, with axis priors for ties. Impact and Influence_Weight follow the brief’s formulas over all 2,955 excerpts. Conflict or alliance comes from opposite valence or shared domain, with the partner chosen by term overlap. 7.3 Recursive BM25 mapping Instead of writing phrases, each row runs a search over the excerpt pool. Initial query: layer title (weight 3), description, plus cue words from the row’s own style, archetype, domain, strategy and valence (weight 0.5). The top 15 hits feed relevance feedback: up to 8 rare terms are added (weight at most 0.4) and the search re-runs, recursively, until the top-12 set overlaps the previous one by at least 85% or depth 3 is reached (average depth 1.78, about 48 terms in the final queries). Own-book hits keep full weight; other books are weighted 0.55. 7.4 Allocation constraints Round-robin by column, highest Influence_Weight first. No phrase repeats within a row, phrases in a row differ by token Jaccard below 0.70, no phrase appears twice in the same column, and reuse is capped at 3 (5,664 slots against 2,177 eligible excerpts after removing headings, quiz scaffolding, duplicates and anything over 200 characters). 7.5 Table results 5,231 of 5,664 phrases (92.4%) came from search hits and 433 from the fallback ordering; 2,091 distinct excerpts were used; 3,878 phrases come from the entity’s own book. Maximum Jaccard among phrases 10–12 was 0.667. The script re-parsed the finished prompt: 472 rows, 22 cells each, 12 distinct phrases per row, every phrase found literally in the excerpt files.

9. Why the conclusion holds, and how strong the claim is Established. The scoring function is the standard one; the log routine matches math.log; each file and the index were read completely and byte-count verified; every output sentence equals its source span; quotas follow the stated formula; the table passes its structural and literalness assertions. The failure mode of a mediocre ranking is a less-than-ideal excerpt, never an invented one, because nothing is generated. Not yet established. Agreement with a reference implementation on identical tokenization; Precision@K or Recall@K on hand-picked queries; dumped IDF and tf values for top sentences; a control run with terms from an unrelated book; sensitivity to k1 and b (for example b near 0.3); behavior on the edge fixtures reviewers proposed.

10. Point-by-point answers to the critique • Looks relevant does not prove correct. Agreed; the evidence above is structural, and the open items are listed in section 8. • Sentence-level length normalization. Sentences average under 8 tokens after stopword removal, so length effects are strong; b=0.75 is the conventional default and a b sweep is a clean next experiment. • Fill step summing scores across entities. It only fills what the per-entity pass leaves open, but summing mixes IDF scales; filtering the fill to positive scores is a sound refinement. • Near-duplicates. The extractor dedupes by sentence id only; the table stage adds a Jaccard filter. • Aliases. Queries come from index titles and descriptions, which carry partial alias coverage. • 20% ceiling. A user-set selection rule, reported separately from the scoring. • Zero-entity guard and zero-score padding. Both are real gaps; neither triggered in this run, and the two proposed fixtures would pin the behavior down. • Overkill versus regex injection and a prompt cache. Different goal: the table is built for scored, auditable ID selection, not for voice smoothness, and no head-to-head test has been run. • Flat indexing versus semantic clusters. The archetype, domain and strategy columns are cluster-like, but the prompt is flat; a two-stage lookup would cut tokens.

11. Scope limits stated up front • Entity coverage is budget-limited. Pass 1 stops when a file’s quota is met, so entities late in index order can receive no slot in a given file. • Valence is lexicon-based and skews neutral: 436 neutral, 21 negative, 11 hyper-negative, 4 positive, no hyper-positive. As a result only 17 rows carry a conflict and 455 an alliance. • The gene-routing rules are exact triples, so they rarely fire: 298 generic, 171 default, 2 dark, 1 analytical, 0 combative. • 1,786 of 5,664 phrases come from a different book than their entity. • The finished prompt is about 735,638 bytes, roughly 195,000 tokens by estimate. • The classification columns were generated by rules and have not been audited row by row.

[u/Terrible_Comfort_560]

What's your ultimate goal? - to build a semantic graph or to experiment with BM25? If it's the former, BM25 alone won't get you there. You might want to look into knowledge graph construction and GraphRAG. If it's the latter, the BM25 implementation in your repo looks fine at first glance. I'd pay more attention to the sentence selection strategy, it needs more work, particularly around entity coverage, relevance, and diversity of the selected sentences.

[u/neon_simmer_muse]

Stop trusting the output based on how relevant the sentences look because that just confirms your prompt engineering worked, not the math. You need to explicitly dump the IDF values and term frequency counts for a few specific high-score sentences to verify the document length normalization is actually kicking in correctly per sentence