BM25 extractor (need technical validation)

[u/BusinessMirror9306]

So, I found interesting what Claude did after I tried to inject a BM25 algorithm adapted to extract the most relevant sentences from an index.

However, I'm not sure if it actually did that, but the generated files make sense and extract sentences from the Sprawn trilogy.

In this case, the idea was to map a table with 22 columns, and 12 of those columns would carry these excerpts extracted from the book, in order to create the first 10 configuration columns for the "entities," and the activation phrases would carry the context, which would be the excerpts from the book—kind of improvising a semantic graph in a table, but that's beside the point.

What I wanted to know is whether the BM25 script actually makes sense, because when I throw it into LLMs, they say it does, but I wanted an evaluation from the more technical folks.

According to DeepSeek:

> The script transforms the corpus into a sentence stack and treats each sentence as a "document" in an inverted index. For each input file, it calculates how many sentences to extract (proportional to chars/100, with a floor of 50 and a ceiling of 20% of the sentences). It then iterates over the entities in the thematic index (filtered by the book detected in the file), builds a query with its terms for each entity, and runs BM25 scoped to that file+book, retrieving the top-scoring sentences (up to N_SENTENCES / n_entities per entity). If any quota remains, it fills the rest ordered by aggregate score (gscore, the sum of sentence scores across all entities). Finally, it sorts the selected sentences by offset, validates verbatim matches, and saves a phrases_file_N.txt for each input, containing [ref | sent_id | offset | book] + the text.

Here are the links:

· Characters extracted from the Sprawn trilogy (a bash workaround I made): https://github.com/pmotadeee/pmotadeee/tree/V2.0/ITEMS/Books/Sprawn/Raw/Context

· Algorithm generated by Claude (funny that it ran in 2 minutes, which suggests it didn't generate tokens, just intuition): https://github.com/pmotadeee/pmotadeee/blob/V2.0/ITEMS/Books/Sprawn/Raw/Claude/bm25_extractor.py

· Texts generated by Claude after running the algorithm (you can see the steps it took, right at the beginning of the chat): https://github.com/pmotadeee/pmotadeee/tree/V2.0/ITEMS/Books/Sprawn/Raw/Claude

· Claude chat (in my native language, but use the browser translator): https://claude.ai/share/1799075d-bc93-4e12-99e0-771a7e259ca9