Pruning RAG context down to what the answer actually needs

rooftopzen

Cliche topic - from a few years ago (the "RAG is dead" vs "All You Need Is Advanced RAG" BS - it came in waves and cycles, spread by bots on social media networks). "Pruning RAG Context" is trying to recycle the old stuff (again), presuming the reader is naive (implies kapa.ai is not going anywhere). The current cycles were "openclaw" (I think that died), now we are on "harnesses" - when that dies the paid social media bots will give you something else. Shell game. Just declare / define dictionary as a variable in your prompt to carry forward (when you decide to continue using LLMs for certain things). Also either summarize or truncate history. 3-4 year old concept. Not a big thing.

agentdev001

Am I wrong to be somewhat peeved by the use of "RAG" in these contexts? I always read things like this, and wonder if instead the author should be saying "Semantic Retrieval" or something something Vector, etc. Retrieval augmented generation captures tool-use, and; semantic search of course is really just a tool under the hood. To make an anology, in my mind, this is akin to saying "fuel air mixture system" when referring to direct fuel injection specifically, when of course, a carburetor also lives in that category.

esafak

tl,dr: They used a rubric to have the LLM grade the chunks on a Likert scale. I think this is a good way to coax numbers out of an LLM.

StackOptimist

The risk in relevance-based pruning is the same one summarization has: it's tuned to drop whatever's rare, and in a lot of domains the rare chunk is the whole answer. A contraindication, an "except when," the single row that contradicts the other forty. Those score low against the query precisely because they're phrased in the exception's language, not the question's, so a similarity cut throws them out first. The strongest version of the pro-pruning case is "prune by relevance to the query, not blindly," and I still don't trust it where being wrong is expensive, because relevance-to-the-query is exactly the signal that misses the buried caveat. If I ask whether drug A interacts with drug B and the warning is written under B's contraindications without naming A, semantic pruning helps me lose it. What's worked better for me is pruning by structure instead of by score: keep whole records or whole sections as units and drop by type (this entire category of source isn't relevant) rather than by ranking individual sentences. You give up some of the token savings of aggressive sentence-level pruning, but you stop silently deleting the one clause the answer depended on. If you must prune fine-grained, the honest test is recall on a set where the answer hinges on a rare chunk, not average-case QA where there's lots of redundancy to spare.