18.2 Comparison and selection of fragments

Content and key concepts

On the previous page, first we find the parts of the document.
We'll compute the proximity of the question and the texts, sort the estimates, and show the selected sources.
The search selects fragments; the beilder adds them to the question. The memory, history and tool results have different sources. The right of action checks the program, and the cycle has to have a reason for stopping.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Query and chunks are compared in one search space.
Dense retriever encodes query and documents in vectors and rankings similarity; lexical retriever uses the term and the statistics of the collection. Cosine removes the influence of the norm and compares direction. Zero vector has no cosine similariness and should give a clear error or separate policy rather than a silent score0.
cos
⁡
(
q
,
d
)
=
q
T
d
∥
q
∥
2
∥
d
∥
2
,
rank
⁡
=
argsort
⁡
d
(
−
cos
⁡
(
q
,
d
)
)
.
\operatorname{cos}(q,d)=\frac{q^Td}{\|q\|_2\|d\|_2},
\qquad \operatorname{rank}=\operatorname{argsort}_{d}(-\operatorname{cos}(q,d)).
cos
(
q
,
d
)
=
∥
q
∥
2
​
∥
d
∥
2
​
q
T
d
​
,
rank
=
argsort
d
​
(
−
cos
(
q
,
d
))
.
On a small set, the full set is accurate. Approximate neighbour index changes latincy/recall trade-off and requires its own evolution. Metadata filters, access policy and freshness are used as part of a retrieval contract.
That's what the model shows.
Scatterplot is just a projection: Franking is considered as a full vector space. The Inspector shows raw query, filters, scores, tie-break and re-acted candidates. Retrieval qualiity is measured separately, e.g. recall@k on querey/relevant pairs, before the generator's evaluation.
Scope of applicability.
High cosine does not mean truth or entailment. Embedding model can work poorly with a new domain, language or numbers. Index only knows a loaded version of the collection; the absence of a document in top-k does not prove that there is no fact in the sources.
from math import sqrt
query_18b=[1.0,1.0]; docs_18b={'d1':[1.0,0.0],'d2':[1.0,1.0],'d3':[-1.0,0.0]}
def cosine_18b(a_18b,b_18b):
return sum(x*y for x,y in zip(a_18b,b_18b))/sqrt(sum(x*x for x in a_18b)*sum(y*y for y in b_18b))
ranking_18b=sorted(docs_18b,key=lambda key_18b:(-cosine_18b(query_18b,docs_18b[key_18b]),key_18b))
assert ranking_18b==['d2','d1','d3']
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
The source remains an external witness and may be obsolete or erroneous.
What did the model really see, how does this question go on?
Python examples of Chapter 18
Academic base