16.5 Learn from Two Responses

Content and key concepts

On the previous page: educational examples of the question and answer.
The data may say which of the two answers preferred: this is a separate learning criterion, not proof of the truth of the answer chosen.
Each known electron produces a projection of the following. The error goes backwards on the program, the optimizer changes the weight, and the individual data checks the result. The dialog and preferences mark changes the objective.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Compare the answers to the question of return policy
Preference dataet contains a prompt and a couple of answers: chosen/projected for some marking process. DOC uses the difference log-ratios of current and reference policies and binary logic loss. Positive margin means that the policy has strengthened the chosens relative to reacted more than reference. F03, F04.
Δ
=
[
log
⁡
π
θ
(
y
+
∣
x
)
−
log
⁡
π
θ
(
y
−
∣
x
)
]
−
[
log
⁡
π
r
e
f
(
y
+
∣
x
)
−
log
⁡
π
r
e
f
(
y
−
∣
x
)
]
,
L
D
P
O
=
−
log
⁡
σ
(
β
Δ
)
.
\Delta=\left[\log\pi_\theta(y^+|x)-\log\pi_\theta(y^-|x)\right]
-\left[\log\pi_{ref}(y^+|x)-\log\pi_{ref}(y^-|x)\right],
\quad L_{DPO}=-\log\sigma(\beta\Delta).
Δ
=
[
lo
g
π
θ
​
(
y
+
∣
x
)
−
lo
g
π
θ
​
(
y
−
∣
x
)
]
−
[
lo
g
π
re
f
​
(
y
+
∣
x
)
−
lo
g
π
re
f
​
(
y
−
∣
x
)
]
,
L
D
PO
​
=
−
lo
g
σ
(
β
Δ
)
.
Log-probability of the answer is the sum of massed token log-probabyties under the same prompt/template. Reference frozen. The parameter beta scalees the preference; it is not a "percentage of humanity". RLHF with reward model and policy acceptance is another pipeline: it is impossible to add the whole signal to one DPO formula.
That's what the model shows.
The model shows four sequence log-probeities, two internal policy differences and final margin. The simple probability of a hosen in policy is not sufficient: objective estimates the relative change against reference. Tis, diagnosis annotators and position bias should remain observed metadata.
Scope of applicability.
Preference is not equal to the truth and can conflict between users. Pairwise dataset does not cover the rare risks well, and reward hacking is possible without a clear reward model. Offline improvisation on preference loss does not replace red-team, actual evaluation and monitoring of real distribution shif.
from math import exp,log
policy_chosen_16e,policy_rejected_16e=-1.2,-2.0
ref_chosen_16e,ref_rejected_16e,beta_16e=-1.0,-1.5,0.5
margin_16e=(policy_chosen_16e-policy_rejected_16e)-(ref_chosen_16e-ref_rejected_16e)
pair_prob_16e=1/(1+exp(-beta_16e*margin_16e)); dpo_loss_16e=-log(pair_prob_16e)
assert abs(margin_16e-0.3)<1e-12 and abs(dpo_loss_16e-0.6209570477895321)<1e-12
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
"Pairwise loss" is being brought into line with the observed chosen/projected labels and frozen reference contract.
A small addition to the large table. How does this question continue?
Python examples of Chapter 16
Academic base