Entry and transformation rule

Content and key concepts

On the previous page, the model is not the whole program.
If the number changes at the entrance, the answer can change. This doesn't mean the rule has been learned.
The model calculates the answer using numerical settings. The program around it prepares the entrance and can trigger a search or a tool.
Where does the weight come from?
Weight (weight) is the generic model coefficient. This is not a word number or answer to neuronets. Weights are stored in tables; the same number can be used to calculate different currents.
In creating the model, the program needs initial values. Many weights are automatically generated by small random numbers; some multipliers and offsets start with units or zeros. Such settings are not yet useful knowledge.
How a trained model is produced from the initial numbers
The model makes a projection of the study example.
The prognosis is compared to the known correct answer.
The difference is estimated by the number — the error or the loss function (loss).
The derivatives show how minor parameter changes will affect the error; their set is called the gradient.
The rule of correction -- the optimist -- changes the parameters a little bit, and the calculation is repeated.
After many of these corrections, the weights can be useful for the task. This is not a quality assurance for new data: they are checked separately. The following chapters of tables E, Wq and U contain the same settings.
Для интерактивного эксперимента нужен JavaScript. Теория, модель и ручные расчёты выше доступны без него.
Four objects that can't be mixed
Consider the Family of Functions
y
=
f
(
w
,
b
)
(
x
)
=
w
x
+
b
.
y=f_{(w,b)}(x)=wx+b.
y
=
f
(
w
,
b
)
​
(
x
)
=
w
x
+
b
.
Here.
architecture
– The order of operations "to multiply, then to add".
Parameters
θ=(w,b)
choose a specific family function.
Input
x
is relevant to the current example.
Activation
y
These objects may be represented by numbers, but their role varies.
Prix
w=1
,
b=0
,
x=2
received
y=2
. If only replaced
x
By 3, the dot will pass with
(2,2)
as at
(3,3)
If you save it, you'll have to keep it on the same line.
x=2
but replace
w
by 2, you get a new function.
y=2x
In both experiments, the answer was changed, but only in the second was the parameter state changed.
Status
x
w
b
y
Reason for change
Basic
2
1
0
2
None
A New Example
3
1
0
3
Other input
New Slope
2
2
0
4
Other parameter
w
New Offset
2
1
1
3
Other parameter
b
Affinal conversion geometry
Ratio
w
causes a change in exit per unit of change in inlet:
f
(
x
+
Δ
x
)
−
f
(
x
)
=
w
Δ
x
.
f(x+\Delta x)-f(x)=w\Delta x.
f
(
x
+
Δ
x
)
−
f
(
x
)
=
w
Δ
x
.
Bias
b=f(0)
It's shifting the line.
b≠0
Conversion is strictly affinity rather than linear: linear display requires
f(0)=0
The traditional term "line regression" is retained.
The effect of the parameter depends on the input:
∂
y
∂
w
=
x
,
∂
y
∂
b
=
1.
\frac{\partial y}{\partial w}=x,\qquad
\frac{\partial y}{\partial b}=1.
∂
w
∂
y
​
=
x
,
∂
b
∂
y
​
=
1.
Prix
x=0
change
w
does not change the current output, although the parameter store itself changes. So the parameter cannot always be assigned the same visible effect in any example.
From scalar to vectors and shapes
If an object has two signs, use a vector
x=[x₁,x₂]
and webs
w=[w₁,w₂]
:
y
=
w
⋅
x
+
b
=
w
1
x
1
+
w
2
x
2
+
b
.
y=w\cdot x+b=w_1x_1+w_2x_2+b.
y
=
w
⋅
x
+
b
=
w
1
​
x
1
​
+
w
2
​
x
2
​
+
b
.
For example,
[1,2]·[3,4]=1·3+2·4=11
. Shapes compatibility is part of the meaning of the operation: dot takes two vectors of the same length and returns the scalar.
n
Examples of a matrix
X
can have shape
[n,d]
vector
w
—
[d]
and preferences -
[n]
.
The big neurostudent repeats matrix and coordinated operations many times. This does not translate each internal coordinate into a human concept. The number within the activity can participate in a useful distributed representation without a permanent signature of "animal" or "grammatical".
Why the Non-linearity?
Composition of affinity change without non-linearity again affinn:
W
2
(
W
1
x
+
b
1
)
+
b
2
=
(
W
2
W
1
)
x
+
(
W
2
b
1
+
b
2
)
.
W_2(W_1x+b_1)+b_2=(W_2W_1)x+(W_2b_1+b_2).
W
2
​
(
W
1
​
x
+
b
1
​
)
+
b
2
​
=
(
W
2
​
W
1
​
)
x
+
(
W
2
​
b
1
​
+
b
2
​
)
.
Enclosure
ReLU(r)=max(0,r)
Between layers it breaks this folding. For vector, the rule is applied in a coordinated manner:
ReLU([-2,3])=[0,3]
Non-linearity expands the class of functions, but does not in itself provide training, synthesis or the truth of the answer.
Dataset, Target and Prevention
Dataset is a set of examples for learning or evaluation.
(x,y*)
where
y*
- A known target.
f_θ(x)
is calculated by model. Target is not an activation, and dataset is not a folder that each forward is required to read again.
During inference, the same parameters are consistently processed by different inputs. The time activities are created and removed and the webs remain fixed until a separate procedure is completed update.
x
,
w
and
b
Separate: Watch not only the new
y
but also whether the whole line has changed.
Same machine in code.
First the multiplication, then the shift, there's no learning or hidden auxiliary functions.
x, w, b = 2, 1, 0
y = w * x + b
print(y) # 2
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
With a fixed w and b, it's the same function used for another example. The change in the answer does not prove learning.
When a model really learns, how does this question go on?
Academic base