Alterna · learning material
07.5. Why one line is not sufficient
Some groups of points cannot be separated by one line. Non-linear intermediates allow for a more flexible solution.
What this material covers
- Deep doesn't create non-linearity itself.
Content and key concepts
On the previous page, non-linear rules of conversion.
Some groups of points cannot be separated by one line. Non-linear intermediates allow for a more flexible solution.
He collects inputs, adds a shift, and applies a non-linear rule. All parts of this calculation are already available in school arithmetic.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Deep doesn't create non-linearity itself.
The non-activated composition of two affinity layers is folded into one:
W
2
(
W
1
x
+
b
1
)
+
b
2
=
(
W
2
W
1
)
x
+
(
W
2
b
1
+
b
2
)
.
W_2(W_1x+b_1)+b_2=(W_2W_1)x+(W_2b_1+b_2).
W
2
(
W
1
x
+
b
1
)
+
b
2
=
(
W
2
W
1
)
x
+
(
W
2
b
1
+
b
2
)
.
XOR at points (0.0),(0.1),(1.0),(1.1) has answers 0.1,1.0. Two diagonal pairs do not share the same line. In order to obtain a suitable view, insert two non-linear hidden topics (N02 §6.1):
s
=
x
1
+
x
2
,
h
1
=
ReLU
(
s
)
,
h
2
=
ReLU
(
s
−
1
)
,
y
^
=
h
1
−
2
h
2
.
s=x_1+x_2,\quad h_1=\operatorname{ReLU}(s),\quad
h_2=\operatorname{ReLU}(s-1),\quad \hat y=h_1-2h_2.
s
=
x
1
+
x
2
,
h
1
=
ReLU
(
s
)
,
h
2
=
ReLU
(
s
−
1
)
,
y
^
=
h
1
−
2
h
2
.
At s = 0.2, the hidden pairs are equal to [0.0], [1.0],[2.1] and exits 0.1.0, therefore all four vertices are treated correctly.
Manually appointed
not received in-service training.
Experiment.
Disable the ReLU by keeping the weights: now output 2-s, i.e. [2.1,1.0]. Some dots coincide, but the function is no longer XOR. This is a counter-example to the conclusion of "nothing has changed" made at one successful point.
Linear output is useful after a non-linear representation. However, a manual example does not prove geneticization outside the four tops and does not guarantee that the GD will find exactly these whole weights.
MLP
and separately implement the present training.
outputs=[]
for x1,x2 in [(0,0),(0,1),(1,0),(1,1)]:
s=x1+x2
outputs.append(max(0,s)-2*max(0,s-1))
assert outputs==[0,1,1,0]
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
The production of matrices and the new bias do not create a non-linear family.
What can a few of these elements do that one can't?
Python examples of Chapter 07
Academic base