09.6 The program calculates derivatives

Content and key concepts

On the previous page, one weight is used several times.
Automatic differentiation retains the connections of action and applies the local rules already studied; it does not guess the gradient or select an optimist.
And we're going to save the behavior and we'll go back to the error, and the deposits of the different paths to the total weight add up. It's an automatic differentiation, not a new way to guess the direction.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
The rules produce a functioning system
Value stores data, grad and edges with local derivatives. Add/mul creates new objects. Backward builds order, purifies gradiants, seeditizes output unit and aggregates contributions.
This is a really limited AD engine, not a table of pre-set gradiants. The browser is supported by a tang/relu and a serealized stepway. Python example can be downloaded and run on its own.
Check.
Compare the gradient to the central difference [L(α+h) to L(α-h)]/(2h) at h=1e-5 for each coordinate. Avoid the non-distinguishability points of ReLU: the difference of 0 does not have to coincide with the convention. The numerical tests check the implementation on the fixtures but do not replace the formal evidence of the entire program.
It's not supported by tensor Broadcasting, high-order grafts, in-place mutation and GPU. This is a transparent basis that can be expanded after testing the invariants.
the physical execution of calculations
so far, the course in the program.
class Value:
def __init__(self,data,parents=()):
self.data=float(data); self.grad=0.0; self.parents=list(parents)
def __add__(self,other):
return Value(self.data+other.data,[(self,1.0),(other,1.0)])
def __mul__(self,other):
return Value(self.data*other.data,[(self,other.data),(other,self.data)])
def backward(self):
# Полный исполняемый пример — по ссылке ниже.
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
It's an obvious contract of engine; other frameworks can accumulate between challenges.
How does a computer store numbers and perform so many similar operations?
Python examples of Chapter 09
Academic base