19.5. Miscellaneous neural areas for different currents

Content and key concepts

On the previous page, the calculation steps on the different devices.
The router rules choose a small number of experts. The weight not used in this step should still be stored.
You can divide data, tables, or steps. Partial results need to be linked. Different patterns first turn into vectors of appropriate width, and experts are selected along a limited route.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Router activates only part of the parameters for each token
Sparse MoE replaces the dense FFN set of exterts and router. In top-1 the token mode goes to one expert with maximum score; total alternatives are growing, but the action alternatives on token can remain close to the same FF N. Switch Transformer explores such spare routing.
So that one expert doesn't get all the tokens, they ask for the capacity. In a local contract for
T
tokens,
E
experience and factor
c
:
c
a
p
a
c
i
t
y
=
⌈
c
T
E
⌉
.
capacity=\left\lceil c\frac{T}{E}\right\rceil.
c
a
p
a
c
i
t
y
=
⌈
c
E
T
​
⌉
.
Overflow policy should be explicit: drop, route or padding. Auxiliary balancing loss affects learning router, but does not guarantee the perfect balance of each batch.
That's what the model shows.
Router view shows the scores selected by expert, capacitity slot and overflow. Total and active parameter counts are produced separately. Expert parallelism requires all-to-all token exchange; basend countes do not mean cheap communication.
Scope of applicability.
MoE is not an ensemble of independent completed models. Exert labels are not obliged to match human themes. Sparse activation reduces part of the aristmetic, but memory for all weights and routing/communication remains; total parameter account cannot be directly compared with dense compute.
from math import ceil
scores_19e=[[.8,.2],[.1,.9],[.7,.3],[.2,.8]]; experts_19e=2
routes_19e=[max(range(experts_19e),key=lambda e_19e:(row_19e[e_19e],-e_19e)) for row_19e in scores_19e]
capacity_19e=ceil(1.0*len(scores_19e)/experts_19e)
loads_19e=[routes_19e.count(e_19e) for e_19e in range(experts_19e)]
assert routes_19e==[0,1,0,1] and loads_19e==[2,2] and capacity_19e==2
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Router and capacity define the active computation; total model storage remains.
Images and sounds also become representations. How does this question continue?
Python examples of Chapter 19
Academic base