19.4. Calculation stages on different devices

Content and key concepts

On the previous page, one table is divided into parts.
The first step transmits the result to the next step. When the circuit is filled and released, some devices wait temporarily.
You can divide data, tables, or steps. Partial results need to be linked. Different patterns first turn into vectors of appropriate width, and experts are selected along a limited route.
Для изменения рисунка включитеJavaScript; расчёт и теория доступны далее.
Layers become stages, batch is a stream of micro-batches
===Pipeline paralle===Papeline parallel places successive layers groups on different devises. Micro-batches pass through steps as parts of the conveyor. While the pipeline fills up and emptys, part of the steps stays idle, the bubble arises.
For simplified forward-only calendar with
p
equals and
m
Microsoft-batches required
m+p-1
ticks, and the share of employment in the same stage:
U
=
m
m
+
p
−
1
.
U=\frac{m}{m+p-1}.
U
=
m
+
p
−
1
m
​
.
Increase
m
It reduces the relative bubble but stores more in-flight activities and can change the batch semantics. Real tranying classes add backwards and different versions of 1F1B/interleaving.
That's what the model shows.
The Gant view shows diagonal wave, empty bubble bells and action gears. The balance of the stages is more important than the same number of layers: different layer can have different values.
Scope of applicability.
The formula does not include backward, recomputation, network overlap and imbalance.Pipeline increases the latincy of a separate micro-batch and complicates the performance review. Activation transporter can become a bottleneck even with equal compute times.
stages_19d,microbatches_19d=3,4
ticks_19d=microbatches_19d+stages_19d-1
utilization_19d=microbatches_19d/ticks_19d
assert ticks_19d==6 and abs(utilization_19d-2/3)<1e-12
The code repeats the operations just taken care of.
Completely checked Python-example Chapter
Available with a separate file; the short piece shown does not replace its implementation.
Microbatches can't show up at the same time in all steps.
Different neural areas for different currents. How does this question continue?
Python examples of Chapter 19
Academic base