Dot Product Application

NN from Scratch

Every neural network computation boils down to dot products. np.dot handles three scenarios — vector·vector, matrix·vector, and matrix·matrix — and knowing the difference is key to understanding how data flows through a network.

1. Vector × Vector (Single Neuron)

np.dot(w, x) multiplies element-wise and sums: w₁x₁ + w₂x₂ + w₃x₃. This is exactly what one neuron computes before adding bias.

For two vectors, the dot product is commutative: np.dot(w, x) == np.dot(x, w). The result is always a single scalar.

Drag the sliders to change x — watch the dot product update
w [3]
0.20
0.80
-0.50
·
x [3]
1.00
2.00
3.00
=
scalar
0.300
Computing w·x
w
0.20
0.80
-0.50
·
x
1.00
2.00
3.00
0.20 × 1.00 = 0.200+0.80 × 2.00 = 1.600+-0.50 × 3.00 = -1.500
0.200 + 1.600 + -1.500=0.300→ w·x
w·x + b = 0.300 + 0.10 = 0.400
x1
1.00
x2
2.00
x3
3.00
Python
Shift+Enter to run
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

2. Matrix × Vector (Layer)

When W is a matrix (rows = neurons), each row of W dots with the input vector to produce one output. This computes an entire layer at once.

Order matters now — non-commutative. np.dot(W, x) is correct (each row dots with x). np.dot(x, W) gives a different result — it treats x as a row vector multiplying columns.

Click an output cell to see that row's computation
W [3×3]
0.20
0.80
-0.50
0.50
-0.91
0.26
-0.26
-0.27
0.17
×
x [3]
1.00
2.00
3.00
+
b [3]
0.10
-0.20
0.05
=
z [3]
0.400
-0.740
-0.240
Computing z[0]
row 0 of W
0.20
0.80
-0.50
·
x
1.00
2.00
3.00
0.20 × 1.00 = 0.200+0.80 × 2.00 = 1.600+-0.50 × 3.00 = -1.500
0.200 + 1.600 + -1.500 + 0.10=0.400→ z[0]
x1
1.00
x2
2.00
x3
3.00
Python
Shift+Enter to run
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

3. Matrix × Matrix (Batch)

In practice we process a batch of samples at once. Each row in X is one sample. We need the transpose: np.dot(X, W.T) + b.

This is full matrix multiplication — each sample's row dots with each neuron's weights (columns of WT). The GPU processes all samples in parallel.

Click any output cell to see its computation
(3,3) · (3,3)T + b → (3,3)
X [3×3]
1.00
2.00
3.00
2.00
5.00
-1.00
-1.50
2.70
3.30
×
WT [3×3]
0.20
0.50
-0.26
0.80
-0.91
-0.27
-0.50
0.26
0.17
=
C [3×3]
0.400
-0.740
-0.240
5.000
-4.010
-1.990
0.310
-2.549
0.272
Computing C[0,0]
row 0 of X
1.00
2.00
3.00
·
col 0 of W<sup>T</sup>
0.20
0.80
-0.50
1.00 × 0.20 = 0.200+2.00 × 0.80 = 1.600+3.00 × -0.50 = -1.500
0.200 + 1.600 + -1.500 + 0.10=0.400→ C[0,0]
Python
Shift+Enter to run
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

Three Levels of Dot Product

Vector · Vector — multiply matching pairs and add them up. One neuron does exactly this to produce a single number.

Matrix · Vector — each row of the weight matrix dots with the input vector. One operation computes an entire layer of neurons at once.

Matrix · Matrix — every sample in the batch gets its own row-by-column dot product. The GPU processes all samples in parallel — that's why neural networks are fast.