How Neural Networks Use GPUs Differently — Compute, Memory & Parallelism

Not every neural network puts the same kind of pressure on a GPU.

Some models spend most of their time calculating. Others spend most of their time waiting for data. Some can run huge amounts of work in parallel. Others are stuck waiting for the previous step to finish.

This post looks at five common architectures — DLRM, MLP, CNN, RNN, and Transformer — through that GPU lens.

Central idea: memorize three dimensions — computation, memory movement, and parallelism — then see how each architecture behaves differently along those dimensions.

1. What Are We Comparing?

Three questions sit behind the whole comparison:

1. Compute-bound or memory-bound? Does the model spend more time performing calculations, or waiting for data from memory?
2. Parallelism: Can the GPU perform many operations at once, or must it wait for previous operations?
3. Memory access and reuse: Does the model repeatedly reuse the same data, or does it keep fetching different data from memory?

Those three ideas explain most of the differences between these workloads.

2. Five Architectures from a GPU Perspective

DLRM — Recommendation Models

Imagine you’re using YouTube.

YouTube needs to decide which video to recommend next.

It has information about:

• You (user ID)
• The video (item ID)
• Your watch history
• Video category
• Your previous interactions
• Other numerical information about your activity

DLRM (Deep Learning Recommendation Model) combines these different types of information to predict a score.

For example:

The model estimates how likely you are to click a particular video.

▸What is DLRM? (short overview)

DLRM (Deep Learning Recommendation Model) is used for ranking and recommendation systems — for example, which ads or products to show a user.

It combines:

• Sparse categorical features (user ID, item ID) looked up from large embedding tables
• Dense features processed by an MLP
• A final MLP that predicts a score (click, conversion, etc.)

The distinctive GPU challenge is not the MLP part — it is the irregular embedding lookups into huge tables.

Tendency: memory-bound

Imagine a recommendation model with 100 million user embeddings.

When a user asks for a recommendation, the GPU needs to fetch specific embedding vectors from memory.

• Lots of data needs to be stored.
• Memory locations accessed are often unpredictable.
• Relatively little computation happens per lookup.
• Data movement becomes expensive.

GPU challenge: fetching data efficiently matters more than raw calculation speed.

For a concrete implementation example, see NVIDIA’s DLRM for PyTorch.

MLP — Dense Neural Network

Imagine an email app deciding whether a message is spam.

The app converts the email into a list of numbers — for example word counts, sender reputation, time of day, and other features.

Those numbers are fed into stacked fully connected layers:

• Every input connects to every neuron in the next layer
• Each layer transforms the representation
• The final layer outputs a probability

For example:

That dense, fully connected structure is an MLP.

▸What is an MLP? (short overview)

An MLP (Multilayer Perceptron) is a classic fully connected neural network: stacked layers where every neuron connects to every neuron in the next layer.

Each layer is basically a matrix multiply:

$$Y = XW + b$$

That maps extremely well onto GPUs, especially with large batch sizes that reuse the same weights across many samples.

Tendency: compute-intensive, often compute-bound

In a multilayer perceptron, every input connects to every output through weights.

The GPU performs large matrix multiplications.

• Many multiplication and addition operations.
• Same weights reused across multiple samples.
• Matrix operations can be executed in parallel.

GPU advantage: GPUs are especially good at matrix multiplication.

Large batches improve weight reuse and can make computation the dominant cost.

Weight reuse example — house prices

Let’s say we’re building an MLP to predict house prices for 3 houses.

Each house has two input features:

• House size
• Number of bedrooms

We want to produce one output: predicted house price.

Each row represents a different house.

Suppose the trained MLP has these weights:

These weights are shared across all three houses.

Look at this:

House 1

• Input: $[1000,\ 2]$
• Same $W$
• Uses weights $5$ and $10$

House 2

• Input: $[1500,\ 3]$
• Same $W$
• Uses weights $5$ and $10$ again

House 3

• Input: $[2000,\ 4]$
• Same $W$
• Uses weights $5$ and $10$ again

Notice that we don’t need a separate weight matrix for each house.

The GPU loads the weight matrix once and uses it to process multiple input rows.

That’s weight reuse.

Imagine processing 1,000 houses:

• The same weight matrix is used 1,000 times.
• Instead of loading completely different weights for every house, the GPU can reuse the same weights.
• Multiple houses can be processed simultaneously through matrix multiplication.

This means more computation can be performed for each time the weights are loaded from memory.

CNN — Convolutional Neural Network

Imagine you open Google Photos and search for “dog.”

The app looks at a photo pixel by pixel, but not by connecting every pixel to every weight. Instead, small filters slide across the image and detect patterns:

• Edges and textures in early layers
• Eyes, fur, and shapes in deeper layers
• “This looks like a dog” at the end

For example:

That sliding-filter approach is a CNN.

▸What is a CNN? (short overview)

A CNN (Convolutional Neural Network) is built for grid-like data such as images.

Instead of connecting every pixel to every weight, a small filter (kernel) slides across the image and detects local patterns — edges, textures, shapes.

Key idea: weight sharing — the same filter is reused across all spatial locations, which cuts parameters and improves data reuse on the GPU.

Tendency: compute-intensive, often compute-bound

The same small filter moves across different image regions.

• Same weights reused many times.
• Different image locations can be processed in parallel.
• Lots of multiplication and addition operations.

GPU advantage: weight sharing and local data reuse make convolution efficient.

However, small or depthwise convolutions can become memory-bound or overhead-bound.

MLP vs CNN — weight sharing

Let’s compare MLP and CNN.

1. In a fully connected network (MLP)

Imagine a 4×4 image containing 16 pixels.

If we connect those 16 pixels to 3 output neurons, every pixel connects to every output neuron.

That means:

• $16 \times 3 = 48$ weights
• Each connection has its own weight

For example, pixel 1 might have weight $0.5$ for neuron 1, but weight $0.8$ for neuron 2.

2. In a CNN

Instead of connecting every pixel to every output neuron, a CNN uses a small filter, such as a 3×3 kernel.

For a short CNN introduction, see

.

The filter contains only 9 weights.

It examines a small region of the image, calculates an output, then moves to another region.

Here’s the important part: the same 9 weights are reused at every position.

The CNN doesn’t learn a completely new set of 9 weights for every image region.

3. Why is this useful?

Imagine detecting a vertical edge.

You want to detect that edge whether it appears:

• At the top-left corner
• In the middle
• At the bottom-right corner

The same filter can detect it in all three locations.

So:

• MLP: different weights for different input-output connections
• CNN: same filter weights reused across different spatial locations

That’s what we mean by weight sharing in CNNs. It reduces the number of parameters and allows efficient reuse of data and computations.

RNN — Recurrent Neural Network

Imagine an old-style phone keyboard predicting the next word as you type.

You type:

I love machine

The model reads one word at a time and keeps a memory of what came before:

• After “I” → memory updates
• After “love” → memory updates again
• After “machine” → it predicts the next word

For example:

Because each step depends on the previous hidden state, an RNN must process the sequence in order.

▸What is an RNN? (short overview)

An RNN (Recurrent Neural Network) processes sequences step by step — text, audio, time series.

At each time $t$, it updates a hidden state $h_t$ using the current input $x_t$ and the previous hidden state $h_{t-1}$:

$$h_t = f(x_t, h_{t-1})$$

That recurrence is why RNNs struggle to parallelize across time: step $t$ must wait for step $t-1$.

Tendency: sequential dependency

An RNN processes a sequence one time step at a time.

• The next step needs the previous hidden state.
• The GPU cannot freely execute all time steps simultaneously.
• More GPU compute units cannot remove this dependency.

GPU challenge: limited parallelism across the sequence.

RNNs can still perform parallel computation across batch elements and within each time step. For a standard walkthrough of the recurrence, see CS231n on RNNs.

Why RNNs exist — variable length and memory

Imagine you’re building a neural network to process sentences.

Consider these two sentences:

• “I love AI”
• “I love learning artificial intelligence”

They have different numbers of words.

In a traditional feedforward neural network (MLP), the input layer has a fixed number of neurons.

For example, if you design an input layer with 3 neurons, it expects 3 input values.

What happens if the input contains 10 words instead?

RNN solves this by processing sequential inputs one at a time.

It can process 3 words, 10 words, or 100 words using the same network parameters.

Word order matters

Suppose we have two sentences:

• “Dog bites man”
• “Man bites dog”

Both contain exactly the same words, but their meanings are different because the order changes.

A traditional MLP doesn’t have a mechanism for remembering what it processed previously.

It receives its input features and processes them through its layers.

It can learn different weights for different input positions, so changing the order can change its output. But it doesn’t inherently maintain a sequential memory of previous inputs.

RNN introduces that memory through something called a hidden state.

Hidden state walkthrough — “I love AI”

An RNN processes the sentence one word at a time.

Input 1: “I”

• RNN processes the first word
• Produces hidden state $h_1$
• $h_1$ is the memory of “I”

Input 2: “love”

• RNN processes both the current input and the previous hidden state $h_1$
• Produces a new hidden state $h_2$
• $h_2$ contains updated information

Input 3: “AI”

• RNN uses the current word and $h_2$
• Produces $h_3$
• $h_3$ carries information from previous steps

Notice what happens:

• First input produces hidden state $h_1$
• Second input uses both the current word and $h_1$ to produce $h_2$
• Third input uses the current word and $h_2$ to produce $h_3$

This is the important difference: the previous output (hidden state) is reused when processing the next input.

The key equation difference

MLP

Only the current input is used.

RNN

Current input + previous hidden state are used.

That’s it. RNN adds the previous hidden state to the normal neural network equation, giving it memory of previous inputs — and that same dependency is why the GPU cannot freely parallelize across time steps.

Parallelism comparison — MLP vs CNN vs RNN

MLP — high parallelism

Multiple input samples can be processed simultaneously.

• Each sample can be processed independently using the same weights.
• Matrix multiplication allows thousands of operations to execute in parallel.
• The same weight matrix is reused across different samples in a batch.

CNN — high parallelism

Multiple image regions can be processed simultaneously using the same convolution filter.

• Instead of connecting every pixel to every output neuron, a CNN uses a small filter, such as a 3×3 kernel.
• The same filter slides across different image locations.
• This reduces the number of parameters and often the computation required compared with fully connected image processing.
• Different spatial positions and output channels can be calculated in parallel.

The key difference from MLP: CNN uses local connections and shared filters, while MLP uses fully connected layers.

RNN — limited parallelism across time

• First, calculate $h_1$.
• Then calculate $h_2$ using $h_1$.
• Then calculate $h_3$ using $h_2$.

You cannot calculate all time steps independently because each step needs the previous hidden state.

The key difference: unlike MLP and CNN, an RNN has a sequential dependency between time steps.

Transformer

Imagine ChatGPT answering your question.

Your prompt is split into tokens:

"Explain" "GPU" "memory"

Unlike an RNN, a Transformer can look at all tokens together during training or encoding. Each token can attend to every other token — for example, relating “memory” back to “GPU.”

When generating the reply, it still writes one token at a time, but it reuses previous Key/Value computations through a KV cache.

For example, after seeing the prompt, it may assign high attention between related words and then generate:

"GPU memory stores tensors..."

That attention-based design is a Transformer.

▸What is a Transformer? (short overview)

A Transformer replaces recurrence with self-attention: every token can look at every other token in the sequence.

During training, that enables excellent parallelism across tokens. The tradeoff is cost — full attention scales about like $O(S^2)$ in sequence length $S$.

During generation, models often store a KV cache so previous keys/values are not recomputed, which grows GPU memory as the sequence gets longer.

Tendency: highly parallel, attention- and memory-intensive

Unlike RNNs, Transformers can process all input tokens in parallel during training.

But every token can interact with every other token.

• Excellent parallelism during training.
• Attention computation grows quadratically with sequence length.
• The KV cache consumes increasing memory during generation.

GPU challenge: handling large attention computations and memory requirements.

Self-attention example — “The cat sat on the mat”

Let’s use a simple example: predicting the next word in the sentence “The cat sat on the mat.”

Imagine the Transformer receives these tokens:

Each token is converted into a numerical representation called an embedding.

Every token looks at the other tokens to understand their relevance.

For example:

• “cat” pays attention to “sat” to understand what the cat is doing.
• “sat” pays attention to “cat” to understand who performed the action.
• “mat” pays attention to “on” to understand the relationship.

The model calculates attention scores between tokens.

Simplified attention score matrix

Illustrative values, not actual model outputs.

Each row represents one token’s attention distribution across the sequence.

The important thing: these attention calculations can be performed in parallel during training. The model doesn’t need to finish processing “The” before processing “cat”.

RNN sequential vs Transformer parallel

Consider the same sentence.

RNN — sequential processing

The → h1
cat → h2
sat → h3
on  → h4
mat → h5

The RNN must calculate the hidden state for “The” before “cat”, then “cat” before “sat”, and so on.

Transformer — parallel processing during training

The   cat   sat   on   mat
 ↓     ↓     ↓    ↓     ↓
   all processed together via self-attention

All token representations can be processed together through self-attention.

That’s why Transformers offer much better parallelism across sequence positions during training.

Why long sequences get expensive

Suppose a sentence has:

• 5 tokens → $5 \times 5 = 25$ attention scores
• 100 tokens → $100 \times 100 = 10{,}000$ scores
• 1,000 tokens → $1{,}000 \times 1{,}000 = 1{,}000{,}000$ scores

So longer sequences create significantly more computation and memory pressure — attention grows like $O(S^2)$.

KV cache during generation

Now imagine the model generates a sentence one token at a time.

It has already processed:

"The cat sat"

Now it needs to generate the next token.

Instead of recalculating the Key and Value representations for “The cat sat”, it stores them in GPU memory. This stored information is called the KV cache.

When it generates “on”, it adds the new token’s Key and Value to the cache. When it generates “the” next, it reuses the previous cache and adds the new ones.

KV cache growth

1. After generating “The” → 1 token stored
2. After generating “cat” → 2 tokens stored
3. After generating “sat” → 3 tokens stored
4. After generating “on” → 4 tokens stored

The cache keeps growing as the generated sequence gets longer.

Why this matters for GPUs: the KV cache occupies GPU memory. Longer conversations and longer generated sequences require more memory to store previous Keys and Values.

3. The Central Comparison

Notice that there are two separate concepts:

• Compute-bound vs memory-bound describes what limits performance.
• Parallelism describes how much work can execute simultaneously.

They are related, but they are not the same thing.

4. Compute-Bound vs Memory-Bound

Compute-bound — waiting for calculations

The GPU already has enough data available, but it needs time to perform all the mathematical operations.

Adding more computational power can improve performance.

Example: a large matrix multiplication.

Memory-bound — waiting for data

The GPU could calculate faster, but it spends time waiting for data to arrive from memory.

Adding more computational units may not solve the problem.

Example: random embedding lookups in DLRM.

A kitchen analogy

Imagine a restaurant kitchen:

• Compute-bound: the kitchen has all the ingredients, but the chef needs time to cook.
• Memory-bound: the chef is fast, but ingredients are not arriving from the storage room quickly enough.

For more background on GPU parallelism, see Dive Into Systems — GPUs.

5. Why Discuss These Five Together?

Because different ML workloads need different hardware optimizations.

This matters because a GPU designed mainly to accelerate matrix multiplication may perform exceptionally well on MLPs, yet still struggle with irregular embedding lookups or sequential RNN dependencies.

6. Test Your Understanding

Question 1

A recommendation model randomly fetches millions of embedding vectors. What is its main bottleneck?

• GPU calculation speed
• Memory access and data movement
• Sequential dependency

Question 2

An RNN processes 100 tokens, where each step depends on the previous hidden state. What limits parallelism?

• Memory capacity
• Dense connectivity
• Sequential dependency

Question 3

A large MLP performs massive matrix multiplications using the same weights across many samples. Which hardware capability helps most?

• Parallel matrix computation
• Random memory lookup
• Sequential execution

Question 4

Why can Transformer attention become expensive for long sequences?

• It cannot process tokens in parallel
• Every token can interact with every other token
• It uses no weight sharing

Final Takeaway

Do not memorize five architectures as five disconnected topics.

Memorize the three dimensions:

computation
memory movement
parallelism

Then ask, for each model:

What is the GPU waiting on?
How much work can run at the same time?
How often is data reused vs freshly fetched?

That is the real story: different neural networks stress GPU hardware differently, so the right optimization depends on the workload.

Q&A — Conceptual, Math & Comparison

Part A: Conceptual Questions

Q1. What is the difference between a compute-bound and a memory-bound workload?

Q2. Why is DLRM considered a memory-intensive workload?

Q3. Why is MLP suitable for GPU parallelism?

Q4. What is weight sharing in CNNs, and why is it important?

Q5. Why are RNNs difficult to parallelize across time steps?

Q6. Why can Transformers process tokens in parallel during training, unlike traditional RNNs?

Q7. Why does Transformer attention become expensive for long sequences?

Q8. What is KV cache, and why is it used in Transformers?

Q9. What is the difference between parallelism and computational intensity?

Part B: Mathematical Questions

Q10. Calculate the total embedding memory required by a DLRM model.

A recommendation model contains:

• Number of embedding entries: 100 million
• Embedding dimension: 128
• Data type: FP32 (4 bytes per value)

Calculate the total embedding table memory.

Q11. Calculate the embedding payload accessed in one DLRM batch.

Suppose:

• Batch size $B = 2048$
• Average embedding rows retrieved per sample $L = 1$
• Embedding dimension $d = 128$
• FP32 values $s = 4$ bytes

Calculate the total embedding data accessed.

Q12. Calculate MACs and FLOPs for an MLP layer.

An MLP has:

• Batch size $B = 64$
• Input features $d_{in} = 512$
• Output features $d_{out} = 256$

Calculate the total MACs and FLOPs.

Q13. Calculate the weight memory required by an MLP layer.

Suppose an MLP has 1024 input features and 512 output features. Each weight uses FP32 (4 bytes).

Q14. Calculate the number of MACs in a CNN convolution layer.

Given:

• Batch size $B = 1$
• Output height $H_{out} = 32$
• Output width $W_{out} = 32$
• Kernel size $K = 3$
• Input channels $C_{in} = 16$
• Output channels $C_{out} = 32$

Calculate the total MACs.

Q15. Calculate the approximate MACs of a vanilla RNN.

Given:

• Batch size $B = 32$
• Sequence length $S = 100$
• Input dimension $d_{in} = 64$
• Hidden dimension $d_h = 128$

Calculate the approximate number of MACs.

Q16. How does Transformer attention complexity change when sequence length doubles?

Suppose a Transformer processes 1,000 tokens. Its sequence length increases to 2,000 tokens.

Q17. Calculate the KV cache memory required by a Transformer.

Suppose:

• Batch size $B = 2$
• Sequence length $S = 1000$
• Model dimension $d = 512$
• FP16 data type (2 bytes per value)

Calculate the KV cache memory per layer using standard multi-head attention.

Part C: Analytical and Comparison Questions

Q18. Compare the five architectures in terms of their primary GPU bottlenecks.

Q19. What happens when batch size increases in an MLP?

Q20. Can adding more GPU cores completely solve the sequential dependency problem in RNNs?

Q21. Which architecture benefits most from weight sharing, and why?

Q22. Why might a GPU perform well on MLP workloads but struggle with DLRM workloads?

Final Revision: Formulas to Remember