Continuous Thought Machines (CTM): What If Neural Networks Could Think More Than Once?

Understanding a neural network architecture that uses neuron activity histories and synchronization to perform iterative computation.

Introduction

Most neural networks follow a relatively straightforward process: they receive an input, perform computations through their layers, and produce an output.

For example, when we give an image to a CNN, it processes the image and predicts what it contains. A Transformer processes tokens and uses attention to understand relationships between them. An RNN processes sequential information while maintaining a hidden state.

These architectures are powerful, but they approach computation differently from the way biological brains are often described. In particular, traditional neural networks generally do not make the individual activity histories of their neurons a central part of their computation.

What if a neural network could repeatedly update its internal state, observe how its neurons behave over time, and use relationships between their activity patterns to guide further computation?

This is the central idea behind the Continuous Thought Machine (CTM), introduced by researchers at Sakana AI.

CTM explores a different approach to neural computation: instead of relying only on the current values of neurons, it uses their activity histories and synchronization patterns as part of the model’s internal representation.

Let’s understand how it works.

1. The Motivation: Moving Beyond a Single Computational Pass

Consider an image classification task.

We provide an image of a cat to a neural network. The network processes the image and produces a prediction:

Input: An image of a cat

Output: Cat

For many tasks, this approach works extremely well. A CNN can extract visual features, and a classifier can use those features to make a prediction.

However, not every problem is equally straightforward.

Imagine solving a maze. You need to understand your current position, consider possible paths, remember previous decisions, and potentially change your direction when you encounter an obstacle.

A single prediction may not be enough to solve such a problem effectively.

This introduces an interesting question:

Can a neural network continue processing its internal information even when it does not receive a new input?

CTM explores this possibility through repeated internal computation.

Instead of treating computation as one continuous journey from input to output, CTM divides it into multiple internal steps called ticks.

At each tick, the model updates its internal state. The updated state influences the next computation, allowing the network to refine its internal representation over time.

Importantly, the input does not need to be supplied again at every tick. The model can continue working with its existing internal information.

2. What Makes CTM Different?

The architecture introduces two central ideas:

Neuron-Level Models (NLMs): Individual neurons process their own activity histories.
Neural Synchronization: Relationships between the activity histories of different neurons become part of the model’s internal representation.

These two ideas work together.

Neuron-Level Models: Every Neuron Has Its Own History

In a conventional neural network, neurons produce activation values as information passes through the network.

CTM takes this further.

Imagine a network with three neurons. Across several internal ticks, they produce different activation values:

Internal tick	Neuron A	Neuron B	Neuron C
1	0.2	0.8	0.1
2	0.5	0.6	0.3
3	0.9	0.2	0.7

Each neuron has its own activity pattern.

Neuron A is becoming more active, neuron B is becoming less active, and neuron C follows another pattern.

A conventional approach might primarily use the current activation values. CTM also considers how each neuron has behaved over recent internal ticks.

To achieve this, CTM introduces a small neural network for each individual neuron. These are called Neuron-Level Models (NLMs).

Each NLM receives the recent history of its corresponding neuron’s pre-activation values and produces its next post-activation value.

In simple terms:

Neuron A has its own activity history.
Neuron B has its own activity history.
Neuron C has its own activity history.
Each neuron uses its own small model to process that history.

This allows the model to capture temporal patterns at the individual-neuron level.

The important distinction is that CTM does not simply give the entire network one shared history. It models the recent history of each neuron separately.

Neural Synchronization: Understanding Relationships Between Neurons

Now comes the second major idea.

Knowing what each neuron has been doing is useful. But what if the relationships between neurons also contain important information?

Consider three people clapping.

Person A and Person B follow a similar rhythm, while Person C follows a different rhythm.

If we observe each person independently, we understand their individual behavior. But if we compare their rhythms, we discover additional information about how they behave in relation to one another.

CTM applies a related concept to neurons.

It examines the activity histories of different neurons and calculates relationships between them.

For example:

Neuron A and B may have similar activity patterns.
Neuron A and C may have a different relationship.
Neuron B and C may show another pattern.

The paper calls this process neural synchronization.

Technically, CTM calculates dot products between the temporal activity histories of neuron pairs. These calculations produce a synchronization representation.

This representation contains information about relationships between neurons over time.

It is important to understand that synchronization here does not simply mean that two neurons activate at exactly the same moment. It is a mathematical comparison of their activity histories.

This is one of the most distinctive parts of CTM: the relationships between neuron histories are used directly as a learned internal representation, rather than being treated only as something to inspect after training.

3. How Does the CTM Architecture Work?

Let’s connect these concepts and understand the complete architecture.

Continuous Thought Machine architecture diagram showing input flowing through neuron-level models, neural synchronization, attention readout, and a loop of internal ticks to produce an output

CTM contains several important components.

Step 1: Input Feature Extraction

The model first receives an input, such as an image.

For image classification, the paper uses a ResNet feature extractor to convert the image into useful features.

These features are used to create keys and values for an attention mechanism.

You can think of this stage as preparing the input information so the model can use it during its internal computation.

Step 2: Attention

CTM uses attention to retrieve relevant information from the input features.

The attention mechanism receives a query derived from the model’s current action-synchronization representation.

It then uses this query to calculate attention over the input keys and values.

The result is an attention output that provides information from the original input to the next computational step.

Step 3: The Synapse Model

The next component is the synapse model.

It receives two things:

The previous post-activation state
The output from the attention mechanism

It combines this information and produces new pre-activation values.

In simple terms, the synapse model decides what information should be sent to the individual neurons for their next update.

The paper uses a recurrent model for this component.

Step 4: Neuron-Level Models

The pre-activation values are stored in a rolling history.

Each neuron-level model processes the recent history belonging to its corresponding neuron.

It then produces the neuron’s next post-activation value.

The collection of these values becomes the updated neural state.

Step 5: Neural Synchronization

CTM stores the post-activation histories and calculates synchronization between selected neuron pairs.

The resulting synchronization representation is used in two important ways:

Action synchronization: Helps generate the next attention query.
Output synchronization: Helps generate the model’s prediction.

This distinction matters because the model needs both a representation for continuing its computation and a representation for producing an output.

Step 6: Repeat the Internal Computation

The updated neural state is used in the next internal tick.

The model repeats the process:

Retrieve information through attention.
Produce pre-activations through the synapse model.
Update individual neurons through their NLMs.
Calculate synchronization.
Produce updated representations.

This continues for the configured number of internal ticks.

The model can therefore refine its internal computation over time before producing its prediction.

4. Understanding the Architecture Through a Simple Flow

The complete process can be summarized as follows:

Input → Feature Extraction → Attention → Synapse Model → Neuron-Level Models → Synchronization → Updated Internal State

The synchronization representation then supports two paths:

It helps produce the next attention query.
It helps produce the output prediction.

The model repeats its internal computation across multiple ticks.

The most important thing to remember is that CTM is not simply a network that repeats the same calculation. Its neuron states evolve, their activity histories change, and the synchronization representation changes as well.

5. What Are Internal Ticks?

An internal tick represents one step of computation inside CTM.

Suppose a model performs five internal ticks:

Tick	What happens
1	Initial internal processing
2	Neurons update their activity
3	Relationships between neuron histories are recalculated
4	The model updates its internal representation
5	The model produces another prediction

The model does not necessarily need to wait until the final tick to produce a prediction. It can produce predictions during its internal computation.

This leads to another interesting feature: adaptive computation.

CTM uses prediction loss and prediction certainty across internal ticks to select its training objectives. In particular, the paper selects a tick with minimum loss and another with maximum certainty, then averages their losses.

This encourages the model to develop useful predictions at different points in its internal computation.

It does not use a separate explicit halting module to decide when to stop thinking.

However, more internal ticks do not always mean better predictions. The authors observed cases where the model initially produced a correct prediction and later changed it to an incorrect one.

In other words, additional computation can sometimes hurt.

6. Where Can CTM Be Useful?

CTM is interesting because its design may be useful for tasks that benefit from iterative computation, temporal information, and evolving internal representations.

The researchers investigated several different tasks.

Maze Solving

A maze is a useful example of a problem requiring multiple computational steps.

The model needs to navigate through a maze, consider possible routes, and reach a destination.

The researchers tested CTM on 2D maze environments and compared it with feed-forward and LSTM baselines.

CTM performed better than the trained baselines in their experiments.

They also investigated generalization to longer routes and larger mazes.

This experiment demonstrates how CTM can use repeated internal computation for a task that requires more than a straightforward input-to-output mapping.

Image Classification

The researchers also tested CTM on ImageNet-1K.

For this experiment, they used ResNet-152 as the feature extractor and allowed CTM to perform multiple internal ticks.

One interesting observation was that the model’s attention patterns could change over time.

For example, attention could initially cover broader image regions and later become more focused.

This suggests that CTM can develop changing internal processing patterns rather than relying on one fixed attention pattern.

The authors also observed that additional thinking could sometimes make predictions worse.

Parity Problems

Parity is a mathematical task where the model must determine whether the number of negative or positive values in a sequence satisfies a particular parity condition.

The researchers used a sequence of values and asked the model to predict cumulative parity at different positions.

Interestingly, CTM learned different computational strategies across training runs. Some models learned to process the sequence forward, while others learned to process it in reverse.

This experiment provides evidence that different internal computation strategies can emerge from the architecture.

Sorting

The researchers also explored sorting.

The model received a sequence of real numbers and had to produce their sorted indices sequentially.

An interesting observation was that the model learned to use different waiting periods before producing outputs.

These waiting periods varied depending on the sequence position and the difference between values.

This is another example of adaptive internal computation emerging from the model.

Question Answering with MNIST

One of the more interesting experiments involved using MNIST digit images in a question-answering task.

The model received digit images and then had to answer questions involving indices and mathematical operations.

The experiment investigated whether synchronization could help the model organize and retrieve information during longer computations.

In the hardest tested in-distribution setting, the multi-tick CTM performed substantially better than the multi-tick LSTM baseline.

However, the single-tick LSTM outperformed CTM.

The authors therefore do not claim that CTM can solve problems that LSTMs fundamentally cannot solve. Instead, their results suggest that synchronization can be beneficial for scaling computation in this particular setup.

7. CTM vs CNN, RNN, and Transformer

Since CNNs, RNNs, and Transformers are already widely used, it is useful to understand how CTM differs from them.

Architecture	Main idea
CNN	Learns spatial patterns using convolution operations.
RNN	Processes sequences while maintaining a recurrent hidden state.
Transformer	Uses attention to model relationships between tokens or other input elements.
CTM	Uses individual neuron activity histories and synchronization between neurons as part of iterative computation.

The important distinction is not that CNNs, RNNs, or Transformers cannot perform multiple computational steps.

They certainly can.

Rather, CTM introduces a different way of representing and updating internal information.

It makes neuron-level temporal histories and relationships between those histories foundational parts of the architecture.

8. When Might CTM Not Be the Right Choice?

Despite its interesting ideas, CTM is not automatically the best architecture for every problem.

Computational Cost

Repeated internal computation requires additional processing.

CTM also needs to maintain histories of neuron activity and calculate synchronization between selected neuron pairs.

This introduces additional computational and memory requirements.

Simple Problems May Not Need It

For straightforward tasks such as basic image classification or numerical transformations, a conventional architecture may already provide excellent results.

Adding multiple internal ticks may introduce unnecessary complexity.

More Computation Does Not Guarantee Better Results

The paper itself shows that predictions can become worse after additional internal processing.

Therefore, simply increasing the number of ticks does not guarantee improved performance.

More Research Is Needed

The authors explicitly position the paper as an introduction and investigation of the CTM architecture rather than a claim that it achieves state-of-the-art performance across all tasks.

Its practical usefulness will depend on the problem, computational budget, and comparison with appropriate alternative architectures.

9. Final Thoughts

The Continuous Thought Machine introduces an interesting perspective on neural network design.

Instead of considering neurons only as units that transform information, CTM gives importance to how individual neurons behave over time and how their activity patterns relate to one another.

Its two central ideas are:

Neuron-Level Models: Each neuron processes its own recent activity history.
Neural Synchronization: Relationships between neuron activity histories become part of the model’s internal representation.

By combining these ideas with attention and repeated internal computation, CTM explores how a neural network can refine its internal state over multiple steps.

It does not mean that every neural network should be replaced with CTM, or that CTM is necessarily closer to human intelligence.

But it raises an interesting question for the future of neural architectures:

What new capabilities could emerge if we designed neural networks not only around what their neurons compute, but also around how their activity evolves and relates over time?

That is the idea that makes Continuous Thought Machines worth exploring.

Q&A — Conceptual & Comparison

Conceptual Questions

Q1. What is a Continuous Thought Machine (CTM)? Explain its main idea.

▸Show answer

A Continuous Thought Machine is a neural network architecture inspired by biological neural activity.

Its main idea is that the model performs multiple internal computation steps called ticks. At each tick:

Each neuron processes its own activity history
Relationships between neuron histories are used as an internal representation

Instead of relying only on current activations in a single pass, CTM iteratively refines its state using neuron histories and synchronization.

Q2. What are Neuron-Level Models (NLMs)? Why are they important?

▸Show answer

Neuron-Level Models are small neural networks attached to individual neurons.

Each NLM:

Processes that neuron’s recent pre-activation history
Generates the neuron’s next activity (post-activation)

They are important because they allow individual neurons to have different temporal behaviours, rather than treating all neurons as simple static units with only a current value.

Q3. What is neural synchronization in CTM?

▸Show answer

CTM maintains histories of neuron activities and calculates relationships between pairs of neurons (for example, using dot products of their temporal histories).

These relationships form a synchronization representation.

That representation helps the model:

Make predictions
Guide attention

Synchronization here means a mathematical comparison of activity histories — not simply that two neurons fire at the exact same moment.

Q4. What are internal ticks? How do they help CTM?

▸Show answer

Internal ticks are repeated computation steps inside CTM.

They help the model by allowing it to:

Refine its internal state over time
Continue computing even when the input does not change

Ticks are independent of the input sequence length. The model can perform multiple ticks even with a static image input.

Q5. Explain the role of the synapse model in CTM.

▸Show answer

The synapse model takes the current neuron state and the attention output and produces new pre-activations.

In other words, it decides what information should be sent to the Neuron-Level Models for their next update.

Q6. Differentiate between pre-activations and post-activations.

▸Show answer

Pre-activation	Post-activation
Produced by the synapse model	Produced by NLMs
Input to NLMs	Updated neuron activity
Stored in pre-activation history	Stored in post-activation history

Q7. How does CTM use synchronization for attention?

▸Show answer

The synchronization representation is projected into an attention query.

This query helps the model retrieve relevant information from the input keys and values.

Q8. Why does CTM maintain two synchronization representations?

▸Show answer

CTM needs both a representation for continuing computation and a representation for producing an output:

Action synchronization: used to generate attention queries
Output synchronization: used to generate predictions

Comparison Questions

Q9. How is CTM different from a traditional MLP?

▸Show answer

A traditional MLP typically:

Uses current activations
Performs a feedforward pass from input to output
Does not maintain per-neuron activity histories as a core mechanism

CTM differs because it:

Maintains individual neuron histories
Uses Neuron-Level Models to update each neuron from its own history
Builds a synchronization representation from relationships between histories
Performs repeated internal computation across ticks

Q10. Compare CTM with RNN and Transformer.

▸Show answer

Feature	RNN	Transformer	CTM
Sequential processing	Yes, across sequence positions	No — self-attention processes positions in parallel	Repeated internal ticks
Attention	Not essential	Core mechanism	Uses attention to retrieve input information
Neuron history	Not the central mechanism	Not the central mechanism	Explicitly maintained
Synchronization	No	No	Core representation

Q11. Why is synchronization useful as a latent representation instead of using individual neuron activations directly?

▸Show answer

Synchronization captures relationships between neurons over time.

That can provide richer information about the network’s internal state than individual activations alone, because it encodes how neuron histories relate to one another, not only what each neuron is doing in isolation.

Q12. Can CTM perform multiple internal computation steps without receiving new input? Explain.

▸Show answer

Yes. CTM can repeatedly update its internal neuron states using stored activity histories, even when the input remains unchanged — for example, a static image.

Q13. Does increasing the number of internal ticks always improve prediction accuracy? Why or why not?

▸Show answer

No. More ticks can sometimes cause overthinking, where the model changes a previously correct prediction into an incorrect one.

Additional computation does not guarantee better predictions.

Q14. What are the main contributions and limitations of CTM?

▸Show answer

Contributions

Introduces Neuron-Level Models
Uses synchronization-based representations
Enables iterative internal computation through ticks

Limitations

Computationally expensive
Sensitive to training and initialization
Not proven to outperform existing architectures on every task