Why do I sometimes forget where I put my phone down? Why can I sometimes not find the word I know exists for a specific situation? Yet, why does this picture always bring back vivid memories of a sunny day with my dad on the shores of Sandbanks Provincial Park? I was only a toddler many decades ago but still have patches of memories from that day. I was talking with my dad, we were on a beautiful beach, I wanted to keep the sandpipers and was sad we could not. For the record, I would not pick up wildlife anymore!
Obviously, there are many types of memories, and this is not a discussion of why I sometimes misplace my phone. This essay is about how stored patterns, such as my feelings from that day, can be retrieved from partial cues, in this case a faded film-based picture. This type of pattern completion from a degraded input was the central insight of Hopfield’s seminal 1982 paper, which showed how networks can recover stored patterns from incomplete versions.
The Opportunity
Modern transformer-based systems handle similarity through attention, and they learn associations implicitly in their weights. Attention itself can be viewed as a form of associative retrieval, with Modern Hopfield Networks (Ramsauer et al., 2020) and self-attention updates having been shown to be mathematically equivalent. Production AI systems also pair transformers with external memory layers, such as RAG systems and vector databases, to provide durable, content-addressable storage across time. These external memory layers use the same broad principle of content-addressable retrieval that motivated Hopfield’s original associative memory, demonstrating the relevance of an insight he published more than 40 years ago.
But Hopfield’s 1982 paper contains more than just the retrieval mechanism that has been captured. But Hopfield’s 1982 paper contains more than the static associative-memory framework that later work scaled. In the main framework, patterns are stored in the network’s connections and recovered through asynchronous, neuron-by-neuron updates that move the network toward stable attractor states. Later in the paper, Equation 13 explores a possibility of adding nonsymmetric connection terms so that the network can move from one stored state toward another. This preliminary extension points toward sequential memory, time ordering, memory transitions, and trajectories through state space rather than only static retrieval. This sequence-forming possibility was only very briefly explored in that paper, but it opened a separate direction from static associative retrieval, and that is the subject of a future post.
This post explores Hopfield’s original formulation as a starting point. I built a small project that demonstrates the classical Hopfield framework for associative storage, attractor formation, and recall, using HRRs for structured binding and Nengo as the implementation substrate.
The Model
The project combines three components in a single implementation. A Hopfield network provides the associative memory substrate to store patterns and recover them from partial or noisy cues. HRRs (introduced in the previous post) provide structured binding, letting the system encode not just what is remembered but how the pieces relate, such as this value at this time, this element in this role. Nengo provides the neural substrate for everything to run on, using spiking neuron populations rather than pure vector arithmetic.
The patterns being stored come from a predictive-coding network. For the purposes of this post, the pattern generator can be treated as an opaque box that produces a continuous signal over time. Everything below is about what happens after the patterns exist; how they get stored, how they get bound with structural context, and how they get recovered.
The original project is written in Python using Nengo and Nengo-SPA, the Semantic Pointer Architecture extension that ties structured semantic representations to neural computation. The experiments described below use classical Hopfield mechanics coded in Python.
Hopfield: Storage and Recall
The heart of the memory architecture is a Hopfield network, a substrate for storing patterns as stable attractor states and recovering them from partial or noisy cues. The mechanics are the same ones Hopfield described in 1982, with a simplification that these experiments use single-step synchronous updates rather than the asynchronous (neuron-by-neuron) updates in the original paper. Continuous input signals are first converted into binary patterns using the sign function, producing ±1 vectors that the network can store. Hebbian learning then builds the weight matrix by summing the outer products of these patterns, with self-connections zeroed out. Each stored pattern becomes a stable attractor of the resulting dynamics, and when presented with a partial or noisy version of a pattern, the network settles into the closest stored memory.
As a first check, I stored four patterns and corrupted one with 20% noise. Recall was 100%. But a single point on a curve does not describe the whole shape, and Hopfield’s paper made several claims about the shape of the curve that are worth demonstrating with three separate experiments.
Experiment 1: Graceful Degradation
The first experiment holds the network setup fixed, four random binary patterns stored in a 64-neuron network, and varies the noise level from 0% to 50%, plotting recall accuracy at each step.
The result is the classic Hopfield “graceful degradation” property. Recall is essentially perfect up to about 15% noise, holds above 95% through the mid-20s, then declines steadily as noise approaches half the bits flipped. The network does not break suddenly, but it weakens before eventually falling. This is what Hopfield emphasized in the original paper with associative memory that degrades smoothly under corruption, rather than flipping between “works” and “doesn’t work.”
Experiment 2: The Capacity Limit
The second experiment fixes the noise level at 20% and the network size at 256 neurons (N) but varies the number of stored random patterns from 4 to 80. The theoretical capacity limit for classical Hopfield networks is approximately 0.14·N, or about 36 patterns for a network this size, after which recall accuracy is expected to decline.
The empirical curve matches the theoretical prediction. Recall holds above 95% through the capacity limit and then declines gradually as more patterns are added. At 80 patterns, recall is still around 85%, highlighting that the decline is gradual rather than dramatic, partly because these experiments use single-step recall rather than the full iterative process Hopfield described, updating neurons one at a time until the network settles. The capacity ceiling is real, but classical Hopfield networks keep working rather than falling off.
Experiment 3: The Prototype-Collapse Failure Mode
The third experiment is where the classical Hopfield network has difficulty with recall. The setup uses the same 256-neuron network with 20% noise from Experiment 2, but the stored patterns are no longer independent. Instead, they all share structure with a vector holding a hidden pattern (the prototype), and each is a variation of that underlying prototype. Three pattern types are compared including “uncorrelated” (random), “moderately correlated” (50% of bits randomized from the prototype), and “strongly correlated” (30% of bits randomized).
Random patterns again degrade the way Experiment 2 showed. But the correlated patterns behave differently with recall dropping immediately and then plateauing. Interestingly, the strongly correlated patterns produce higher apparent recall at higher pattern counts. But like a classification model only predicting the majority class, the network has stopped trying to distinguish individual memories and is instead returning ‘approximately the prototype’ for any query. The plateau measures how similar each test pattern happens to be to the prototype, not actual memory recall. Strongly correlated patterns are more like the prototype, so the fall off produces higher apparent accuracy, but the memories themselves have been lost.
This is the “prototype attractor” failure mode of classical Hopfield networks. Correlation between the stored patterns does not only reduce capacity, but it also results in the network falling back to retrieving the prototype. The attractor landscape reduces toward the shared prototype, and the associative memory focuses on prototype recognition instead.
What Modern Hopfield Networks Solved
The classical Hopfield architecture has two well-known limitations that have been documented. Capacity scales linearly with network size (about 0.14·N patterns), and correlated patterns collapse to a prototype attractor, whose basin grows large enough to absorb the individual memory basins.
Modern Hopfield Networks (Ramsauer et al., 2020) address both by replacing the quadratic energy function with a log-sum-exp equation. The change is architectural but significant, and the attractor landscape now supports exponentially many sharp valleys that can be packed close together in pattern space, without collapsing to shared prototypes. The capacity scales from ~0.14·N to ~2(N/2), and correlated patterns can coexist as distinct memories. Everything else from the original Hopfield, including the associative retrieval, the pattern completion from partial cues, the content-addressable memory property, is preserved.
The same architectural principle, allowing associative retrieval by similarity in high-dimensional embedding space, underpins much of the persistent memory infrastructure now used in production AI systems. The experiments above are not new, but the results are consistent with what Hopfield’s 1982 analysis predicted and what four decades of subsequent work refined. But demonstrating them highlights what Modern Hopfield Networks were designed to solve.
HRR: Structured Binding
The Hopfield network handles the storage and the recall of flat patterns. But real memories carry structure, such as time, roles and facts about that memory. Storing “sunny day” and “sandpiper birds” as flat patterns loses the relationship between them.
Holographic Reduced Representations (HRRs) provide this ability for context, and HRRs are producing memory traces for the structured content here. The mechanism is the same, with two random role vectors bound with the values they modify using circular convolution, and the resulting bound vectors are combined by superposition (vector addition) into a single memory trace.
In this project, the two role vectors (TIME and VALUE) are bound with the actual values being remembered. The whole memory trace can then be queried with either role, convolving the memory with the inverse of TIME to recover the corresponding VALUE, and vice versa.
A small but useful result is the resulting memory vectors have unit length (a magnitude of 1.00). Without normalizing the final memory vector, the vectors could grow large or drift toward zero across multiple operations, eventually resulting in the memory trace becoming unusable.
What HRRs contribute to the memory architecture is the ability to store structured content. That structural richness is what lets associative memory store more than just whether a pattern exists, and it is what makes HRRs a natural fit with a Hopfield substrate that handles the associative recall.
Bringing It Together
The two components are useful on their own, but the point of the project was to show that they can work together, with a single pipeline that can run end to end. Starting with a continuous signal from the pattern generator, snapshots of the signal are binarized and stored as Hopfield attractors, and time values are bound with role vectors into HRR memory traces. Both mechanisms produce stable, recoverable representations. The Hopfield network settles noisy inputs to their stored patterns, and the HRR traces preserve unit length across binding and combination.
The scope of this demonstration should be stated plainly though. It is not a benchmark of memory performance or claims about superiority over other memory systems. What the project demonstrates is that classical associative memory can be built end-to-end in a biologically motivated substrate, alongside structured binding, driven by a signal source generated by another cognitive mechanism entirely.
Future Work
Two directions naturally extend this project, and they are the subject of upcoming posts in this series.
First, this post treated the pattern generator as an opaque signal source. The next post explores that predictive-coding network built on the Free Energy Principle, which generates the patterns that Hopfield and HRR store and bind. Predictive coding is a distinct research program from Hopfield’s associative memory, but the two connect naturally since memory needs something to remember, and predictions of the world are among the most useful things a system could store.
The second direction is more speculative and returns to a possibility Hopfield briefly explored near the end of the original paper. Modern Hopfield Networks substantially scaled the static associative-memory framework. Equation 13 points in a different direction by adding nonsymmetric connection terms to demonstrate that a network could sometimes remain near one stored state before moving toward the next. The early results were limited, but the idea anticipated later work on sequential memory, memory transitions, time ordering, and trajectories through state space.
The broader thread across this series is that cognitive science research from the 1980s onward, including Hopfield’s memory networks, predictive coding, and free-energy models of the brain, has more to give current AI than has been leveraged so far.
Sandbanks, Revisited
I still have the picture, and it sits on my desk, a reminder of a sunny afternoon on a beach in Ontario many years ago. Whenever I glance at it, everything comes back; the birds, the sand, the wanting to keep them, the sadness when I could not. The photograph is the cue, and the memory is the completion.
That mechanism is associative memory. Hopfield showed in 1982 that neural networks could implement associative memory by using repeated neuron-by-neuron updates to settle toward stable attractor states. Modern Hopfield Networks later scaled that associative-retrieval principle and revealed its close mathematical relationship with transformer attention. In 2024, Hopfield received the Nobel Prize in Physics, shared with Geoffrey Hinton, for foundational discoveries and inventions that enable machine learning with artificial neural networks. It took decades for the field to scale the static associative-memory framework, but Hopfield’s tentative exploration of transitions between memories points toward another thread that remains worth exploring.
The next post opens the opaque box.
Sources
Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8), 2554–2558. https://doi.org/10.1073/pnas.79.8.2554
Kelly, M.A., Arora, N., West, R.L. & Reitter, D. (2020), Holographic Declarative Memory: Distributional Semantics as the Architecture of Memory. Cognitive Science, 44: e12904. https://doi.org/10.1111/cogs.12904
Kelly, M. A. & Tomkins-Flanagan, E. (2016). HRR: Holographic reduced representations. Software. Available from: https://github.com/ecphory/hrr.
Plate, T. A. (1995). Holographic reduced representations. IEEE Transactions on Neural Networks, 6(3), 623–641. https://doi.org/10.1109/72.377968
Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G. K., Greiff, V., Kreil, D., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2021). Hopfield networks is all you need [Conference paper]. International Conference on Learning Representations. https://arxiv.org/abs/2008.02217
The Nobel Foundation. (2024). The Nobel Prize in Physics 2024 [Press release]. https://www.nobelprize.org/prizes/physics/2024/press-release/
Inferential Loop 