An embedded AI chip can advertise dozens or even hundreds of trillions of operations per second and still struggle to deliver the frame rate, latency, or power efficiency developers expect. The reason is often not a shortage of arithmetic capability. It is the much less glamorous problem of getting data to the arithmetic units fast enough.
That is the central idea behind the memory wall in AI acceleration. Neural networks constantly move weights, activations and intermediate results between storage levels. When those transfers cannot keep up with the accelerator, the compute engine spends cycles waiting rather than calculating.
The problem is particularly important in embedded systems, where designers must balance performance, memory capacity, board size, thermal limits and battery or power budgets. An NPU connected to relatively constrained memory has very different characteristics from a data-center accelerator paired with high-bandwidth memory.
Why More TOPS Does Not Automatically Mean Faster AI
TOPS is useful for describing an accelerator’s theoretical arithmetic capability, but it does not describe how quickly a particular neural-network workload can be fed with data. The roofline performance model makes this distinction explicit: attainable performance is bounded by both peak compute throughput and available memory bandwidth, with arithmetic intensity determining which limit dominates.
Arithmetic intensity describes how much computation is performed for every byte transferred through the relevant memory system. A workload with high arithmetic intensity can reuse data heavily and keep an accelerator busy. A workload with low arithmetic intensity spends comparatively more time moving data.
Consider NVIDIA’s Jetson Orin Nano Super. NVIDIA specifies up to 67 INT8 TOPS and 102 GB/s of memory bandwidth. Those figures imply a theoretical machine balance of roughly 657 INT8 operations per byte if the peak numbers are treated as a simple compute-to-bandwidth ratio.
That does not mean every neural network needs 657 operations per byte to run efficiently; the calculation is simply a useful way to visualize the gap between the accelerator’s arithmetic ceiling and its external memory interface. A hypothetical workload performing 10 operations per byte would have a bandwidth-side ceiling of only about 1.02 TOPS on that simple model, far below the chip’s 67-TOPS headline figure.
The exact result for a real workload will depend on caches, on-chip SRAM, data reuse, access patterns, compression, operator scheduling and the actual sustained bandwidth available to the accelerator. The important lesson is that peak TOPS is not the same thing as usable application throughput.
The Real Cost of Moving Neural-Network Data
Every inference involves more than multiplying matrices. The system must fetch weights, read input activations, write intermediate tensors and move outputs to subsequent operations. If those tensors leave fast on-chip memory and reach external DRAM, the cost of data movement can quickly become significant.
Arm’s Ethos-U documentation provides a useful view of how embedded NPUs deal with this hierarchy. The architecture distinguishes between low-latency, high-bandwidth memory, typically on-chip SRAM, and higher-latency, lower-bandwidth external memory such as DRAM or Flash. The Ethos-U toolchain explicitly uses scratch buffers and fast scratch memory to keep frequently accessed data close to the accelerator.
Arm’s own Ethos-U performance example shows why the distinction matters. In an SRAM-only configuration, a compiled network used about 5.18 MiB of SRAM, transferred about 35.65 MB per inference through the SRAM interface and recorded no DRAM access cycles. The example reported approximately 0.12 TOPS and a 4.94 ms batch inference time.
The example is not a universal benchmark for every Ethos-U implementation, but it demonstrates the engineering reality: accelerator performance depends not only on how many MAC units exist, but also on how intelligently data is stored, reused and moved.
Why Lightweight Networks Can Be Surprisingly Memory-Bound
It sounds counterintuitive, but reducing the number of floating-point operations does not necessarily make a neural network easier for hardware to execute.
Depthwise convolutions are a classic example. They drastically reduce arithmetic work compared with conventional convolutions, which is one reason they became popular in lightweight architectures such as MobileNet. But their reduced computation also means there can be less work available to amortize the cost of reading and writing data.
Research on ARMv8 implementations has found that depthwise convolution performance is primarily constrained by memory access rather than arithmetic operations. The study showed that improving data reuse, register tiling and communication overhead could produce substantial speedups without simply adding more compute.
This creates an important distinction between FLOP reduction and system-level efficiency. A model can require fewer operations yet perform poorly when the reduction in computation is accompanied by inefficient memory movement.
Activations Can Become the Hidden Bandwidth Consumer
Weights receive most of the attention when discussing AI memory, but intermediate activations can create enormous traffic as well. A layer may read an input tensor, fetch its weights, produce an output tensor and then pass that tensor to another layer. If the intermediate result cannot remain in an appropriate on-chip buffer, it may be written to and read back from external memory.
The traffic compounds across a deep network. The model does not necessarily need a huge parameter count to become bandwidth-sensitive. Repeated movement of moderate-size feature maps can consume substantial bandwidth during real-time processing.
This is one reason accelerator compilers spend so much effort on tiling, buffering, scheduling and data reuse. Instead of moving an entire tensor repeatedly, hardware and software can divide it into smaller tiles that fit into SRAM, process those tiles while data remains local, and minimize trips to slower memory.
SRAM Is Fast, but It Is Not Infinite
The obvious solution would be to keep everything in SRAM. The problem is that embedded SoCs cannot simply provide unlimited on-chip memory.
Arm explicitly notes that SRAM-only execution offers the best performance because traffic stays in low-latency, high-bandwidth memory, but also points out that embedded systems have limited SRAM and that neural networks continue to grow. When models or peak intermediate tensors become too large, external memory becomes necessary.
That creates a three-way engineering trade-off:
- More SRAM: better locality and less external-memory traffic, but greater silicon area and cost.
- More DRAM bandwidth: more data can be supplied to the accelerator, but memory interfaces consume power, pins and board resources.
- More compute: higher theoretical throughput, but limited benefit when the workload is already bandwidth-bound.
The best design is therefore not necessarily the chip with the most compute. It is the one that balances compute capacity with enough local storage and memory bandwidth for the target workloads.
Quantization Helps, but It Is Not a Magic Wand
Quantization is one of the most direct ways to reduce memory traffic. Moving from FP32 values to INT8 reduces the storage requirement for those tensors by a factor of four, assuming the same number of elements. INT4 can reduce it further.
That can increase effective arithmetic intensity because fewer bytes have to be transferred for the same logical tensor. Recent research has begun explicitly treating quantization as a way to optimize arithmetic intensity, rather than merely as a method for reducing model size. One 2025 study reported throughput gains on a memory-bound MobileNetV2 workload using arithmetic-intensity-aware quantization.
Still, quantization does not automatically eliminate the bandwidth bottleneck. Activations, temporary buffers, layout conversions and unsupported operators can continue to generate traffic. Hardware must also support the chosen precision efficiently; otherwise, a smaller representation can be offset by inefficient execution.
Operator Fusion Reduces Unnecessary Memory Trips
Another important technique is operator fusion. Consider a sequence such as convolution followed by activation and normalization. If each operation writes its output back to external memory before the next operation begins, the system creates additional traffic that contributes little useful computation.
A fused implementation can keep intermediate values in registers or SRAM and perform multiple operations before writing the final result. This reduces memory transactions and improves data locality.
For embedded hardware, this kind of optimization can be more valuable than another increase in raw arithmetic capacity. The 2025 roofline research from Verhelst, Benini and Verma emphasizes the importance of understanding how accelerator architecture, memory systems and application characteristics interact rather than optimizing compute in isolation.
Transformers Introduce a Different Memory Problem
The memory-bandwidth problem becomes especially visible as transformer models move onto edge devices.
During autoregressive generation, the model maintains a key-value cache containing information from previous tokens. As the context grows, the system must access more cached data. Recent research describes LLM inference as increasingly constrained by memory bandwidth, with frequent KV-cache accesses becoming a major source of data movement.
For embedded or mobile hardware, this creates a difficult combination of limited memory capacity and limited bandwidth. A model may technically fit in the device’s memory while still producing poor token-generation performance because the system cannot move the relevant data quickly enough.
That is an important shift in how edge AI should be evaluated. For small sensor networks, the critical concern might be moving feature maps efficiently. For on-device generative AI, it can increasingly be the movement of weights and persistent attention state.
How Engineers Recognize a Bandwidth Bottleneck
A practical diagnosis requires more than looking at the accelerator’s advertised utilization. Engineers should examine both compute and memory counters while running the actual workload.
Several signs point toward a bandwidth-bound system:
- NPU or accelerator utilization remains well below its theoretical maximum.
- Increasing the accelerator clock produces little improvement in throughput.
- Reducing tensor precision or tensor size produces a disproportionately large speedup.
- Memory traffic rises close to the system’s sustainable bandwidth.
- Latency changes significantly when data placement or buffer sizes change.
- Performance improves when operators are fused or intermediate tensors remain on-chip.
Roofline analysis provides a useful first approximation, but real embedded systems require measurements of sustained bandwidth, cache or SRAM behavior, DMA activity and contention from other system components.
Why Camera and Sensor Workloads Make the Problem Harder
Embedded AI rarely operates in isolation. A robotics platform may simultaneously handle camera capture, image processing, neural inference, display, storage and communications. These components can compete for the same memory subsystem.
That makes bandwidth contention just as important as the headline bandwidth number. A system with 100 GB/s of theoretical memory bandwidth does not necessarily provide an AI accelerator with 100 GB/s continuously. Operating-system activity, CPU workloads, other accelerators and input/output traffic all consume part of the available capacity.
This is why end-to-end benchmarks are more informative than isolated TOPS figures. MLCommons’ MLPerf Tiny, for example, was created specifically to evaluate inference performance on extremely constrained devices and includes workload and power considerations relevant to embedded deployment.
The Optimization Strategy Is About Moving Fewer Bytes
When a workload is memory-bound, the optimization question changes. Instead of asking only, “How do we perform more operations?” engineers should ask, “How do we perform the same useful work while moving less data?”
That leads to a practical hierarchy of techniques:
- Keep frequently reused data on-chip. Use SRAM, caches and local buffers aggressively.
- Tile large tensors. Process blocks that fit into fast local memory and maximize reuse.
- Fuse operators. Avoid unnecessary materialization of intermediate tensors.
- Quantize appropriately. Reduce the number of bytes transferred while preserving acceptable accuracy.
- Exploit sparsity where hardware supports it. Skipping zeros can reduce both computation and data movement.
- Optimize tensor layouts. Poorly aligned or inefficient access patterns can waste memory bandwidth.
- Profile the complete pipeline. Optimize the memory path that actually limits the application rather than assuming the NPU is the bottleneck.
Research on edge accelerators increasingly reflects this data-movement focus. Work on flexible accelerator dataflows, for example, has emphasized reducing communication between memory and compute because movement can dominate energy consumption.
Where Embedded AI Architecture Is Heading
The memory-bandwidth problem is pushing hardware designers toward architectures that bring computation closer to data instead of repeatedly transporting data between separate blocks.
Processing-in-memory and near-memory computing are two responses to this challenge. Rather than relying exclusively on conventional transfers between compute engines and external memory, these approaches attempt to execute some operations within or close to the memory arrays.
Research into edge AI has described the traditional separation between compute and memory as a fundamental memory wall because data movement introduces both latency and energy costs. More recent NPU-PIM research is similarly motivated by the bandwidth and energy demands of transformer inference.
The commercial direction is also becoming clearer. Modern embedded platforms increasingly treat memory bandwidth, local storage and accelerator architecture as a combined design problem rather than three independent specifications.
The Bigger Lesson for Embedded AI Designers
Memory bandwidth is easy to overlook because it lacks the marketing appeal of a larger NPU or a higher TOPS figure. Yet it often determines whether that compute capability can be translated into real application performance.
A neural network that performs fewer operations can outperform a larger one if it reuses data more effectively. An accelerator with fewer TOPS can beat a theoretically faster accelerator when its memory hierarchy better matches the workload. And a modest increase in memory bandwidth can sometimes produce a larger real-world improvement than a substantial increase in arithmetic capacity.
The central metric, therefore, is not simply how many operations an embedded AI processor can perform. It is how effectively the entire system moves the right bytes, to the right place, at the right time.
For engineers designing the next generation of robotics, smart cameras, vehicles, industrial controllers and local generative-AI devices, that makes memory architecture a first-order AI performance decision—not an afterthought.
Key Takeaways
- Peak TOPS describes theoretical compute capacity, not application throughput.
- Low-arithmetic-intensity workloads can become memory-bandwidth-bound even on powerful NPUs.
- On-chip SRAM is valuable because it offers faster, more local access than external DRAM.
- Depthwise convolutions and other lightweight operations can be memory-bound precisely because they perform relatively little computation per byte moved.
- Transformer inference adds another major memory challenge through growing KV caches.
- Tiling, operator fusion, quantization, sparsity and better dataflows can reduce memory traffic.
- Future edge accelerators are likely to place greater emphasis on near-memory and in-memory computing.
References for Further Reading
- NERSC — Roofline Performance Model
- Verhelst, Benini & Verma — How to keep pushing ML accelerator performance? Know your rooflines!
- PyTorch ExecuTorch — Arm Ethos-U Backend and Memory Modes
- Arm — Evaluating Ethos-U Performance
- NVIDIA — Jetson Orin Nano Super Developer Kit
- MLCommons — MLPerf Inference: Tiny
Discover more from Aree Blog
Subscribe now to keep reading and get access to the full archive.
