Co-Designing Cache and TLB Management for High-Performance Instruction Prefetching

The continuous scaling of scale-out cloud applications, microservice architectures, and modern database management systems has profoundly impacted processor microarchitecture. Unlike traditional loop-heavy computational kernels, contemporary enterprise server workloads are characterized by sprawling, multi-megabyte instruction footprints. This massive expansion of the instruction working set routinely exceeds the capacity of standard Level-1 Instruction (L1I) caches, resulting in frequent front-end starvation and establishing instruction supply as a primary execution bottleneck in modern Out-of-Order (OoO) architectures.

To mitigate this bottleneck, substantial research has been dedicated to developing sophisticated state-of-the-art L1I prefetchers operating within the virtual address space. However, as demonstrated in our recent ISCA 2026 publication, the efficacy of even the most advanced prefetching algorithms is fundamentally constrained by underlying hardware management structures.

In our paper, “Enhancing Instruction Prefetching via Cache and TLB Management”, we characterize a critical microarchitectural “double bottleneck”: the decoupling of prefetch address generation from the virtual-to-physical translation path and the unmanaged insertion of instruction streams into the lower cache hierarchy. To overcome these limitations, we propose IP-CaT (Instruction Prefetch Centric Cache and TLB Management), an integrated, low-overhead microarchitectural framework that co-designs Translation Lookaside Buffer (TLB) and cache management policies around the unique behaviors of instruction streams [1].

Unveiling the Microarchitectural Bottlenecks

While contemporary prefetchers (such as EPI, FNL+MMA, and Barça) exhibit high accuracy in predicting future instruction stream trajectories, their real-world execution gains are heavily suppressed by two main microarchitectural phenomena:

1. The Translation Latency Barrier (The Virtual Memory Wall)

Because sophisticated L1I prefetchers operate utilizing virtual addresses, every generated prefetch line request must undergo address translation before interacting with the physical cache hierarchy. Due to the expansive nature of server code, prefetch sequences frequently cross 4KB virtual page boundaries. When a prefetch request references a page not currently mapped in the iTLB, nor in the sTLB, it triggers a translation miss, forcing an invocation of the page-table walker.

Consequently, the translation latency often exceeds the prefetch lead time. By the time the physical address is resolved, the demand fetch from the execution pipeline has already arrived at the corresponding Instruction Pointer (IP), rendering the prefetch late and structurally useless. Furthermore, injecting these speculative page walks directly into the primary Translation Lookaside Buffer hierarchy risks evicting critical, highly reusable demand mappings, thereby degrading overall translation throughput.

2. The Cache Pollution Crisis

Instruction streams fundamentally differ from data streams regarding their temporal reuse characteristics. To effectively mask deep memory latencies, virtual prefetchers aggressively inject a high volume of blocks into the Level-2 Cache (L2C). However, instructions typically exhibit highly volatile reuse behavior; a substantial portion of prefetched lines are consumed exactly once by the hardware front-end and never referenced again.

Standard L2C insertion and replacement policies (e.g., standard LRU or basic pseudo-LRU variations) fail to differentiate between regular data streams and highly transient instruction prefetch streams. As a result, these “Dead-on-Arrival” (DoA) instruction cache lines reside in the L2 allocation sets for extended periods, causing severe cache pollution by prematurely evicting highly reusable data and instruction blocks, which ultimately suppresses the system’s overall Instructions Per Cycle (IPC).

Core Thesis: Maximizing next-generation front-end performance requires shifting focus from prediction accuracy to structural enablement—ensuring that the underlying cache and memory translation hierarchies are explicitly optimized to handle aggressive prefetch streams.

The IP-CaT Architectural Framework

To address these coupled challenges, IP-CaT introduces two low-overhead, specialized hardware mechanisms designed to eliminate translation delays and cache degradation simultaneously.

+-------------------------------------------------------------------+
|                        IP-CaT Framework Architecture              |
+-------------------------------------------------------------------+
|                                                                   |
|             [ Virtual Address Prefetch Stream ]                   |
|                              |                                    |
|                              v                                    |
|     +------------------------+--------+-----------------------+   |
|     |  tPB (Translation      |        |  TIPRP (Trimodal      |   |
|     |  Prefetch Buffer)      |        |  L2C Replacement)     |   |
|     +------------------------+        +-----------------------+   |
|     | Intercepts page-cross  |        | Decision-tree driven; |   |
|     | prefetch translations; |        | Identifies & filters  |   |
|     | decouples sTLB path.   |        | DoA instruction blocks|   |
|     +------------------------+        +-----------------------+   |
|                              |                                |   |
+------------------------------|--------------------------------|---+
v                                v
[ Timely Prefetch Execution ]     [ Optimized L2C Capacity ]

(Figure 1: Architectural block diagram illustrating the dual-component execution flow of the IP-CaT framework.)

1. Decoupling Translations via the tPB

To insulate the critical demand translation path from speculative prefetch overheads, we introduce the tPB (translation Prefetch Buffer). The tPB functions as a small, highly specialized cache structure situated parallel to the secondary TLB (sTLB) layer.

When an L1I prefetch stream detects an impending page-boundary crossing, the tPB eagerly intercepts the virtual address, coordinates with the hardware page-table walker, and caches the resulting translation mapping locally. By isolating these prefetch-induced translations within the tPB, IP-CaT prevents the pollution of the main TLB hierarchy. When the pipeline subsequently generates the corresponding demand fetch, the translation is retrieved instantly from the tPB with near-zero latency, effectively transforming late prefetches into highly effective, on-time hits.

2. Adaptive Cache Insulation via TIPRP

To mitigate L2C capacity degradation, we propose the TIPRP (Trimodal Instruction Prefetch Replacement Policy). Traditional cache partitioning or set-dueling frameworks adapt too slowly to the rapid, phase-driven changes characteristic of modern instruction streams. TIPRP replaces these mechanisms with a highly efficient, hardware-implemented decision tree classifier.

TIPRP monitors the real-time reuse characteristics of incoming blocks, dynamically evaluating the likelihood of an instruction prefetch being Dead-on-Arrival. Based on this real-time telemetry, the policy dynamically shifts each cache set between three specialized sub-policies:

• Aggressive Demotion: Inserting speculative lines with minimal dead-time residency.
• Instant Eviction: Bypassing or immediately marking low-confidence prefetch blocks for replacement.
• Preservation Mode: Protecting instruction streams that exhibit dense temporal or spatial locality.

By executing this trimodal classification at hardware speeds, TIPRP effectively purges dead prefetch blocks before they can displace critical working sets.

Quantitative Evaluation and Performance Impact

We evaluated the architectural efficacy of the IP-CaT framework using cycle-accurate simulation across an extensive corporate benchmarking suite encompassing 105 complex server workloads. The framework was co-evaluated alongside several state-of-the-art instruction prefetchers and compared against contemporary cache and TLB management mechanisms, including Morrigan, CHiRP, Emissary, Mockingjay, and SHiP++.

Our experimental results demonstrate that integrating IP-CaT with an advanced Edge Prefetch Indexing (EPI) configuration yields a 6.1% geometric mean speedup in overall system performance.

(Figure 2: Empirical performance analysis displaying normalized speedups across diverse enterprise server workloads compared to state-of-the-art baseline models including Mockingjay and Morrigan).

Detailed microarchitectural analysis highlights two primary vectors of improvement:

• Translation Latency Mitigation: The implementation of the tPB successfully counteracts over 80% of the execution penalties historically introduced by page-boundary prefetch serialization.
• Cache Hit Rate Enhancement: By filtering out transient instruction streams, TIPRP significantly optimizes the L2C allocation space, minimizing dead block residency without necessitating large tracking structures or high power overheads.

Conclusion

For decades, microprocessor design paradigms have treated prefetching, virtual memory translation, and cache replacement policies as orthogonal research domains. However, as the instruction footprints of modern cloud-scale software scale past physical hardware boundaries, these isolated silos introduce profound performance liabilities. IP-CaT demonstrates that an integrated, co-designed management layer can unlock latent execution potential within existing execution pipelines.

For a comprehensive review of our microarchitectural methodology, hardware overhead analysis, and workload breakdowns, we invite you to read our full preprint on arXiv or engage with our research team directly at ISCA 2026.