Alexandre V. Jamet, PhD.

Modern server workloads have massive instruction footprints, exacerbating pressure on the processor front-end, and making microarchitectural techniques like L1 instruction (L1I) prefetching essential to alleviate this bottleneck. This paper reveals that although L1I prefetchers deliver significant IPC gains, their full potential remains underutilized due to two main factors: (i) L1I prefetch requests that cross page boundaries require address translation before being issued, and the latency involved in retrieving these translations undermines the timeliness of the L1I prefetches; (ii) the reuse potential of code lines fetched in the cache hierarchy by L1I prefetches is highly variable—while a few lines are accessed multiple times, many are dead-on-arrival. This paper proposes the Instruction Prefetch Centric Cache and TLB Management (IP-CaT), the first microarchitectural scheme orchestrating TLB and cache management to maximize the benefits of L1I prefetching. IP-CaT comprises of two modules: (i) the translation Prefetch Buffer (tPB), a small buffer located alongside the last-level TLB (sTLB) that accommodates page table entries (PTEs) fetched by L1I page-cross prefetches to reduce the address translation cost of L1I prefetching and (ii) the Trimodal Instruction Prefetch Replacement Policy (TIPRP), a decision-tree based replacement policy for the L2 cache (L2C) specialized in the management of lines fetched by L1I prefetches. Our evaluation shows that IP-CaT delivers significant performance benefits when integrated with three state-of-the-art L1I prefetchers (EPI, FNL+MMA, Barça). For example, IP CaT+EPI achieves an 8.7% geomean speedup over EPI across a set of 105 contemporary server workloads. We also show that IP-CaT outperforms the state-of-the-art instruction TLB prefetcher, the leading TLB replacement policy (CHiRP), and the state-of-the-art code-aware, prefetch-aware, and general-purpose cache replacement policies (Emissary, SHiP++, Mockingjay).

The present invention relates to a first-level perceptron (FLP) off-chip predictor communicatively connectable to a computing core and to a DRAM, wherein the core and the DRAM are communicatively connected through a multi-level cache hierarchy of levels L1D, L2C, ..., LLC. The FLP is advantageously adapted with an FLP off-chip prediction mechanism comprising two thresholds, Tlow and Thigh. The invention also relates to a two-level perceptron (TLP) off-chip predictor comprising a first-level perceptron (FLP) off-chip predictor according to any of the preceding claims; and a second-level perceptron (SLP) off-chip predictor communicatively connectable to a multi-level cache hierarchy of levels L1D, L2C, ..., LLC through a L1D prefetcher, wherein the multi-level cache hierarchy is communicatively connected to a computing core and to a DRAM.

Since its inception with the first computing systems, computer architecture has lived through many revolutions and saw plenty of technological innovations. However, a significant challenge has persisted throughout the evolution of computing systems: the Memory Wall. To address this challenge, architects have devised various latency tolerance techniques, including cache hierarchy, cache replacement policies, hardware prefetching, and off-chip prediction. Cache hierarchy involves the use of intermediate memories, such as caches, to store frequently accessed data close to the processor, thereby reducing memory access latencies. Cache replacement policies determine which data blocks should be stored or evicted from caches based on predictions of future reuse. Hardware prefetching mechanisms aim to bring data blocks that are likely to be needed in the near future into the cache proactively. Off-chip prediction predicts whether a load demand request will benefit from cache access or if it will require a DRAM access, allowing for speculative fetching of data blocks from DRAM to hide memory access latencies. This thesis addresses the challenges of cache management and memory access optimization in modern computer architectures, focusing on improving performance and energy efficiency across a variety of workloads. It presents three main contributions. The first contribution critically assesses the effectiveness of contemporary Last Level Cache (LLC) replacement policies across a diverse spectrum of workloads, encompassing graph processing, scientific, industrial applications, as well as standard benchmark suites like SPEC CPU 2006 and SPEC CPU 2017. Despite exhibiting notable performance enhancements in conventional benchmark scenarios, these existing LLC replacement policies often falter in capturing the nuanced access patterns characteristic of modern High-Performance Computing (HPC) and big data workloads. In response to this challenge, two novel LLC replacement policies, namely Multi-Sampler Multiperspective (MS-MPPPB) and Multiperspective with Dynamic Features Selector (DS-MPPPB), are introduced and rigorously evaluated. Demonstrating superior efficacy across a broad array of workloads, these innovative policies offer heightened performance benefits tailored specifically for HPC and big data applications. The second contribution is dedicated to enhancing memory access patterns, specifically for graph-processing workloads. These workloads are renowned for their irregular memory access patterns and suboptimal data locality. This contribution targets the first level of the cache hierarchy, as careful analysis reveals that, when considering graph-processing workloads, the vast majority of L1D misses eventually require a DRAM access. Introducing the innovative Large Predictor (LP), this endeavor aims to discern between regular and irregular memory accesses, channeling irregular accesses efficiently through a dedicated Side Data Cache (SDC). By synergizing LP with SDC, notable performance enhancements are achieved, surpassing conventional cache hierarchies and state-of-the-art cache replacement policies, particularly within the realm of graph-processing applications. The third contribution presents the Two Level Perceptron (TLP) predictor, a sophisticated approach that integrates off-chip prediction with adaptive prefetch filtering within the first-level data cache (L1D). Leveraging a dual-layered structure composed of the First Level Predictor (FLP) and Second Level Predictor (SLP), TLP effectively mitigates average DRAM transactions while enhancing overall performance across both single-core and multi-core workloads. Collectively, these contributions advance the state-of-the-art in cache management and memory access optimization, providing insights and techniques to enhance the performance and energy efficiency of modern computer architectures across a variety of workloads.

To alleviate the performance and energy overheads of contemporary applications with large data footprints, we propose the Two Level Perceptron (TLP) predictor, a neural mechanism that effectively combines predicting whether an access will be off-chip with adaptive prefetch filtering at the first-level data cache (L1D). TLP is composed of two connected microarchitectural perceptron predictors, named First Level Predictor (FLP) and Second Level Predictor (SLP). FLP performs accurate off-chip prediction by using several program features based on virtual addresses and a novel selective delay component. The novelty of SLP relies on leveraging off-chip prediction to drive L1D prefetch filtering by using physical addresses and the FLP prediction as features. TLP constitutes the first hardware proposal targeting both off-chip prediction and prefetch filtering using a multilevel perceptron hardware approach. TLP only requires 7KB of storage. To demonstrate the benefits of TLP we compare its performance with state-of-the-art approaches using off-chip prediction and prefetch filtering on a wide range of single-core and multi-core workloads. Our experiments show that TLP reduces the average DRAM transactions by 30.7% and 17.7%, as compared to a baseline using state-of-the-art cache prefetchers but no off-chip prediction mechanism, across the single-core and multi-core workloads, respectively, while recent work significantly increases DRAM transactions. As a result, TLP achieves geometric mean performance speedups of 6.2% and 11.8% across single-core and multi-core workloads, respectively. In addition, our evaluation demonstrates that TLP is effective independently of the L1D prefetching logic.