V3I7P46

Evolution of Dedicated Processor Architectures for Generative AI Models (LLMs): A Comprehensive Review

Zainab H. Mohammad^1*

Abstract

The scaling of auto-regressive Large Language Models (LLMs) has revolutionized the generative artificial intelligence, and has demonstrated bottlenecks in the architectural design of the traditional von Neumann computing fabrics. The runtime performance ceases to be computation-bound in pre-fill phases, and becomes memory-bandwidth-bound in the iterative token decoding process, with simple models with hundreds of billions of parameters. This incompatibility is manifested in the form of structural memory stalls, underutilization of the execution units and extreme thermal dissipation concerns and all of which are also termed the memory wall. This paper gives a comprehensive survey of the development of dedicated hardware accelerators which are specifically developed to reduce these data transport penalties. We create a systematic taxonomy of Application-Specific Integrated Circuits (ASICs), Reconfigurable Field-Programmable Gate Arrays (FPGAs), and non-von Neumann Processing-In-Memory (PIM) macrostructures. In addition, we discuss the Hardware-Software Co-design paradigm, tracing the direct remodeling of the underlying physical silicon dataflows by low-precision algorithmic tye-castings (INT4/FP4) as well as structural sparsity metrics (2:4). Finally, we also draw attention to the research gaps, i.e., to microarchitectural side-channel vulnerabilities and the extreme heat flux of 3D-stacked semiconductor boundaries, as a clean blueprint to future secure, energy-efficient domain-specific computing cores.

Keywords:

Hardware accelerators; Dedicated Processors; Memory Wall; Large Language Models (LLMs); Processing-In-Memory (PIM); Systolic Arrays; Edge AI; Hardware-Software Co-Design; Semiconductor Security.