SNU Innovations(vol.07)

• Custom accelerator chip design for machine learning algorithms
• Hardware and software co-design for big data processing
• Algorithm and software optimizations for machine learning and big data processing

The Architecture and Code Optimization Lab (ARC Lab), led by Prof. Jae W. Lee in the Department of Computer Science and Engineering, focuses on innovative computer designs tailored for AI and big data processing. As of April 2024, Prof. Lee guides 16 Ph.D. and 4 master's students in conducting research on topics like custom machine learning (ML) hardware and software optimization, hardware/software co-design for big data processing, and memory system architectures for future computing platforms.

The explosive growth of generative AI has led to a massive increase in the amount of computing resources required for training and inference of ML models. Serving ChatGPT could cost OpenAI over $700,000 per day to operate¹, and the training cost of the GPT-4 model reportedly exceeds $100M². Conventional general-purpose computers utilizing CPUs cannot meet the desired performance, power efficiency, and cost requirements due to the rapid expansion of ML applications.

To address this challenge, the ARC Lab focuses on developing hardware specialized for ML task acceleration. The lab embraces a holistic co-design methodology, producing software tailored to this ML-specific hardware as well. For instance, Prof. Lee's team introduced A³, the first custom hardware accelerator aimed specifically at the attention mechanism-the key operation for generative AI models with trillion-scale parameters. Its successor, ELSA^4-5, advances execution efficiency and broadens algorithmic applicability. These innovations underscore the benefits of co-designing algorithm and hardware, pushing the boundaries of algorithmic optimization well beyond the capabilities of today's GPUs. Both designs have been successfully prototyped using TSMC's 40nm CMOS technology. The ARC Lab strives to provide comprehensive solutions that enable efficient and cost-effective execution of ML workloads, ultimately keeping pace with the ever-increasing demands of the generative AI era.

In addition to the AI/ML-focused research, the ARC Lab is actively conducting research on specialized architectures for non-ML workloads. The lab is developing various accelerator architectures specifically designed for big data processing, with a particular emphasis on tasks like search^6-7, vector database⁸, genomic analysis⁹, and data analytics¹⁰. Again, these specialized hardware designs are complemented by the development of software frameworks that support and optimize the execution of big data workloads on the accelerators. By creating a synergistic combination of domain-specific hardware and software, the ARC Lab aims to provide efficient and cost-effective solutions for the ever-growing demands of big data processing, ensuring that these systems can keep pace with the increasing complexity and scale of data-intensive applications.

Finally, the ARC Lab's research efforts extend beyond hardware to include extensive work in the algorithm and software domain. As previously mentioned, the proliferation of ML and big data applications has led to an explosive growth in the demand for memory and storage resources. However, traditional software stacks struggle to meet the stringent requirements of performance, power efficiency, and cost-effectiveness posed by these data-intensive workloads. Recognizing this challenge, the ARC Lab is actively investigating novel software stack designs, with emphasis on leveraging emerging memory and storage devices. For example, Prof. Lee's team proposes to leverage emerging nonvolatile memory and high-performance SSDs to lessen the memory capacity wall for graph neural network training¹¹ and key-value stores^12-13. There are other proposals from his team that propose a novel lossless image compression format for high-throughput training of deep neural networks (DNNs)¹⁴ and pruning and quantization techniques^15-16. By optimizing and tailoring these components to the unique characteristics of ML and big data workloads, the ARC Lab strives to unlock the full potential of the underlying hardware while ensuring optimal resource utilization and cost-efficiency.

In conclusion, the ARC Lab's comprehensive approach to tackling the challenges posed by the rapid advancement of AI sets it apart as a leader in the field. The lab's unique strength lies in its ability to cultivate a new generation of researchers who possess a deep understanding of both hardware and software. By fostering a culture of collaboration and cross-pollination between hardware and software researchers, the ARC Lab is creating a synergistic environment where groundbreaking ideas can flourish. With their cutting-edge research and dedication to pushing the boundaries of what is possible, the ARC Lab is poised to make significant contributions to the fields of AI and big data, shaping the future of computing for years to come.