หลักสูตร/ปี พ.ศ. วิศวกรรมศาสตรบัณฑิต สาขาวิชาวิศวกรรมคอมพิวเตอร์ ปี พ.ศ. 2568
ภาคและปีการศึกษาที่สำเร็จการศึกษา ภาคปลาย ปีการศึกษา 2568
ประเภทโครงงาน
โครงงานวิศวกรรม
ชื่อโครงงานภาษาไทย
Automated LLM pruning using combinatorial optimization
ชื่อโครงงานภาษาอังกฤษ
Automated LLM pruning using combinatorial optimization
ผู้พัฒนา
6410501099 พชรพล ราชสภา
อาจารย์ที่ปรึกษาหลัก
จันทนา จันทราพรชัย
อาจารย์ที่ปรึกษาร่วม
-
บทคัดย่อ
Nowadays, large Language Models (LLMs) have been utilized in many aspects of 1
natural language processing. However, due to their significant size and high computational 2
demands, large computational resources are required for deployment. In this research, 3
we focus on the automated approach for size reduction of such a model. We propose the 4
framework to perform the automated pruning based on combinatorial optimization. Two 5
techniques were particularly studied, i.e., Particle Swarm Optimization (PSO) and Whale 6
Optimization Algorithm (WOA). The model pruning problem was modeled as a combina- 7
torial optimization task, whose the goal is to minimize model size while maintaining model 8
accuracy. The framework systematically explores the search space to identify the most 9
optimal pruning configurations, removing redundant or non-contributory parameters. The 10
two optimizations, PSO and WOA, were evaluated for their ability to efficiently navigate 11
the search space. As a results, with PSO, the proposed framework can reduce the model 12
size of Llama-3.1-70B by 13.44%, while keeping the loss of model accuracy at 19.25% and 13
with WOA, the model size reduction is 12.07% with 22.81% loss of model accuracy. Since 14
the accuracy degradation may occur during pruning process. The framework integrates 15
the post-process to recover the model accuracy. After this process, the pruned model loss 16
can reduce to 12.72% and 14.83% using PSO and WOA, respectively.
Abstract
Nowadays, large Language Models (LLMs) have been utilized in many aspects of 1
natural language processing. However, due to their significant size and high computational 2
demands, large computational resources are required for deployment. In this research, 3
we focus on the automated approach for size reduction of such a model. We propose the 4
framework to perform the automated pruning based on combinatorial optimization. Two 5
techniques were particularly studied, i.e., Particle Swarm Optimization (PSO) and Whale 6
Optimization Algorithm (WOA). The model pruning problem was modeled as a combina- 7
torial optimization task, whose the goal is to minimize model size while maintaining model 8
accuracy. The framework systematically explores the search space to identify the most 9
optimal pruning configurations, removing redundant or non-contributory parameters. The 10
two optimizations, PSO and WOA, were evaluated for their ability to efficiently navigate 11
the search space. As a results, with PSO, the proposed framework can reduce the model 12
size of Llama-3.1-70B by 13.44%, while keeping the loss of model accuracy at 19.25% and 13
with WOA, the model size reduction is 12.07% with 22.81% loss of model accuracy. Since 14
the accuracy degradation may occur during pruning process. The framework integrates 15
the post-process to recover the model accuracy. After this process, the pruned model loss 16
can reduce to 12.72% and 14.83% using PSO and WOA, respectively.
คำสำคัญ (Keywords)
Large Language Models; Model Pruning; Particle Swarm Optimization; 18
Resource-Constrained Devices; Whale Optimization Algorithm 19
เว็บไซต์โครงงาน
https://github.com/Pilleteer/Towards-Automated-Pruning-of-LLMs-using-Combinatorial-Optimization
วีดีโอคลิปของโครงงาน
-
ที่เก็บเวอร์ชันซอร์สโค้ด
-
ผู้นำเข้าข้อมูลครั้งแรก
พชรพล
ราชสภา
(b6410501099)
แก้ไขครั้งสุดท้าย
เมื่อ April 7, 2025, 11:21 p.m. โดย
พชรพล
ราชสภา
(b6410501099)
สถานะการอนุมัติ
รออนุมัติ