Título: Benchmarking quantized LLaMa-based models on the Brazilian Secondary School Exam
Autores: Matheus L. O. Santos, Claudio E. C. Campelo
Resumo: Although Large Language Models (LLMs) represent a revolution in the way we interact with computers, allowing the construction of complex questions and the ability to reason over a sequence of statements, their use is restricted due to the need for dedicated hardware for execution. In this study, we evaluate the performance of LLMs based on the 7 and 13 billion LLaMA models, subjected to a quantization process and run on home hardware. The models considered were Alpaca, Koala, and Vicuna. To evaluate the effectiveness of these models, we developed a database containing 1,006 questions from the ENEM (Brazilian National Secondary School Exam). Our analysis revealed that the best performing models achieved an accuracy of approximately 46% for the original texts of the Portuguese questions and 49% on their English translations. In addition, we evaluated the computational efficiency of the models by measuring the time required for execution. On average, the 7 and 13 billion LLMs took approximately 20 and 50 seconds, respectively, to process the queries on a machine equipped with an AMD Ryzen 5 3600x processor
Palavras-chave: Large language models, LLMs, ENEM, GGML, LLaMA, Quantization
Páginas: 8
Código DOI: 10.21528/CBIC2023-177
Artigo em pdf: CBIC_2023_paper177.pdf
Arquivo BibTeX: CBIC_2023_177.bib