Hizkuntzetarako neurona espezifikoak LLMetan?

Abstract

Large Language Models (LLM) are revolutionary artificial intelligence neural networks with billions of parameters. This study investigates language-specific neurons in LLMs, focusing on their adaptation to Basque. Using the Language Activation Probability Entropy (LAPE) metric, we identify neurons in Llama-3.1-8B and its Basque-adapted variant (Llama-eus-8B) that are specialized for Basque, French, Spanish, and English. Experiments reveal that language-specific neurons predominantly cluster in final layers, with Basque showing the highest count. Perplexity analysis indicates that deactivating these neurons disproportionately affects their target languages when these are not predominant in the model, validating their specificity. These findings highlight the interplay between language adaptation and neuron distribution, offering insights for optimizing LLMs for low-resource languages.