InA: Inhibition Adaption on pre-trained language models

 Fine-tuning pre-trained language models (LMs) may not always be the most practical approach for downstream tasks. While adaptation fine-tuning methods have shown promising results, a clearer explanation of their mechanisms and further inhibition of the transmission of information is needed. To address this, we propose an Inhibition Adaptation (InA) fine-tuning method that aims to reduce the number of added tunable weights and appropriately reweight knowledge derived from pre-trained LMs. The InA method involves (1) inserting a small trainable vector into each Transformer attention architecture and (2) setting a threshold to directly eliminate irrelevant knowledge. This approach draws inspiration from the shunting inhibition, which allows the inhibition of specific neurons to gate other functional neurons. With the inhibition mechanism, InA achieves competitive or even superior performance compared to other fine-tuning methods on 𝐵𝐸𝑅𝑇−𝑙𝑎𝑟𝑔𝑒, 𝑅𝑜𝐵𝐸𝑅𝑇𝑎−𝑙𝑎𝑟𝑔𝑒, and 𝐷𝑒𝐵𝐸𝑅𝑇𝑎−𝑙𝑎𝑟𝑔𝑒 for text classification and question-answering tasks.