{"notes":[{"id":"model-tuning/prefix-tuning","category":"model-tuning","legacyPath":"model-tuning/2101-prefix-tuning","legacySlug":"2101-prefix-tuning","legacyPaths":[],"title":"Prefix-Tuning","subtitle":"Is it the Same or Different?","author":"Z. Yuan","paperDate":"2021-01-01","publishedOn":"2024-10-30","paperTitle":"Prefix-Tuning: Optimizing Continuous Prompts for Generation","paperUrl":"https://arxiv.org/abs/2101.00190","bibliography":{"title":"Prefix-Tuning: Optimizing Continuous Prompts for Generation","authors":["Xiang Lisa Li","Percy Liang"],"url":"https://arxiv.org/abs/2101.00190","pdfUrl":"https://arxiv.org/pdf/2101.00190","codeUrl":null,"preprint":{"id":"2101.00190","submittedOn":"2021-01-01","url":"https://arxiv.org/abs/2101.00190"},"publication":{"venue":"ACL-IJCNLP","year":2021,"kind":"conference","url":"https://doi.org/10.18653/v1/2021.acl-long.353"},"citation":"@misc{modeltuningprefixtuning2021,\n  title = {{Prefix-Tuning: Optimizing Continuous Prompts for Generation}},\n  author = {Xiang Lisa Li and Percy Liang},\n  year = {2021},\n  eprint = {2101.00190},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2101.00190}\n}\n","citationPath":"/content/papers/citations/model-tuning/prefix-tuning.fda64a808b55.bib"},"thumbnail":{"src":"/content/papers/assets/21f9834f78ef22b11e57cdbfd5c39edca8f15b7ba5ce59a699bd30d0a6c1bc7c.jpg","width":1834,"height":620},"locale":"en","path":"/content/papers/en/model-tuning/prefix-tuning.a33d1eeb4a83.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"implementation-details","text":"Implementation Details","depth":3},{"id":"dataset-setup","text":"Dataset Setup","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"performance-on-table-to-text-generation","text":"Performance on Table-to-Text Generation","depth":3},{"id":"performance-on-summarization","text":"Performance on Summarization","depth":3},{"id":"low-resource-settings","text":"Low-Resource Settings","depth":3},{"id":"extrapolation-ability","text":"Extrapolation Ability","depth":3},{"id":"how-long-should-the-prefix-be","text":"How Long Should the Prefix Be?","depth":3},{"id":"initialization-strategies","text":"Initialization Strategies","depth":3},{"id":"suitable-use-cases","text":"Suitable Use Cases","depth":3},{"id":"why-does-prefix-tuning-perform-well-on-unseen-topics","text":"Why Does Prefix-Tuning Perform Well on Unseen Topics?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"model-tuning/adapter","relation":"comparison","topic":"Adapting a model with fewer trainable parameters"},{"id":"model-tuning/lora","relation":"comparison","topic":"Adapting a model with fewer trainable parameters"},{"id":"model-tuning/autoprompt","relation":"comparison","topic":"Discrete prompts, soft prompts and prefixes"}]},{"id":"model-tuning/adapter","category":"model-tuning","legacyPath":"model-tuning/1902-adapter","legacySlug":"1902-adapter","legacyPaths":[],"title":"Adapter","subtitle":"Saving 96% of Parameters","author":"Z. Yuan","paperDate":"2019-02-02","publishedOn":"2024-05-16","paperTitle":"Parameter-Efficient Transfer Learning for NLP","paperUrl":"https://arxiv.org/abs/1902.00751","bibliography":{"title":"Parameter-Efficient Transfer Learning for NLP","authors":["Neil Houlsby","Andrei Giurgiu","Stanislaw Jastrzebski","Bruna Morrone","Quentin de Laroussilhe","Andrea Gesmundo","Mona Attariyan","Sylvain Gelly"],"url":"https://arxiv.org/abs/1902.00751","pdfUrl":"https://arxiv.org/pdf/1902.00751","codeUrl":null,"preprint":{"id":"1902.00751","submittedOn":"2019-02-02","url":"https://arxiv.org/abs/1902.00751"},"publication":{"venue":"ICML","year":2019,"kind":"conference","url":"https://proceedings.mlr.press/v97/houlsby19a.html"},"citation":"@misc{modeltuningadapter2019,\n  title = {{Parameter-Efficient Transfer Learning for NLP}},\n  author = {Neil Houlsby and Andrei Giurgiu and Stanislaw Jastrzebski and Bruna Morrone and Quentin de Laroussilhe and Andrea Gesmundo and Mona Attariyan and Sylvain Gelly},\n  year = {2019},\n  eprint = {1902.00751},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1902.00751}\n}\n","citationPath":"/content/papers/citations/model-tuning/adapter.fa08e84b43d7.bib"},"thumbnail":{"src":"/content/papers/assets/8bca77531dbddb82e18f6dedd27d962405dc5c7a0f976c6c0d09a501c918c122.jpg","width":980,"height":712},"locale":"en","path":"/content/papers/en/model-tuning/adapter.0bd929e0bef7.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"wastefulness","text":"Wastefulness","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"discussion","text":"Discussion","depth":2},{"id":"glue-evaluation","text":"GLUE Evaluation","depth":3},{"id":"ablation-study","text":"Ablation Study","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"model-tuning/lora","relation":"comparison","topic":"Adapting a model with fewer trainable parameters"},{"id":"model-tuning/prefix-tuning","relation":"comparison","topic":"Adapting a model with fewer trainable parameters"}]},{"id":"model-tuning/lora","category":"model-tuning","legacyPath":"model-tuning/2106-lora","legacySlug":"2106-lora","legacyPaths":[],"title":"LoRA","subtitle":"One in Ten Thousand LLM","author":"Z. Yuan","paperDate":"2021-06-17","publishedOn":"2024-10-26","paperTitle":"LoRA: Low-Rank Adaptation of Large Language Models","paperUrl":"https://arxiv.org/abs/2106.09685","bibliography":{"title":"LoRA: Low-Rank Adaptation of Large Language Models","authors":["Edward J. Hu","Yelong Shen","Phillip Wallis","Zeyuan Allen-Zhu","Yuanzhi Li","Shean Wang","Lu Wang","Weizhu Chen"],"url":"https://arxiv.org/abs/2106.09685","pdfUrl":"https://arxiv.org/pdf/2106.09685","codeUrl":null,"preprint":{"id":"2106.09685","submittedOn":"2021-06-17","url":"https://arxiv.org/abs/2106.09685"},"publication":{"venue":"ICLR","year":2022,"kind":"conference","url":"https://openreview.net/pdf?id=nZeVKeeFYf9"},"citation":"@misc{modeltuninglora2021,\n  title = {{LoRA: Low-Rank Adaptation of Large Language Models}},\n  author = {Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen},\n  year = {2021},\n  eprint = {2106.09685},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2106.09685}\n}\n","citationPath":"/content/papers/citations/model-tuning/lora.e23eb465372e.bib"},"thumbnail":{"src":"/content/papers/assets/547204d875b324a959dc2fb7c92601337563d9262cd3612847a3a772d092c53f.jpg","width":516,"height":472},"locale":"en","path":"/content/papers/en/model-tuning/lora.2b41c1123168.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"rank-full-rank-and-low-rank","text":"Rank, Full Rank, and Low Rank","depth":3},{"id":"why-are-low-rank-matrices-useful","text":"Why Are Low-Rank Matrices Useful?","depth":3},{"id":"low-rank-decomposition-lora","text":"Low-Rank Decomposition: LoRA","depth":3},{"id":"lora-on-transformer-models","text":"LoRA on Transformer Models","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-baselines","text":"Experimental Baselines","depth":3},{"id":"comparison-with-other-fine-tuning-methods","text":"Comparison with Other Fine-Tuning Methods","depth":3},{"id":"fine-tuning-gpt-3","text":"Fine-Tuning GPT-3","depth":3},{"id":"which-weight-matrices-should-lora-target-in-transformers","text":"Which Weight Matrices Should LoRA Target in Transformers?","depth":3},{"id":"what-is-the-optimal-rank-r-for-lora","text":"What is the Optimal Rank r for LoRA?","depth":3},{"id":"subspace-similarity-analysis","text":"Subspace Similarity Analysis","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"model-tuning/adapter","relation":"comparison","topic":"Adapting a model with fewer trainable parameters"},{"id":"model-tuning/prefix-tuning","relation":"comparison","topic":"Adapting a model with fewer trainable parameters"}]},{"id":"model-tuning/autoprompt","category":"model-tuning","legacyPath":"model-tuning/2010-autoprompt","legacySlug":"2010-autoprompt","legacyPaths":[],"title":"AutoPrompt","subtitle":"Model Language","author":"Z. Yuan","paperDate":"2020-10-29","publishedOn":"2024-06-15","paperTitle":"AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts","paperUrl":"https://arxiv.org/abs/2010.15980","bibliography":{"title":"AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts","authors":["Taylor Shin","Yasaman Razeghi","Robert L. Logan","Eric Wallace","Sameer Singh"],"url":"https://arxiv.org/abs/2010.15980","pdfUrl":"https://arxiv.org/pdf/2010.15980","codeUrl":null,"preprint":{"id":"2010.15980","submittedOn":"2020-10-29","url":"https://arxiv.org/abs/2010.15980"},"publication":{"venue":"EMNLP","year":2020,"kind":"conference","url":"https://doi.org/10.18653/v1/2020.emnlp-main.346"},"citation":"@misc{modeltuningautoprompt2020,\n  title = {{AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts}},\n  author = {Taylor Shin and Yasaman Razeghi and Robert L. Logan and Eric Wallace and Sameer Singh},\n  year = {2020},\n  eprint = {2010.15980},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2010.15980}\n}\n","citationPath":"/content/papers/citations/model-tuning/autoprompt.1466df03434e.bib"},"thumbnail":{"src":"/content/papers/assets/3a9a46741e3e735774bacb3425027a4fb544d1551d111e9321f90d72ac8b7865.jpg","width":1730,"height":544},"locale":"en","path":"/content/papers/en/model-tuning/autoprompt.7aadc16b7702.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"manually-crafting-prompts-is-time-consuming-and-tedious","text":"Manually Crafting Prompts is Time-Consuming and Tedious","depth":3},{"id":"the-model-is-highly-sensitive-to-inputs","text":"The Model is Highly Sensitive to Inputs","depth":3},{"id":"lack-of-expertise","text":"Lack of Expertise","depth":3},{"id":"new-model-new-prompts","text":"New Model, New Prompts","depth":3},{"id":"solution","text":"Solution","depth":2},{"id":"model-architecture-design","text":"Model Architecture Design","depth":3},{"id":"gradient-based-prompt-search","text":"Gradient-Based Prompt Search","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"sentiment-analysis-experiment","text":"Sentiment Analysis Experiment","depth":3},{"id":"fact-retrieval","text":"Fact Retrieval","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"model-tuning/soft-prompts","relation":"comparison","topic":"Discrete prompts, soft prompts and prefixes"},{"id":"model-tuning/prefix-tuning","relation":"comparison","topic":"Discrete prompts, soft prompts and prefixes"}]}],"text":"There are a few common ways to leverage pre-trained language models:\n\n1. **Fine-tuning**: Adjusting the model’s parameters to adapt it to new tasks.\n\n   Full fine-tuning stores gradients and optimizer states for the model. Its cost depends on model size and task; neither inaccessibility nor a large-data requirement is universal.\n\n2. **Prompt-based methods**: Designing prompts to guide the model in generating the desired outputs.\n\n   For example, if the task is to summarize a text, different models might require different prompts:\n\n   1. GPT-3: `Summarize with 3 sentences about the following article:`\n   2. T5: `summarize the following article:`\n   3. BART: `Document summarization:`\n   4. ...\n\n   **Drawback**: Each model has a different way of interpreting prompts, so the wording needs repeated testing.\n\n---\n\nIs there any other solution?\n\nOf course! Creativity is never in short supply among researchers.\n\nRecently, **AutoPrompt** introduced an automated approach to discover optimal prompts. This method combines the benefits of both fine-tuning and prompt-based techniques.\n\n- [**[20.10] AutoPrompt: Model Prompt Optimization**](/blog/papers/model-tuning/autoprompt)\n\nAutoPrompt uses gradients to guide a search for discrete trigger tokens. It does not learn unrestricted continuous prefix vectors as Prefix-Tuning does.\n\nThe prompt strings above are illustrative. Whether a T5 or BART checkpoint follows them depends on its training and adaptation; replacing the opening text alone is not sufficient.\n\n## Defining the Problem\n\n\n\n![problem](/content/papers/assets/9b657da1b13be47a6a1ba95da46b449e5312673d08ed967c8e34671bdd5ce453.jpg)\n\n\n\nThe question is how to share a large model across generation tasks without updating and storing a full set of weights for each task.\n\nPrefix-Tuning adds continuous “virtual token” activations that subsequent positions attend to. The full method optimizes prefix activations across layers, rather than inserting one new word into the input embedding layer.\n\nWait, isn’t this the same as what AutoPrompt does?\n\nExactly!\n\nThe difference lies in how the prompt is created. AutoPrompt selects and combines already **pre-trained tokens** to build a suitable prompt. In contrast, **Prefix-Tuning trains new tokens directly in the feature space** to achieve optimal results.\n\nThe distinction is the optimization space: discrete prompts are constrained to vocabulary embeddings; continuous prefixes can optimize layer activations freely. That does not make the prefix a smoother natural-language sentence—it generally is not readable text.\n\n## Solving the Problem\n\n### Model Architecture\n\n![model](/content/papers/assets/21f9834f78ef22b11e57cdbfd5c39edca8f15b7ba5ce59a699bd30d0a6c1bc7c.jpg)\n\nThe diagram above illustrates the architecture of Prefix-Tuning. The model consists of the following key components:\n\n1. **Trainable Prefix Parameters $P_\\theta$**:\n   These are newly introduced continuous activations at virtual prefix positions across the Transformer layers. The prefix length can be configured. For an encoder–decoder model such as BART, the encoder and decoder each have a prefix. Training optimizes these task-specific parameters to steer the generated output.\n\n2. **Frozen Pre-trained Language Model $\\text{LM}_\\phi$**:\n   The experiments use GPT-2 for table-to-text and BART for summarization. Parameters $\\phi$ are frozen, but computation through the backbone and backpropagation to the prefix remain necessary; this does not eliminate backbone compute or activation memory.\n\n3. **Input Sequence $x$**:\n   This is the task-specific input, such as an article for summarization or tabular data that needs to be described.\n\n4. **Output Sequence $y$**:\n   This is the output generated by the model, such as a summary of the article or a textual description of the table.\n\n### Implementation Details\n\nThe implementation of Prefix-Tuning involves several key steps:\n\n1. **Prefix Parameter Initialization**:\n\n   - A matrix of trainable prefix parameters $P_\\theta$ is created with dimensions $|P_{\\text{idx}}| \\times \\text{dim}(h_i)$, where $|P_{\\text{idx}}|$ is the length of the prefix and $\\text{dim}(h_i)$ is the total dimension of activations concatenated across layers at one position, not a single layer’s hidden width.\n\n2. **Reparameterization**:\n\n   - Directly optimizing the high-dimensional matrix $P_\\theta$ can lead to instability. To address this, the authors introduce a reparameterization strategy:\n\n     - **Low-dimensional Representation $P'_\\theta$**:\n       A smaller matrix $P'_\\theta$ with dimensions $|P_{\\text{idx}}| \\times d'$, where $d' < \\text{dim}(h_i)$.\n     - **Multi-Layer Perceptron (MLP)**:\n       The MLP maps the smaller matrix to the original high-dimensional space:\n       $$\n       P_\\theta[i, :] = \\text{MLP}_\\theta \\left( P'_\\theta[i, :] \\right)\n       $$\n     - **Optimization Process**:\n       During training, only the parameters of $P'_\\theta$ and the MLP weights $\\theta$ are optimized, while the parameters of the pre-trained language model $\\phi$ remain fixed.\n\n3. **Forward Propagation**:\n   - **Prefix Activation**:\n     For each position $i \\in P_{\\text{idx}}$, the activation value is directly taken from $P_\\theta[i, :]$:\n     $$\n     h_i = P_\\theta[i, :]\n     $$\n   - **Input Sequence Processing**:\n     For the input sequence $x$ and the output sequence $y$, the activations follow the standard calculations of the pre-trained language model, with the prefix influencing the results:\n     $$\n     h_i = \\text{LM}_\\phi(z_i, h_{<i})\n     $$\n     This describes the autoregressive case with $z=[x;y]$. Here $h_i$ concatenates activations across layers at one position (implemented using layerwise keys and values), not merely one layer’s hidden width. BART uses encoder and decoder prefixes.\n\nThe objective of training is to **maximize the log-likelihood** of the output sequence $y$:\n\n$$\n\\max_\\theta \\log p_\\phi(y | x, P_\\theta) = \\sum_{i \\in Y_{\\text{idx}}} \\log p_\\phi(z_i | h_{<i})\n$$\n\nHere, only the prefix parameters $\\theta$ are optimized, while the parameters of the language model $\\phi$ remain fixed.\n\nAfter training, materialize and store $P_\\theta$, then discard the reparameterization MLP. At inference, load the layerwise prefix activations so that the new input and output can attend to them; the high-dimensional matrix is not concatenated as ordinary text tokens.\n\n### Dataset Setup\n\n1. **Table-to-Text Generation Task**:\n\n   - The authors use three datasets, sorted by complexity and size:\n\n     1. **E2E (Novikova et al., 2017)**:\n\n        - A single-domain dataset focusing on restaurant reviews, containing around 50K data samples with 8 fields per entry.\n        - Each sample has multiple reference outputs, with an average output length of 22.9 tokens.\n        - **Evaluation metrics**: BLEU, NIST, METEOR, ROUGE-L, CIDEr.\n\n     2. **WebNLG (Gardent et al., 2017)**:\n\n        - A dataset containing 22K samples, with each entry represented as a (subject, property, object) triple.\n        - It spans 14 domains, with the test set divided into **“Seen”** and **“Unseen”** categories to assess the model’s generalization capability.\n        - The average output length is 22.5 tokens.\n        - **Evaluation metrics**: BLEU, METEOR, TER.\n\n     3. **DART (Radev et al., 2020)**:\n        - An open-domain dataset containing 82K samples, with an input format similar to WebNLG.\n        - It is compiled from multiple sources, including WikiSQL, WikiTableQuestions, E2E, and WebNLG.\n        - The average output length is 21.6 tokens.\n        - **Evaluation metrics**: BLEU, METEOR, TER, MoverScore, BERTScore, BLEURT.\n\n2. **Summarization Task**:\n\n   - **XSUM (Narayan et al., 2018)**:\n     - A large dataset with 225K news articles paired with corresponding summaries.\n     - The average article length is 431 words, while the average summary length is 23.3 words.\n     - **Evaluation metrics**: ROUGE-1, ROUGE-2, ROUGE-L.\n\n## Discussion\n\n### Performance on Table-to-Text Generation\n\n![table-to-text](/content/papers/assets/27438a4f23a4a999fbe5317cf69202fcd30b2a69488fb28e3ed26d516dfff218.jpg)\n\nDespite using only **0.1%** of the task-specific parameters, Prefix-Tuning outperforms other lightweight fine-tuning methods:\n\n- **Adapter-Tuning (ADAPTER)**: Although both methods use 0.1% of the parameters, Prefix-Tuning achieves an average **4.1-point increase** in **BLEU** scores.\n- **FT-TOP2**: Fine-tunes the top two layers nearest the output, not the first two layers nearest the input.\n\nPrefix-Tuning performs on par with, and in some cases even surpasses, full fine-tuning. Additionally, compared to Adapter-Tuning, which uses 3.0% of the model’s parameters, Prefix-Tuning remains competitive in performance.\n\nThe comparison is favorable across **E2E**, **WebNLG**, and **DART**, supporting its use on these table-to-text tasks. Its DART results show that it can handle diverse table domains and relations; generalization to unseen topics is tested separately below.\n\nExperiments with **GPT-2 Medium** and **GPT-2 Large** show that Prefix-Tuning scales well with model size, suggesting its applicability to even larger models, such as GPT-3.\n\n### Performance on Summarization\n\n\n\n![summarization](/content/papers/assets/57e66559fdc9512caa3086cf1a1ecd39cdbd216a320342d33b9b0b25115aac6e.jpg)\n\n\n\nUsing **2% of task-specific parameters**, Prefix-Tuning achieves a **ROUGE-L score of 36.05**, only slightly below the full fine-tuning score of **37.25**. However, when the parameter usage is further reduced to **0.1%**, the ROUGE-L score drops to **35.05**, indicating a performance decline with fewer parameters.\n\nPrefix-Tuning’s performance in summarization lags slightly behind full fine-tuning, likely due to the following reasons:\n\n1. **Data Volume**:\n   The **XSUM** dataset contains **four times** the average sample count of the three table-to-text datasets, requiring the model to process a larger amount of data.\n\n2. **Input Length**:\n   On average, XSUM articles are **17 times longer** than the tabular inputs, resulting in significantly more complex input sequences.\n\n3. **Task Complexity**:\n   Summarization demands more from the model, requiring it to **comprehend** the content, identify key information, and generate concise summaries.\n\nAlthough Prefix-Tuning falls short of full fine-tuning in the summarization task, it still demonstrates **strong potential in resource-constrained environments**. Different tasks pose different challenges for models, and Prefix-Tuning’s advantage diminishes when processing long and complex content. Nonetheless, the method delivers performance comparable to full fine-tuning while drastically reducing the number of trainable parameters.\n\n### Low-Resource Settings\n\n![low-resource](/content/papers/assets/76426644eb17514d08aada1a9c70c2afe116665d23ae0073d315caff548069fb.jpg)\n\nIn low-resource scenarios, **Prefix-Tuning** outperforms fine-tuning, with an **average improvement of 2.9 BLEU points** while using far fewer parameters. As the size of the dataset increases, the performance gap between Prefix-Tuning and fine-tuning gradually narrows.\n\nTo evaluate performance under low-resource conditions, the authors sampled subsets from the complete datasets—**E2E** for table-to-text generation and **XSUM** for summarization. They created smaller datasets containing **{50, 100, 200, 500}** samples. Each dataset size was tested using **five different random samples**, with **two random seeds** for each sample, resulting in the **average performance across 10 models** per size.\n\nBoth Prefix-Tuning and fine-tuning exhibit **undergeneration** issues in low-data settings, meaning some table content is omitted from the output. However, Prefix-Tuning produces outputs that remain more faithful to the input tables, even with only 100 or 200 samples. In contrast, fine-tuning occasionally generates misleading outputs—such as misreporting a customer rating as “low” when it is actually “average.” Prefix-Tuning avoids such inaccuracies, maintaining fidelity to the input data.\n\nIn these low-resource settings, Prefix-Tuning has an advantage while using fewer trainable parameters. Figure 3 also gives examples in which its descriptions reflect the input table more faithfully than fine-tuning's.\n\n### Extrapolation Ability\n\n\n\n![extrapolation](/content/papers/assets/e8dd40d0b18dcca1fc34afac14897fd18da116a5a4777ce3afa8806c9c9dbc15.jpg)\n\n\n\nThe authors also explore **Prefix-Tuning’s extrapolation capability**—the model’s ability to handle topics during testing that differ from those seen during training.\n\n- **Extrapolation Setup**:\n\n  - **Table-to-Text Generation**:\n\n    - The **WebNLG** dataset is used, with the table topics divided into:\n      - **SEEN**: Nine categories present in both the training and validation sets.\n      - **UNSEEN**: Five categories that appear only in the test set.\n\n    The model is trained on the SEEN categories, and its performance is evaluated on the UNSEEN categories.\n\n  - **Summarization**:\n    Two types of extrapolation scenarios are tested:\n    1. **News-to-Sports Transfer**:\n       The model is trained on news articles and tested on sports articles.\n    2. **Intra-News Transfer**:\n       The model is trained on articles from categories such as **World, UK, and Business** and tested on articles from other news domains, such as **Health and Technology**.\n\nPrefix-Tuning outperforms fine-tuning across **all evaluation metrics**, demonstrating **strong extrapolation capabilities** in both table-to-text generation and summarization tasks.\n\nInterestingly, **Adapter-Tuning** performs similarly to Prefix-Tuning in these extrapolation scenarios. Both methods **freeze the pre-trained language model’s parameters**. The authors suggest that preserving those parameters may help with unseen topics, while leaving the cause of the gain open.\n\n### How Long Should the Prefix Be?\n\n![prefix-length](/content/papers/assets/f93cd37cb3d8c64d207b98554aaf04b71198e202c2ce4a730b375c33451de69c.jpg)\n\nExperiments show that **longer prefixes** tend to improve performance, but performance may slightly decline beyond a certain threshold.\n\n- In the **summarization task**, the optimal prefix length is **200** tokens.\n- In the **table-to-text generation task**, the best performance is achieved with a prefix length of **10** tokens.\n\nThe tested GPU setup showed little inference-speed impact from longer prefixes. This is not a zero-cost guarantee: longer prefixes add attention keys/values and memory requirements; latency depends on model and deployment.\n\n### Initialization Strategies\n\n\n\n![initialization](/content/papers/assets/b71081a459d4cc1df3ea18aa222656e0a489539595396fef9d869c4ad452597a.jpg)\n\n\n\nIn **low-data scenarios**, the **initialization strategy** has a significant impact on performance:\n\n- **Random initialization** results in lower performance with high variability.\n- **Activation-based initialization using meaningful words** substantially improves performance.\n\nUsing words relevant to the task (e.g., \"summarization\" or \"table-to-text\") yields better results than using irrelevant words (e.g., \"elephant\" or \"divide\"). However, even initializing with irrelevant words performs better than random initialization.\n\n### Suitable Use Cases\n\nPrefix-Tuning is especially useful when many tasks need independent training, as in a personalized service. Each user's data can be kept separate and each user treated as a task, with an independently trained prefix that can be added or removed while the frozen backbone is shared. Data isolation and access control still require system-level safeguards.\n\nRequests with different users' prefixes can share a batch because the remaining language-model computation is the same.\n\nAdapters can also be stored and swapped per user. The paper’s comparison concerns differing intermediate-layer computations, which complicate ordinary shared batching—not an inability to personalize models.\n\n### Why Does Prefix-Tuning Perform Well on Unseen Topics?\n\nThe authors devote a short section to this question.\n\nFirst, a language model trained on broad corpora may retain useful general knowledge when its parameters are frozen. Both Prefix-Tuning and Adapter-Tuning leave those pretrained parameters in place and adjust only a small set of task-specific parameters.\n\nFull fine-tuning could overfit to a particular domain or a small training set. Preserving the pretrained model might help on unseen topics, but the authors say that the actual reason for the gain remains an open question.\n\nThere is also a trade-off: with the pretrained parameters frozen, lightweight tuning cannot adapt the whole model to a new task. On a more complex task that demands deeper comprehension or reasoning, full fine-tuning may still perform better.\n\nThe results may depend on the particular datasets and tasks. Unseen test topics could even overlap with the model's pretraining data, making the measured extrapolation look better. These are possible explanations or limitations, not findings established by this experiment.\n\n\n> [!TIP]\n> \n> So although the authors discuss why Prefix-Tuning might do well on unseen topics, they do not claim to know the exact cause.\n\n\n## Conclusion\n\nPrefix-Tuning is a **lightweight fine-tuning method** that guides the model to generate task-specific outputs by adding a small set of trainable prefix parameters.\n\nThis approach delivers **impressive performance** in both **table-to-text generation** and **summarization** tasks, even when using only a fraction of the parameters. In the tested **low-resource** and **unseen-topic** settings, it also performs well.\n\n\n> [!TIP]\n> \n> If you often read about language models, you may wonder about a drawback: **prefixes still occupy positions and add memory and attention work**. The summarization experiment favored a length of 200, while its source articles were truncated to 512 BPE tokens. That looks like substantial overhead. These are layerwise activation prefixes, though, so the usable source length depends on the implementation; this result does not show that every model with a 512-token window loses almost half its text capacity.\n","imageSizes":{"/content/papers/assets/9b657da1b13be47a6a1ba95da46b449e5312673d08ed967c8e34671bdd5ce453.jpg":{"width":1224,"height":924},"/content/papers/assets/21f9834f78ef22b11e57cdbfd5c39edca8f15b7ba5ce59a699bd30d0a6c1bc7c.jpg":{"width":1834,"height":620},"/content/papers/assets/27438a4f23a4a999fbe5317cf69202fcd30b2a69488fb28e3ed26d516dfff218.jpg":{"width":1760,"height":512},"/content/papers/assets/57e66559fdc9512caa3086cf1a1ecd39cdbd216a320342d33b9b0b25115aac6e.jpg":{"width":1224,"height":324},"/content/papers/assets/76426644eb17514d08aada1a9c70c2afe116665d23ae0073d315caff548069fb.jpg":{"width":1816,"height":600},"/content/papers/assets/e8dd40d0b18dcca1fc34afac14897fd18da116a5a4777ce3afa8806c9c9dbc15.jpg":{"width":1224,"height":300},"/content/papers/assets/f93cd37cb3d8c64d207b98554aaf04b71198e202c2ce4a730b375c33451de69c.jpg":{"width":1224,"height":440},"/content/papers/assets/b71081a459d4cc1df3ea18aa222656e0a489539595396fef9d869c4ad452597a.jpg":{"width":1224,"height":692}}}
