Back to paper library
arXiv 2021ACL-IJCNLP 2021Model adaptationPaper notes

Prefix-Tuning

Is it the Same or Different?

Original paper

Prefix-Tuning: Optimizing Continuous Prompts for Generation

Paper authors

Xiang Lisa Li · Percy Liang

Publication
ACL-IJCNLP 2021Conference
First submitted to arXiv
arXiv:2101.00190
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{modeltuningprefixtuning2021,
  title = {{Prefix-Tuning: Optimizing Continuous Prompts for Generation}},
  author = {Xiang Lisa Li and Percy Liang},
  year = {2021},
  eprint = {2101.00190},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.00190}
}
On this page

There are a few common ways to leverage pre-trained language models:

  1. Fine-tuning: Adjusting the model’s parameters to adapt it to new tasks.

    Full fine-tuning stores gradients and optimizer states for the model. Its cost depends on model size and task; neither inaccessibility nor a large-data requirement is universal.

  2. Prompt-based methods: Designing prompts to guide the model in generating the desired outputs.

    For example, if the task is to summarize a text, different models might require different prompts:

    1. GPT-3: Summarize with 3 sentences about the following article:
    2. T5: summarize the following article:
    3. BART: Document summarization:
    4. ...

    Drawback: Each model has a different way of interpreting prompts, so the wording needs repeated testing.


Is there any other solution?

Of course! Creativity is never in short supply among researchers.

Recently, AutoPrompt introduced an automated approach to discover optimal prompts. This method combines the benefits of both fine-tuning and prompt-based techniques.

AutoPrompt uses gradients to guide a search for discrete trigger tokens. It does not learn unrestricted continuous prefix vectors as Prefix-Tuning does.

The prompt strings above are illustrative. Whether a T5 or BART checkpoint follows them depends on its training and adaptation; replacing the opening text alone is not sufficient.

Defining the Problem

The question is how to share a large model across generation tasks without updating and storing a full set of weights for each task.

Prefix-Tuning adds continuous “virtual token” activations that subsequent positions attend to. The full method optimizes prefix activations across layers, rather than inserting one new word into the input embedding layer.

Wait, isn’t this the same as what AutoPrompt does?

Exactly!

The difference lies in how the prompt is created. AutoPrompt selects and combines already pre-trained tokens to build a suitable prompt. In contrast, Prefix-Tuning trains new tokens directly in the feature space to achieve optimal results.

The distinction is the optimization space: discrete prompts are constrained to vocabulary embeddings; continuous prefixes can optimize layer activations freely. That does not make the prefix a smoother natural-language sentence—it generally is not readable text.

Solving the Problem

Model Architecture

The diagram above illustrates the architecture of Prefix-Tuning. The model consists of the following key components:

  1. Trainable Prefix Parameters PθP_\theta: These are newly introduced continuous activations at virtual prefix positions across the Transformer layers. The prefix length can be configured. For an encoder–decoder model such as BART, the encoder and decoder each have a prefix. Training optimizes these task-specific parameters to steer the generated output.

  2. Frozen Pre-trained Language Model LMϕ\text{LM}_\phi: The experiments use GPT-2 for table-to-text and BART for summarization. Parameters ϕ\phi are frozen, but computation through the backbone and backpropagation to the prefix remain necessary; this does not eliminate backbone compute or activation memory.

  3. Input Sequence xx: This is the task-specific input, such as an article for summarization or tabular data that needs to be described.

  4. Output Sequence yy: This is the output generated by the model, such as a summary of the article or a textual description of the table.

Implementation Details

The implementation of Prefix-Tuning involves several key steps:

  1. Prefix Parameter Initialization:

    • A matrix of trainable prefix parameters PθP_\theta is created with dimensions ∣Pidx∣×dim(hi)|P_{\text{idx}}| \times \text{dim}(h_i), where ∣Pidx∣|P_{\text{idx}}| is the length of the prefix and dim(hi)\text{dim}(h_i) is the total dimension of activations concatenated across layers at one position, not a single layer’s hidden width.
  2. Reparameterization:

    • Directly optimizing the high-dimensional matrix PθP_\theta can lead to instability. To address this, the authors introduce a reparameterization strategy:

      • Low-dimensional Representation Pθ′P'_\theta: A smaller matrix Pθ′P'_\theta with dimensions ∣Pidx∣×d′|P_{\text{idx}}| \times d', where d′<dim(hi)d' < \text{dim}(h_i).
      • Multi-Layer Perceptron (MLP): The MLP maps the smaller matrix to the original high-dimensional space: Pθ[i,:]=MLPθ(Pθ′[i,:])P_\theta[i, :] = \text{MLP}_\theta \left( P'_\theta[i, :] \right)
      • Optimization Process: During training, only the parameters of Pθ′P'_\theta and the MLP weights θ\theta are optimized, while the parameters of the pre-trained language model ϕ\phi remain fixed.
  3. Forward Propagation:

    • Prefix Activation: For each position i∈Pidxi \in P_{\text{idx}}, the activation value is directly taken from Pθ[i,:]P_\theta[i, :]: hi=Pθ[i,:]h_i = P_\theta[i, :]
    • Input Sequence Processing: For the input sequence xx and the output sequence yy, the activations follow the standard calculations of the pre-trained language model, with the prefix influencing the results: hi=LMϕ(zi,h<i)h_i = \text{LM}_\phi(z_i, h_{<i}) This describes the autoregressive case with z=[x;y]z=[x;y]. Here hih_i concatenates activations across layers at one position (implemented using layerwise keys and values), not merely one layer’s hidden width. BART uses encoder and decoder prefixes.

The objective of training is to maximize the log-likelihood of the output sequence yy:

max⁡θlog⁡pϕ(y∣x,Pθ)=∑i∈Yidxlog⁡pϕ(zi∣h<i)\max_\theta \log p_\phi(y | x, P_\theta) = \sum_{i \in Y_{\text{idx}}} \log p_\phi(z_i | h_{<i})

Here, only the prefix parameters θ\theta are optimized, while the parameters of the language model ϕ\phi remain fixed.

After training, materialize and store PθP_\theta, then discard the reparameterization MLP. At inference, load the layerwise prefix activations so that the new input and output can attend to them; the high-dimensional matrix is not concatenated as ordinary text tokens.

Dataset Setup

  1. Table-to-Text Generation Task:

    • The authors use three datasets, sorted by complexity and size:

      1. E2E (Novikova et al., 2017):

        • A single-domain dataset focusing on restaurant reviews, containing around 50K data samples with 8 fields per entry.
        • Each sample has multiple reference outputs, with an average output length of 22.9 tokens.
        • Evaluation metrics: BLEU, NIST, METEOR, ROUGE-L, CIDEr.
      2. WebNLG (Gardent et al., 2017):

        • A dataset containing 22K samples, with each entry represented as a (subject, property, object) triple.
        • It spans 14 domains, with the test set divided into “Seen” and “Unseen” categories to assess the model’s generalization capability.
        • The average output length is 22.5 tokens.
        • Evaluation metrics: BLEU, METEOR, TER.
      3. DART (Radev et al., 2020):

        • An open-domain dataset containing 82K samples, with an input format similar to WebNLG.
        • It is compiled from multiple sources, including WikiSQL, WikiTableQuestions, E2E, and WebNLG.
        • The average output length is 21.6 tokens.
        • Evaluation metrics: BLEU, METEOR, TER, MoverScore, BERTScore, BLEURT.
  2. Summarization Task:

    • XSUM (Narayan et al., 2018):
      • A large dataset with 225K news articles paired with corresponding summaries.
      • The average article length is 431 words, while the average summary length is 23.3 words.
      • Evaluation metrics: ROUGE-1, ROUGE-2, ROUGE-L.

Discussion

Performance on Table-to-Text Generation

Despite using only 0.1% of the task-specific parameters, Prefix-Tuning outperforms other lightweight fine-tuning methods:

  • Adapter-Tuning (ADAPTER): Although both methods use 0.1% of the parameters, Prefix-Tuning achieves an average 4.1-point increase in BLEU scores.
  • FT-TOP2: Fine-tunes the top two layers nearest the output, not the first two layers nearest the input.

Prefix-Tuning performs on par with, and in some cases even surpasses, full fine-tuning. Additionally, compared to Adapter-Tuning, which uses 3.0% of the model’s parameters, Prefix-Tuning remains competitive in performance.

The comparison is favorable across E2E, WebNLG, and DART, supporting its use on these table-to-text tasks. Its DART results show that it can handle diverse table domains and relations; generalization to unseen topics is tested separately below.

Experiments with GPT-2 Medium and GPT-2 Large show that Prefix-Tuning scales well with model size, suggesting its applicability to even larger models, such as GPT-3.

Performance on Summarization

Using 2% of task-specific parameters, Prefix-Tuning achieves a ROUGE-L score of 36.05, only slightly below the full fine-tuning score of 37.25. However, when the parameter usage is further reduced to 0.1%, the ROUGE-L score drops to 35.05, indicating a performance decline with fewer parameters.

Prefix-Tuning’s performance in summarization lags slightly behind full fine-tuning, likely due to the following reasons:

  1. Data Volume: The XSUM dataset contains four times the average sample count of the three table-to-text datasets, requiring the model to process a larger amount of data.

  2. Input Length: On average, XSUM articles are 17 times longer than the tabular inputs, resulting in significantly more complex input sequences.

  3. Task Complexity: Summarization demands more from the model, requiring it to comprehend the content, identify key information, and generate concise summaries.

Although Prefix-Tuning falls short of full fine-tuning in the summarization task, it still demonstrates strong potential in resource-constrained environments. Different tasks pose different challenges for models, and Prefix-Tuning’s advantage diminishes when processing long and complex content. Nonetheless, the method delivers performance comparable to full fine-tuning while drastically reducing the number of trainable parameters.

Low-Resource Settings

In low-resource scenarios, Prefix-Tuning outperforms fine-tuning, with an average improvement of 2.9 BLEU points while using far fewer parameters. As the size of the dataset increases, the performance gap between Prefix-Tuning and fine-tuning gradually narrows.

To evaluate performance under low-resource conditions, the authors sampled subsets from the complete datasets—E2E for table-to-text generation and XSUM for summarization. They created smaller datasets containing {50, 100, 200, 500} samples. Each dataset size was tested using five different random samples, with two random seeds for each sample, resulting in the average performance across 10 models per size.

Both Prefix-Tuning and fine-tuning exhibit undergeneration issues in low-data settings, meaning some table content is omitted from the output. However, Prefix-Tuning produces outputs that remain more faithful to the input tables, even with only 100 or 200 samples. In contrast, fine-tuning occasionally generates misleading outputs—such as misreporting a customer rating as “low” when it is actually “average.” Prefix-Tuning avoids such inaccuracies, maintaining fidelity to the input data.

In these low-resource settings, Prefix-Tuning has an advantage while using fewer trainable parameters. Figure 3 also gives examples in which its descriptions reflect the input table more faithfully than fine-tuning's.

Extrapolation Ability

The authors also explore Prefix-Tuning’s extrapolation capability—the model’s ability to handle topics during testing that differ from those seen during training.

  • Extrapolation Setup:

    • Table-to-Text Generation:

      • The WebNLG dataset is used, with the table topics divided into:
        • SEEN: Nine categories present in both the training and validation sets.
        • UNSEEN: Five categories that appear only in the test set.

      The model is trained on the SEEN categories, and its performance is evaluated on the UNSEEN categories.

    • Summarization: Two types of extrapolation scenarios are tested:

      1. News-to-Sports Transfer: The model is trained on news articles and tested on sports articles.
      2. Intra-News Transfer: The model is trained on articles from categories such as World, UK, and Business and tested on articles from other news domains, such as Health and Technology.

Prefix-Tuning outperforms fine-tuning across all evaluation metrics, demonstrating strong extrapolation capabilities in both table-to-text generation and summarization tasks.

Interestingly, Adapter-Tuning performs similarly to Prefix-Tuning in these extrapolation scenarios. Both methods freeze the pre-trained language model’s parameters. The authors suggest that preserving those parameters may help with unseen topics, while leaving the cause of the gain open.

How Long Should the Prefix Be?

Experiments show that longer prefixes tend to improve performance, but performance may slightly decline beyond a certain threshold.

  • In the summarization task, the optimal prefix length is 200 tokens.
  • In the table-to-text generation task, the best performance is achieved with a prefix length of 10 tokens.

The tested GPU setup showed little inference-speed impact from longer prefixes. This is not a zero-cost guarantee: longer prefixes add attention keys/values and memory requirements; latency depends on model and deployment.

Initialization Strategies

In low-data scenarios, the initialization strategy has a significant impact on performance:

  • Random initialization results in lower performance with high variability.
  • Activation-based initialization using meaningful words substantially improves performance.

Using words relevant to the task (e.g., "summarization" or "table-to-text") yields better results than using irrelevant words (e.g., "elephant" or "divide"). However, even initializing with irrelevant words performs better than random initialization.

Suitable Use Cases

Prefix-Tuning is especially useful when many tasks need independent training, as in a personalized service. Each user's data can be kept separate and each user treated as a task, with an independently trained prefix that can be added or removed while the frozen backbone is shared. Data isolation and access control still require system-level safeguards.

Requests with different users' prefixes can share a batch because the remaining language-model computation is the same.

Adapters can also be stored and swapped per user. The paper’s comparison concerns differing intermediate-layer computations, which complicate ordinary shared batching—not an inability to personalize models.

Why Does Prefix-Tuning Perform Well on Unseen Topics?

The authors devote a short section to this question.

First, a language model trained on broad corpora may retain useful general knowledge when its parameters are frozen. Both Prefix-Tuning and Adapter-Tuning leave those pretrained parameters in place and adjust only a small set of task-specific parameters.

Full fine-tuning could overfit to a particular domain or a small training set. Preserving the pretrained model might help on unseen topics, but the authors say that the actual reason for the gain remains an open question.

There is also a trade-off: with the pretrained parameters frozen, lightweight tuning cannot adapt the whole model to a new task. On a more complex task that demands deeper comprehension or reasoning, full fine-tuning may still perform better.

The results may depend on the particular datasets and tasks. Unseen test topics could even overlap with the model's pretraining data, making the measured extrapolation look better. These are possible explanations or limitations, not findings established by this experiment.

Conclusion

Prefix-Tuning is a lightweight fine-tuning method that guides the model to generate task-specific outputs by adding a small set of trainable prefix parameters.

This approach delivers impressive performance in both table-to-text generation and summarization tasks, even when using only a fraction of the parameters. In the tested low-resource and unseen-topic settings, it also performs well.