Back to paper library
arXiv 2025Language & TransformersPaper notes

Command A

Training in separate dojos

Original paper

Command A: An Enterprise-Ready Large Language Model

Paper authors

Team Cohere · Aakanksha · Arash Ahmadian …

Show all 229 authors

Team Cohere · Aakanksha · Arash Ahmadian · Marwan Ahmed · Jay Alammar · Milad Alizadeh · Yazeed Alnumay · Sophia Althammer · Arkady Arkhangorodsky · Viraat Aryabumi · Dennis Aumiller · Raphaël Avalos · Zahara Aviv · Sammie Bae · Saurabh Baji · Alexandre Barbet · Max Bartolo · Björn Bebensee · Neeral Beladia · Walter Beller-Morales · Alexandre Bérard · Andrew Berneshawi · Anna Bialas · Phil Blunsom · Matt Bobkin · Adi Bongale · Sam Braun · Maxime Brunet · Samuel Cahyawijaya · David Cairuz · Jon Ander Campos · Cassie Cao · Kris Cao · Roman Castagné · Julián Cendrero · Leila Chan Currie · Yash Chandak · Diane Chang · Giannis Chatziveroglou · Hongyu Chen · Claire Cheng · Alexis Chevalier · Justin T. Chiu · Eugene Cho · Eugene Choi · Eujeong Choi · Tim Chung · Volkan Cirik · Ana Cismaru · Pierre Clavier · Henry Conklin · Lucas Crawhall-Stein · Devon Crouse · Andres Felipe Cruz-Salinas · Ben Cyrus · Daniel D'souza · Hugo Dalla-Torre · John Dang · William Darling · Omar Darwiche Domingues · Saurabh Dash · Antoine Debugne · Théo Dehaze · Shaan Desai · Joan Devassy · Rishit Dholakia · Kyle Duffy · Ali Edalati · Ace Eldeib · Abdullah Elkady · Sarah Elsharkawy · Irem Ergün · Beyza Ermis · Marzieh Fadaee · Boyu Fan · Lucas Fayoux · Yannis Flet-Berliac · Nick Frosst · Matthias Gallé · Wojciech Galuba · Utsav Garg · Matthieu Geist · Mohammad Gheshlaghi Azar · Ellen Gilsenan-McMahon · Seraphina Goldfarb-Tarrant · Tomas Goldsack · Aidan Gomez · Victor Machado Gonzaga · Nithya Govindarajan · Manoj Govindassamy · Nathan Grinsztajn · Nikolas Gritsch · Patrick Gu · Shangmin Guo · Kilian Haefeli · Rod Hajjar · Tim Hawes · Jingyi He · Sebastian Hofstätter · Sungjin Hong · Sara Hooker · Tom Hosking · Stephanie Howe · Eric Hu · Renjie Huang · Hemant Jain · Ritika Jain · Nick Jakobi · Madeline Jenkins · JJ Jordan · Dhruti Joshi · Jason Jung · Trushant Kalyanpur · Siddhartha Rao Kamalakara · Julia Kedrzycki · Gokce Keskin · Edward Kim · Joon Kim · Wei-Yin Ko · Tom Kocmi · Michael Kozakov · Wojciech Kryściński · Arnav Kumar Jain · Komal Kumar Teru · Sander Land · Michael Lasby · Olivia Lasche · Justin Lee · Patrick Lewis · Jeffrey Li · Jonathan Li · Hangyu Lin · Acyr Locatelli · Kevin Luong · Raymond Ma · Lukáš Mach · Marina Machado · Joanne Magbitang · Brenda Malacara Lopez · Aryan Mann · Kelly Marchisio · Olivia Markham · Alexandre Matton · Alex McKinney · Dominic McLoughlin · Jozef Mokry · Adrien Morisot · Autumn Moulder · Harry Moynehan · Maximilian Mozes · Vivek Muppalla · Lidiya Murakhovska · Hemangani Nagarajan · Alekhya Nandula · Hisham Nasir · Shauna Nehra · Josh Netto-Rosen · Daniel Ohashi · James Owers-Bardsley · Jason Ozuzu · Dennis Padilla · Gloria Park · Sam Passaglia · Jeremy Pekmez · Laura Penstone · Aleksandra Piktus · Case Ploeg · Andrew Poulton · Youran Qi · Shubha Raghvendra · Miguel Ramos · Ekagra Ranjan · Pierre Richemond · Cécile Robert-Michon · Aurélien Rodriguez · Sudip Roy · Sebastian Ruder · Laura Ruis · Louise Rust · Anubhav Sachan · Alejandro Salamanca · Kailash Karthik Saravanakumar · Isha Satyakam · Alice Schoenauer Sebag · Priyanka Sen · Sholeh Sepehri · Preethi Seshadri · Ye Shen · Tom Sherborne · Sylvie Shang Shi · Sanal Shivaprasad · Vladyslav Shmyhlo · Anirudh Shrinivason · Inna Shteinbuk · Amir Shukayev · Mathieu Simard · Ella Snyder · Ava Spataru · Victoria Spooner · Trisha Starostina · Florian Strub · Yixuan Su · Jimin Sun · Dwarak Talupuru · Eugene Tarassov · Elena Tommasone · Jennifer Tracey · Billy Trend · Evren Tumer · Ahmet Üstün · Bharat Venkitesh · David Venuto · Pat Verga · Maxime Voisin · Alex Wang · Donglu Wang · Shijian Wang · Edmond Wen · Naomi White · Jesse Willman · Marysia Winkels · Chen Xia · Jessica Xie · Minjie Xu · Bowen Yang · Tan Yi-Chern · Ivan Zhang · Zhenyu Zhao · Zhoujie Zhao

Publication
Publication venue not yet verified
First submitted to arXiv
arXiv:2504.00698
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{transformerscommanda2025,
  title = {{Command A: An Enterprise-Ready Large Language Model}},
  author = {Team Cohere and Aakanksha and Arash Ahmadian and Marwan Ahmed and Jay Alammar and Milad Alizadeh and Yazeed Alnumay and Sophia Althammer and Arkady Arkhangorodsky and Viraat Aryabumi and Dennis Aumiller and Raphaël Avalos and Zahara Aviv and Sammie Bae and Saurabh Baji and Alexandre Barbet and Max Bartolo and Björn Bebensee and Neeral Beladia and Walter Beller-Morales and Alexandre Bérard and Andrew Berneshawi and Anna Bialas and Phil Blunsom and Matt Bobkin and Adi Bongale and Sam Braun and Maxime Brunet and Samuel Cahyawijaya and David Cairuz and Jon Ander Campos and Cassie Cao and Kris Cao and Roman Castagné and Julián Cendrero and Leila Chan Currie and Yash Chandak and Diane Chang and Giannis Chatziveroglou and Hongyu Chen and Claire Cheng and Alexis Chevalier and Justin T. Chiu and Eugene Cho and Eugene Choi and Eujeong Choi and Tim Chung and Volkan Cirik and Ana Cismaru and Pierre Clavier and Henry Conklin and Lucas Crawhall-Stein and Devon Crouse and Andres Felipe Cruz-Salinas and Ben Cyrus and Daniel D'souza and Hugo Dalla-Torre and John Dang and William Darling and Omar Darwiche Domingues and Saurabh Dash and Antoine Debugne and Théo Dehaze and Shaan Desai and Joan Devassy and Rishit Dholakia and Kyle Duffy and Ali Edalati and Ace Eldeib and Abdullah Elkady and Sarah Elsharkawy and Irem Ergün and Beyza Ermis and Marzieh Fadaee and Boyu Fan and Lucas Fayoux and Yannis Flet-Berliac and Nick Frosst and Matthias Gallé and Wojciech Galuba and Utsav Garg and Matthieu Geist and Mohammad Gheshlaghi Azar and Ellen Gilsenan-McMahon and Seraphina Goldfarb-Tarrant and Tomas Goldsack and Aidan Gomez and Victor Machado Gonzaga and Nithya Govindarajan and Manoj Govindassamy and Nathan Grinsztajn and Nikolas Gritsch and Patrick Gu and Shangmin Guo and Kilian Haefeli and Rod Hajjar and Tim Hawes and Jingyi He and Sebastian Hofstätter and Sungjin Hong and Sara Hooker and Tom Hosking and Stephanie Howe and Eric Hu and Renjie Huang and Hemant Jain and Ritika Jain and Nick Jakobi and Madeline Jenkins and JJ Jordan and Dhruti Joshi and Jason Jung and Trushant Kalyanpur and Siddhartha Rao Kamalakara and Julia Kedrzycki and Gokce Keskin and Edward Kim and Joon Kim and Wei-Yin Ko and Tom Kocmi and Michael Kozakov and Wojciech Kryściński and Arnav Kumar Jain and Komal Kumar Teru and Sander Land and Michael Lasby and Olivia Lasche and Justin Lee and Patrick Lewis and Jeffrey Li and Jonathan Li and Hangyu Lin and Acyr Locatelli and Kevin Luong and Raymond Ma and Lukáš Mach and Marina Machado and Joanne Magbitang and Brenda Malacara Lopez and Aryan Mann and Kelly Marchisio and Olivia Markham and Alexandre Matton and Alex McKinney and Dominic McLoughlin and Jozef Mokry and Adrien Morisot and Autumn Moulder and Harry Moynehan and Maximilian Mozes and Vivek Muppalla and Lidiya Murakhovska and Hemangani Nagarajan and Alekhya Nandula and Hisham Nasir and Shauna Nehra and Josh Netto-Rosen and Daniel Ohashi and James Owers-Bardsley and Jason Ozuzu and Dennis Padilla and Gloria Park and Sam Passaglia and Jeremy Pekmez and Laura Penstone and Aleksandra Piktus and Case Ploeg and Andrew Poulton and Youran Qi and Shubha Raghvendra and Miguel Ramos and Ekagra Ranjan and Pierre Richemond and Cécile Robert-Michon and Aurélien Rodriguez and Sudip Roy and Sebastian Ruder and Laura Ruis and Louise Rust and Anubhav Sachan and Alejandro Salamanca and Kailash Karthik Saravanakumar and Isha Satyakam and Alice Schoenauer Sebag and Priyanka Sen and Sholeh Sepehri and Preethi Seshadri and Ye Shen and Tom Sherborne and Sylvie Shang Shi and Sanal Shivaprasad and Vladyslav Shmyhlo and Anirudh Shrinivason and Inna Shteinbuk and Amir Shukayev and Mathieu Simard and Ella Snyder and Ava Spataru and Victoria Spooner and Trisha Starostina and Florian Strub and Yixuan Su and Jimin Sun and Dwarak Talupuru and Eugene Tarassov and Elena Tommasone and Jennifer Tracey and Billy Trend and Evren Tumer and Ahmet Üstün and Bharat Venkitesh and David Venuto and Pat Verga and Maxime Voisin and Alex Wang and Donglu Wang and Shijian Wang and Edmond Wen and Naomi White and Jesse Willman and Marysia Winkels and Chen Xia and Jessica Xie and Minjie Xu and Bowen Yang and Tan Yi-Chern and Ivan Zhang and Zhenyu Zhao and Zhoujie Zhao},
  year = {2025},
  eprint = {2504.00698},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.00698}
}
On this page

Cohere is a Canadian language-model company focused on enterprise customers; one of its founders, Aidan Gomez, is a coauthor of "Attention Is All You Need." Its models emphasize retrieval-augmented generation (RAG), tool use, and multilinguality. They usually don't compete for first place on chat leaderboards, and instead stress being useful and easy to deploy in real enterprise settings.

In March 2025, Cohere released Command A. It is a 111-billion-parameter model that supports 23 business languages and can be deployed on just two A100s or H100s. The weights are released under a non-commercial license (CC-BY-NC) for research use.

This technical report focuses on the post-training pipeline. The usual approach takes one model through SFT, preference learning, and reinforcement learning in sequence. Command A instead has several teams each train expert models that excel at different capabilities, and in the end combines them into one with model merging. The report says this is the first time model merging has been made the core of post-training at this scale.

Defining the problem

An enterprise language model has to do many things at once: answer questions from documents, call tools, write code, do math, speak many languages, and follow safety rules, all while handling very long contexts.

These capabilities are trained in very different ways. Code and math are easy to verify, so data can be synthesized in bulk and reinforcement learning can use verifiable rewards. Safety and RAG rely mostly on human annotation, with far less data, and are better suited to training with preference comparisons. Mixing all of this data together and training one model with a single set of hyperparameters makes it hard to get every capability to its best.

The same problem shows up on the organizational side. When multiple teams improve different capabilities at the same time, all working on the same model, their experiments interfere with one another, and each team ends up waiting on the others.

Can each capability be trained to its best separately and then cheaply combined? That is the question the whole report tries to answer.

Solving the problem

Architecture and pretraining

First, a quick look at the base. Command A is a decoder-only Transformer with three main design choices.

  1. Interleaved attention layers: in every 4 layers, 3 use sliding-window attention and 1 uses full attention, a 3:1 ratio. The sliding-window layers use RoPE, while the full-attention layers use no positional encoding at all (NoPE).
  2. Parallel Transformer blocks: attention and the feedforward network are computed in parallel rather than one after the other. The authors say performance is comparable and throughput improves markedly.
  3. Others: GQA, SwiGLU, no bias terms, and a single embedding matrix shared by input and output. The vocabulary has 255,000 tokens, so sharing embeddings saves a lot of memory.

Training uses µP to tune hyperparameters on small models and then transfers them directly to the large model. The main matrix multiplications are computed in FP8. The authors observed no instability, but training in FP8 the whole way slightly hurts downstream performance, so they train in BF16 for a while before switching to FP8. Pretraining ends with a cooldown phase in which the learning rate drops from 2.5×10−42.5\times10^{-4} to 10−610^{-6} over 50,000 steps; the context length first stays at 8K, then is gradually extended to 32K, 128K, and 256K.

Post-training: two merges

Original Figure 3. A post-training flowchart. At the far left is an instruction-tuned checkpoint, followed by several SFT experts trained in parallel, including RAG & tool use & agents, multilingual, code, math & reasoning, long context, safety, and instruction following; these experts go through one model merge (drawn as a set of adjustable expert-weight sliders) to produce the SFT Soup. Next, matching reinforcement-learning experts are trained in parallel on top of the SFT Soup and merged once more to produce the RL Soup. Finally, a polishing stage runs SFT, offline preference RL, RLHF, and offline preference RL in sequence, producing Command A.
Original Figure 3, Cohere, from PDF page 5, CC BY 4.0.

Read this figure from left to right.

  1. Instruct model: first run one round of SFT on the pretrained model to get an instruct model with basic capabilities.
  2. SFT experts: starting from the same instruct model, train several experts in parallel, each with its own data mix and hyperparameters. The main text lists six: code, safety, RAG, math, multilingual, and general plus long context.
  3. SFT Soup: merge these experts into a single model.
  4. RL experts: starting from the SFT Soup, train another set of reinforcement-learning experts in parallel, each domain with a method that suits it: verifiable rewards for math and code, preference comparisons for safety and multilingual.
  5. RL Soup: merge once more.
  6. Polishing: finally, run a round of global alignment to fix problems left by merging and bring the overall style closer to human preferences.

The two "Model Merging" columns in the figure each show a row of sliders, meaning each expert's weight in the merge can be adjusted. This is the key to the whole pipeline. If some capability falls short after merging, there is no need to retrain; just adjust the sliders and merge again.

Four uses of merging

The report sorts model merging into four uses, and Command A employs all four:

  1. Expert merging: combining models that excel at different capabilities into one. This is the main act.
  2. Polyak averaging: averaging several checkpoints from the same training run to reduce stochastic-gradient noise.
  3. Seed merging: averaging models trained with different random seeds to reduce the effects of chance.
  4. Interpolation to recover capabilities: experts trained only on short contexts lose long-context ability. Interpolating an expert with the original base model recovers most of the long-context ability while keeping the expert's new capabilities. The authors point out that this is very close to Google DeepMind's WARP.

Linear merging is enough

Expert merging uses a linear average:

θmerged=∑k=1Kwk θk,∑kwk=1\theta_{\text{merged}}=\sum_{k=1}^{K}w_k\,\theta_k,\qquad \sum_k w_k = 1

The authors tried SLERP and task arithmetic and found no clear improvement, only added complexity. Linear merging also has a practical property: it is associative, so a merge of merges can still be written as a single merge, which makes the whole pipeline easier to trace.

The weights wkw_k are searched by hand. Raising an expert's weight roughly improves performance in that domain, an intuitive relationship. But because all weights must sum to 1, the other domains' weights go down accordingly, and how much those domains will drop is hard to predict. The authors' search combines the two: turn up the expert for whichever domain falls short, then, centered on the current candidate, apply small perturbations to each expert's weight and search by brute force.

The whole process tracked 500 evaluation metrics. The authors say that looking after this many metrics at once would have been much harder with traditional centralized development.

Consistency matters more than peak scores

Whether a merge succeeds depends largely on whether the experts are "compatible" with one another. The authors sum up several lessons:

All experts start from the same instruct model. One reason is special tokens. Some experts use special tokens, such as those for tool calls, that other experts never use, so in those other experts the embeddings for these tokens are never trained. The authors tried taking these embeddings only from the experts that had trained them, but it worked worse than having every expert start from the same point and averaging as usual. The other reason is long context. Post-training usually weakens long-context ability, and it is hard to win back; starting from an instruct model that already handles long contexts makes that ability easier to preserve.

Run leave-one-out merges. Deliberately comparing merges that leave out one expert during the search reveals which expert is dragging the others down. The authors call this phenomenon a "collision."

Put a little data from other domains into each expert. A small amount of cross-domain data acts like a regularizer, keeping each expert from drifting too far and keeping the experts compatible with one another. Collisions sometimes also come from small differences in the style or format of shared data.

The authors' conclusion is that keeping the experts consistent with one another matters more than chasing the best performance of any single expert.

Merging is cheap, evaluation is expensive

This is a problem you only run into in real products. Academic research usually fixes a set of experts and then looks for the best merging method, targeting a handful of metrics. Command A, however, has to look after hundreds of metrics at once, and every set of weights tried means running the full evaluation suite. The merge itself takes very little compute, but the evaluation takes a lot of inference time.

On top of that, the experts themselves keep changing. The authors say a large part of the work of making merging succeed actually consists of going back and changing how the experts are trained.

Discussion

How much expert capability does merging preserve?

Original Figure 12. The left plot is a histogram, with the score difference between the merged model and the best expert on the horizontal axis (from −10 to 10) and the number of metrics on the vertical axis; most metrics fall between −6 and +1.5, and one metric lies between +5 and +6.5. The right plot is a bar chart showing, by domain, the average percentage change of the merged model relative to the expert scores, with error bars: code about −3%, general about −1.1%, multilingual close to 0, RAG about −0.7%, reasoning about −1.8%, and safety about −2.3% with a very long error bar, from about −9% to +4%.
Original Figure 12, Cohere, from PDF page 35, CC BY 4.0.

The left plot covers every tracked metric, showing the score difference between the merged model and the best expert. The vast majority cluster around 0, leaning slightly to the left. The report says the average drop is only 1.8%, the vast majority of metrics stay within 2.5% of the best expert, and a few even beat the expert.

The right plot breaks this down by domain. Multilingual, RAG, and general capabilities are preserved best; code loses the most, about 3% on average. Safety's error bar is very long, meaning different safety metrics change by very different amounts after merging.

The authors also observe that at the 111B scale, merging is quite stable. Large adjustments to expert weights usually move domain performance by only 2% to 3%, and it is rare for a capability to collapse entirely.

Merging is a coarse operation

A merged model is not guaranteed to sit at a local optimum of the loss landscape. The authors found that running a small number of SFT steps on the merged model, with a learning rate 10 to 20 times lower than in the main SFT, clearly improves performance, with many metrics reaching or even exceeding 100% of the expert's score.

The authors' intuitive explanation is that merging can combine the capabilities, but they may not yet be "aligned" in representation space; a few gradient updates can adjust them into place. This is consistent with the role of the polishing stage, which is also used to recover scores lost in merging.

Polishing itself alternates between two of Cohere's methods: offline SRPO and online CoPG. The authors found that alternating them lets each correct the other; if one stage regresses, the next stage usually pulls it back.

Overall results

On academic benchmarks, Command A trades wins and losses with DeepSeek-V3, GPT-4o, and Llama 3.3 70B. For example, it scores 80.0 on MATH, higher than DeepSeek-V3's 70.2 and GPT-4o's 68.5; it scores 50.8 on GPQA, lower than DeepSeek-V3's 59.1. On the enterprise agent task τ-bench, it scores 51.7, above DeepSeek-V3's 39.1.

In human evaluations, Command A's win rate against GPT-4o (the November 2024 version) and DeepSeek-V3 is roughly 50/50. On code, its win rate is 54.7% against DeepSeek-V3 and 46.8% against GPT-4o. On speed, the authors claim Command A outputs up to 156 tokens per second, 1.75 times GPT-4o and 2.4 times DeepSeek-V3; this is the vendor's own measurement, and the numbers will change under different deployment conditions.

Things to watch for when reading this report

The effect of merging is shown only as distributions and averages. There are no specific scores for each expert or each weight setting, and no control group that skips merging and trains one model on mixed data directly. The weights were found by manual search, and the report does not verify whether the same lessons hold with a different set of experts or at a different scale.

Conclusion

Command A turns model merging from a research technique into a development pipeline for large models. Each team trains one capability to its best, the simplest possible linear average combines them into one model, and when needed the weights are adjusted and the merge rerun, with no retraining.

The most practical part of the report is that handful of lessons: start every expert from the same point, mix a little of the others' data into each expert, find the troublemaking expert with leave-one-out merges, and follow the merge with a few low-learning-rate SFT steps.

Academic papers mostly compete over which merging method is best. Cohere picked the simplest one and spent its effort on making the experts get along. Merging is only the final move; the real work happens before it.