Command A: An Enterprise-Ready Large Language Model
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:2504.00698
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{transformerscommanda2025,
title = {{Command A: An Enterprise-Ready Large Language Model}},
author = {Team Cohere and Aakanksha and Arash Ahmadian and Marwan Ahmed and Jay Alammar and Milad Alizadeh and Yazeed Alnumay and Sophia Althammer and Arkady Arkhangorodsky and Viraat Aryabumi and Dennis Aumiller and Raphaël Avalos and Zahara Aviv and Sammie Bae and Saurabh Baji and Alexandre Barbet and Max Bartolo and Björn Bebensee and Neeral Beladia and Walter Beller-Morales and Alexandre Bérard and Andrew Berneshawi and Anna Bialas and Phil Blunsom and Matt Bobkin and Adi Bongale and Sam Braun and Maxime Brunet and Samuel Cahyawijaya and David Cairuz and Jon Ander Campos and Cassie Cao and Kris Cao and Roman Castagné and Julián Cendrero and Leila Chan Currie and Yash Chandak and Diane Chang and Giannis Chatziveroglou and Hongyu Chen and Claire Cheng and Alexis Chevalier and Justin T. Chiu and Eugene Cho and Eugene Choi and Eujeong Choi and Tim Chung and Volkan Cirik and Ana Cismaru and Pierre Clavier and Henry Conklin and Lucas Crawhall-Stein and Devon Crouse and Andres Felipe Cruz-Salinas and Ben Cyrus and Daniel D'souza and Hugo Dalla-Torre and John Dang and William Darling and Omar Darwiche Domingues and Saurabh Dash and Antoine Debugne and Théo Dehaze and Shaan Desai and Joan Devassy and Rishit Dholakia and Kyle Duffy and Ali Edalati and Ace Eldeib and Abdullah Elkady and Sarah Elsharkawy and Irem Ergün and Beyza Ermis and Marzieh Fadaee and Boyu Fan and Lucas Fayoux and Yannis Flet-Berliac and Nick Frosst and Matthias Gallé and Wojciech Galuba and Utsav Garg and Matthieu Geist and Mohammad Gheshlaghi Azar and Ellen Gilsenan-McMahon and Seraphina Goldfarb-Tarrant and Tomas Goldsack and Aidan Gomez and Victor Machado Gonzaga and Nithya Govindarajan and Manoj Govindassamy and Nathan Grinsztajn and Nikolas Gritsch and Patrick Gu and Shangmin Guo and Kilian Haefeli and Rod Hajjar and Tim Hawes and Jingyi He and Sebastian Hofstätter and Sungjin Hong and Sara Hooker and Tom Hosking and Stephanie Howe and Eric Hu and Renjie Huang and Hemant Jain and Ritika Jain and Nick Jakobi and Madeline Jenkins and JJ Jordan and Dhruti Joshi and Jason Jung and Trushant Kalyanpur and Siddhartha Rao Kamalakara and Julia Kedrzycki and Gokce Keskin and Edward Kim and Joon Kim and Wei-Yin Ko and Tom Kocmi and Michael Kozakov and Wojciech Kryściński and Arnav Kumar Jain and Komal Kumar Teru and Sander Land and Michael Lasby and Olivia Lasche and Justin Lee and Patrick Lewis and Jeffrey Li and Jonathan Li and Hangyu Lin and Acyr Locatelli and Kevin Luong and Raymond Ma and Lukáš Mach and Marina Machado and Joanne Magbitang and Brenda Malacara Lopez and Aryan Mann and Kelly Marchisio and Olivia Markham and Alexandre Matton and Alex McKinney and Dominic McLoughlin and Jozef Mokry and Adrien Morisot and Autumn Moulder and Harry Moynehan and Maximilian Mozes and Vivek Muppalla and Lidiya Murakhovska and Hemangani Nagarajan and Alekhya Nandula and Hisham Nasir and Shauna Nehra and Josh Netto-Rosen and Daniel Ohashi and James Owers-Bardsley and Jason Ozuzu and Dennis Padilla and Gloria Park and Sam Passaglia and Jeremy Pekmez and Laura Penstone and Aleksandra Piktus and Case Ploeg and Andrew Poulton and Youran Qi and Shubha Raghvendra and Miguel Ramos and Ekagra Ranjan and Pierre Richemond and Cécile Robert-Michon and Aurélien Rodriguez and Sudip Roy and Sebastian Ruder and Laura Ruis and Louise Rust and Anubhav Sachan and Alejandro Salamanca and Kailash Karthik Saravanakumar and Isha Satyakam and Alice Schoenauer Sebag and Priyanka Sen and Sholeh Sepehri and Preethi Seshadri and Ye Shen and Tom Sherborne and Sylvie Shang Shi and Sanal Shivaprasad and Vladyslav Shmyhlo and Anirudh Shrinivason and Inna Shteinbuk and Amir Shukayev and Mathieu Simard and Ella Snyder and Ava Spataru and Victoria Spooner and Trisha Starostina and Florian Strub and Yixuan Su and Jimin Sun and Dwarak Talupuru and Eugene Tarassov and Elena Tommasone and Jennifer Tracey and Billy Trend and Evren Tumer and Ahmet Üstün and Bharat Venkitesh and David Venuto and Pat Verga and Maxime Voisin and Alex Wang and Donglu Wang and Shijian Wang and Edmond Wen and Naomi White and Jesse Willman and Marysia Winkels and Chen Xia and Jessica Xie and Minjie Xu and Bowen Yang and Tan Yi-Chern and Ivan Zhang and Zhenyu Zhao and Zhoujie Zhao},
year = {2025},
eprint = {2504.00698},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2504.00698}
}
Cohere is a Canadian language-model company focused on enterprise customers; one of its founders, Aidan Gomez, is a coauthor of "Attention Is All You Need." Its models emphasize retrieval-augmented generation (RAG), tool use, and multilinguality. They usually don't compete for first place on chat leaderboards, and instead stress being useful and easy to deploy in real enterprise settings.
In March 2025, Cohere released Command A. It is a 111-billion-parameter model that supports 23 business languages and can be deployed on just two A100s or H100s. The weights are released under a non-commercial license (CC-BY-NC) for research use.
This technical report focuses on the post-training pipeline. The usual approach takes one model through SFT, preference learning, and reinforcement learning in sequence. Command A instead has several teams each train expert models that excel at different capabilities, and in the end combines them into one with model merging. The report says this is the first time model merging has been made the core of post-training at this scale.
Defining the problem
An enterprise language model has to do many things at once: answer questions from documents, call tools, write code, do math, speak many languages, and follow safety rules, all while handling very long contexts.
These capabilities are trained in very different ways. Code and math are easy to verify, so data can be synthesized in bulk and reinforcement learning can use verifiable rewards. Safety and RAG rely mostly on human annotation, with far less data, and are better suited to training with preference comparisons. Mixing all of this data together and training one model with a single set of hyperparameters makes it hard to get every capability to its best.
The same problem shows up on the organizational side. When multiple teams improve different capabilities at the same time, all working on the same model, their experiments interfere with one another, and each team ends up waiting on the others.
Can each capability be trained to its best separately and then cheaply combined? That is the question the whole report tries to answer.
Solving the problem
Architecture and pretraining
First, a quick look at the base. Command A is a decoder-only Transformer with three main design choices.
- Interleaved attention layers: in every 4 layers, 3 use sliding-window attention and 1 uses full attention, a 3:1 ratio. The sliding-window layers use RoPE, while the full-attention layers use no positional encoding at all (NoPE).
- Parallel Transformer blocks: attention and the feedforward network are computed in parallel rather than one after the other. The authors say performance is comparable and throughput improves markedly.
- Others: GQA, SwiGLU, no bias terms, and a single embedding matrix shared by input and output. The vocabulary has 255,000 tokens, so sharing embeddings saves a lot of memory.
Training uses µP to tune hyperparameters on small models and then transfers them directly to the large model. The main matrix multiplications are computed in FP8. The authors observed no instability, but training in FP8 the whole way slightly hurts downstream performance, so they train in BF16 for a while before switching to FP8. Pretraining ends with a cooldown phase in which the learning rate drops from to over 50,000 steps; the context length first stays at 8K, then is gradually extended to 32K, 128K, and 256K.
Post-training: two merges

Read this figure from left to right.
- Instruct model: first run one round of SFT on the pretrained model to get an instruct model with basic capabilities.
- SFT experts: starting from the same instruct model, train several experts in parallel, each with its own data mix and hyperparameters. The main text lists six: code, safety, RAG, math, multilingual, and general plus long context.
- SFT Soup: merge these experts into a single model.
- RL experts: starting from the SFT Soup, train another set of reinforcement-learning experts in parallel, each domain with a method that suits it: verifiable rewards for math and code, preference comparisons for safety and multilingual.
- RL Soup: merge once more.
- Polishing: finally, run a round of global alignment to fix problems left by merging and bring the overall style closer to human preferences.
The two "Model Merging" columns in the figure each show a row of sliders, meaning each expert's weight in the merge can be adjusted. This is the key to the whole pipeline. If some capability falls short after merging, there is no need to retrain; just adjust the sliders and merge again.
Four uses of merging
The report sorts model merging into four uses, and Command A employs all four:
- Expert merging: combining models that excel at different capabilities into one. This is the main act.
- Polyak averaging: averaging several checkpoints from the same training run to reduce stochastic-gradient noise.
- Seed merging: averaging models trained with different random seeds to reduce the effects of chance.
- Interpolation to recover capabilities: experts trained only on short contexts lose long-context ability. Interpolating an expert with the original base model recovers most of the long-context ability while keeping the expert's new capabilities. The authors point out that this is very close to Google DeepMind's WARP.
Linear merging is enough
Expert merging uses a linear average:
The authors tried SLERP and task arithmetic and found no clear improvement, only added complexity. Linear merging also has a practical property: it is associative, so a merge of merges can still be written as a single merge, which makes the whole pipeline easier to trace.
The weights are searched by hand. Raising an expert's weight roughly improves performance in that domain, an intuitive relationship. But because all weights must sum to 1, the other domains' weights go down accordingly, and how much those domains will drop is hard to predict. The authors' search combines the two: turn up the expert for whichever domain falls short, then, centered on the current candidate, apply small perturbations to each expert's weight and search by brute force.
The whole process tracked 500 evaluation metrics. The authors say that looking after this many metrics at once would have been much harder with traditional centralized development.
Consistency matters more than peak scores
Whether a merge succeeds depends largely on whether the experts are "compatible" with one another. The authors sum up several lessons:
All experts start from the same instruct model. One reason is special tokens. Some experts use special tokens, such as those for tool calls, that other experts never use, so in those other experts the embeddings for these tokens are never trained. The authors tried taking these embeddings only from the experts that had trained them, but it worked worse than having every expert start from the same point and averaging as usual. The other reason is long context. Post-training usually weakens long-context ability, and it is hard to win back; starting from an instruct model that already handles long contexts makes that ability easier to preserve.
Run leave-one-out merges. Deliberately comparing merges that leave out one expert during the search reveals which expert is dragging the others down. The authors call this phenomenon a "collision."
Put a little data from other domains into each expert. A small amount of cross-domain data acts like a regularizer, keeping each expert from drifting too far and keeping the experts compatible with one another. Collisions sometimes also come from small differences in the style or format of shared data.
The authors' conclusion is that keeping the experts consistent with one another matters more than chasing the best performance of any single expert.
Merging is cheap, evaluation is expensive
This is a problem you only run into in real products. Academic research usually fixes a set of experts and then looks for the best merging method, targeting a handful of metrics. Command A, however, has to look after hundreds of metrics at once, and every set of weights tried means running the full evaluation suite. The merge itself takes very little compute, but the evaluation takes a lot of inference time.
On top of that, the experts themselves keep changing. The authors say a large part of the work of making merging succeed actually consists of going back and changing how the experts are trained.
Discussion
How much expert capability does merging preserve?

The left plot covers every tracked metric, showing the score difference between the merged model and the best expert. The vast majority cluster around 0, leaning slightly to the left. The report says the average drop is only 1.8%, the vast majority of metrics stay within 2.5% of the best expert, and a few even beat the expert.
The right plot breaks this down by domain. Multilingual, RAG, and general capabilities are preserved best; code loses the most, about 3% on average. Safety's error bar is very long, meaning different safety metrics change by very different amounts after merging.
The authors also observe that at the 111B scale, merging is quite stable. Large adjustments to expert weights usually move domain performance by only 2% to 3%, and it is rare for a capability to collapse entirely.
Merging is a coarse operation
A merged model is not guaranteed to sit at a local optimum of the loss landscape. The authors found that running a small number of SFT steps on the merged model, with a learning rate 10 to 20 times lower than in the main SFT, clearly improves performance, with many metrics reaching or even exceeding 100% of the expert's score.
The authors' intuitive explanation is that merging can combine the capabilities, but they may not yet be "aligned" in representation space; a few gradient updates can adjust them into place. This is consistent with the role of the polishing stage, which is also used to recover scores lost in merging.
Polishing itself alternates between two of Cohere's methods: offline SRPO and online CoPG. The authors found that alternating them lets each correct the other; if one stage regresses, the next stage usually pulls it back.
Overall results
On academic benchmarks, Command A trades wins and losses with DeepSeek-V3, GPT-4o, and Llama 3.3 70B. For example, it scores 80.0 on MATH, higher than DeepSeek-V3's 70.2 and GPT-4o's 68.5; it scores 50.8 on GPQA, lower than DeepSeek-V3's 59.1. On the enterprise agent task τ-bench, it scores 51.7, above DeepSeek-V3's 39.1.
In human evaluations, Command A's win rate against GPT-4o (the November 2024 version) and DeepSeek-V3 is roughly 50/50. On code, its win rate is 54.7% against DeepSeek-V3 and 46.8% against GPT-4o. On speed, the authors claim Command A outputs up to 156 tokens per second, 1.75 times GPT-4o and 2.4 times DeepSeek-V3; this is the vendor's own measurement, and the numbers will change under different deployment conditions.
Things to watch for when reading this report
The effect of merging is shown only as distributions and averages. There are no specific scores for each expert or each weight setting, and no control group that skips merging and trains one model on mixed data directly. The weights were found by manual search, and the report does not verify whether the same lessons hold with a different set of experts or at a different scale.
Conclusion
Command A turns model merging from a research technique into a development pipeline for large models. Each team trains one capability to its best, the simplest possible linear average combines them into one model, and when needed the weights are adjusted and the merge rerun, with no retraining.
The most practical part of the report is that handful of lessons: start every expert from the same point, mix a little of the others' data into each expert, find the troublemaking expert with leave-one-out merges, and follow the merge with a few low-learning-rate SFT steps.
Academic papers mostly compete over which merging method is best. Cohere picked the simplest one and spent its effort on making the experts get along. Merging is only the final move; the real work happens before it.