ART is a new open-source framework for training agents using reinforcement learning (RL). RL allows you to train an agent to perform better at any task whose outcome can be measured and quantified.
There are many excellent projects focused on training LLMs with RL, such as GRPOTrainer (https://huggingface.co/docs/trl/main/en/grpo_trainer) and verl (https://github.com/volcengine/verl). We've used these frameworks extensively for customer-facing projects at OpenPipe, but grew frustrated with some key limitations:
- Multi-turn workflows, where the agent calls a tool, gets a response, and calls another, are not well supported. This makes them a non-starter for any task that requires an agent to perform a sequence of actions.
- Other frameworks typically have low GPU efficiency. They may require multiple H100 GPUs just to train a small 7B parameter model, and aren't able to keep the GPUs busy consistently during both the "rollout" and "training" phases of the training loop.
- Existing frameworks are typically not a convenient shape for integrating with existing agentic codebases. Existing trainers expect you to call raw text completion endpoints, and don't automatically provide industry-standard chat completion APIs.
ART is designed to address these limitations and make it easy to train high-quality agents. We've also shared many details and practical lessons learned is in this post, which walks through a demo of training an email research agent that outperforms o3 (https://openpipe.ai/blog/art-e-mail-agent). You can also find out more about ART's architecture in our announcement post (https://openpipe.ai/blog/art-trainer-a-new-rl-trainer-for-ag...).
Happy to answer any questions you have!
No callbacks or straitjacket flows. Instead we serve an OpenAI API-compatible endpoint that you can use as a drop-in replacement for any proprietary APIs you may be hitting.
After collecting responses from the inference API, you can tune the model with your own custom rewards and repeat the process as long as you like, until performance converges. We believe this level of flexibility will make it easier for you to train state-of-the-art models for your own use cases, much like Kyle's new email agent[1].
Also happy to answer any questions you have about the framework.
- How does training with RL differ from fine tuning?
- When would it make sense to fine tune instead of using RL?
Do you have documentation for the API response from the `/_train_model` endpoint?