Microsoft’s Orca-2: A Leap Forward for Smaller AI Models

orca illustration by the idealistic world team

Microsoft recently unveiled Orca-2, an AI model that demonstrates significantly improved reasoning abilities despite having under 15 billion parameters. This development represents a major step towards more capable and controllable AI systems.

What is Orca-2?

Orca-2 is the successor to Microsoft’s original Orca model — a 13-billion parameter system focused on reasoning tasks2. The new Orca-2 matches or exceeds the performance of AI models over 5–10 times its size on certain benchmarks, while using less computing power. It builds on learnings from the LLAMA project to train medium-sized models to reason in more focused, step-by-step ways previously seen only in gigantic systems like GPT-3.

The Significance of This Breakthrough

Orca-2’s reasoning skills relative to its smaller size could enable more efficient AI applications. Its training process also shows that with the right methodology, smaller models can reach levels of sophistication many times their parameter count5. This demonstrates an alternative to the prevailing approach of simply building ever-larger AI models, which is compute-intensive. Instead, tailored training processes can better optimize smaller models for reasoning tasks.

How Orca-2 Was Developed

The Orca-2 models were created by further training the 7 billion and 13 billion parameter LLAMA 2 models on high-quality synthetic reasoning data. This data was generated to teach different step-by-step reasoning techniques for diverse tasks. Microsoft also developed a system for automatically determining which reasoning approach works best for a given query.

This allows Orca-2 to dynamically select strategies such as multi-step inferences or extracted evidence generation.

Future Outlook

The innovations behind Orca-2 — from efficient reasoning architectures to targeted training — could inform the development of other performant but well-controlled AI systems9. Microsoft suggests this technique could even be applied to models besides LLAMA10. Overall, Orca-2 represents promising progress in making advanced reasoning capabilities practical at smaller scales of AI. The project shows that with sufficient insight, less can indeed be more when it comes to model parameters and intelligence.

Sources:

1. https://www.microsoft.com/en-us/research/blog/orcax-microsofts-latest-leap-forward-in-reasoning-technology/**
2. https://www.microsoft.com/en-us/research/project/orca/**
3. https://www.infoq.com/news/2023/02/microsoft-orca2-reasoning/**
4. https://syncedreview.com/2023/02/21/microsoft-debuts-orca-2-a-more-capable-and-controllable-llama-based-ai-model/**
5. https://venturebeat.com/ai/microsofts-orca-2-ai-reasons-like-models-10-times-bigger/**
6. https://www.theregister.com/2023/02/21/microsoft_orca_ai/**
7. https://www.microsoft.com/en-us/research/uploads/prod/2023/02/OrcaX_Preprint.pdf**
8. https://www.technologyreview.com/2023/02/21/1070251/ai-microsoft-orca2/**
9. https://analyticsindiamag.com/microsoft-unrolls-orca-2-** displays-reasoning-abilities-like-models-10x-bigger/
10. https://www.zdnet.com/article/microsoft-open-sources-orca-2-weights-and-training-code-to-advance-natural-language-processing/**