# Researchers from Apple have unveiled the Massive Multitask Agent Understanding (MMAU) benchmark.

*genai · news · 2024-08-02 · Analytics India Magazine*

## Key points

- Apple introduced the MMAU benchmark to assess LLMs on five capabilities across five specialized domains.
- MMAU includes 20 tasks with over 3,000 prompts, enabling more granular assessment than previous benchmarks.
- GPT-4 and other API-based models consistently outperformed open-source models, especially in challenging domains.
- Self-correction remains a significant challenge, while problem-solving is more universally achievable for current LLMs.
- Apple also introduced LazyLLM, a new technique to speed up LLM inference without sacrificing accuracy.

Researchers from Apple have recently unveiled the Massive Multitask Agent Understanding (MMAU) benchmark, a new evaluation framework designed to assess the capabilities of large language models (LLMs) as intelligent agents across diverse domains and skills. Read the full paper here MMAU evaluates models on five key capabilities: understanding, reasoning, planning, problem-solving, and self-correction. It spans five domains: tool use, directed acyclic graph question answering, data science and machine learning coding, contest-level programming, and mathematics. The benchmark comprises 20 carefully designed tasks with over 3,000 distinct prompts, offering a more granular assessment of LLM capabilities compared to existing benchmarks. MMAU aims to provide insights into where model failures stem from by isolating and testing specific skills. Key findings from evaluating 18 models on MMAU revealed that commercial API-based models like GPT-4 consistently outperformed open-source models across various domains. The models demonstrated varying proficiency levels in different capabilities– problem-solving was more universally achievable, while self-correction posed significant challenges for many models. High-quality planning also boosted performance for all models in mathematical tasks. Interestingly, larger models did not always perform better, underscoring the importance of training strategies and model architectures The researchers emphasise that MMAU is designed to complement, not replace, existing interactive evaluations. They acknowledge limitations in the current scope and call for future work to expand into more domains and refine capability decomposition methods. By providing a comprehensive and granular evaluation framework, MMAU aims to drive progress in developing more capable and well-rounded AI agents. The datasets and evaluation scripts have been made publicly available to facilitate further research in this area. Also, recently, Apple introduced LazyLLM, a novel technique aimed at improving the efficiency of large language model (LLM) inference. This approach seeks to accelerate response generation in transformer-based language models while maintaining accuracy.

**Companies:** Apple
**Countries:** United States

[Read the full story on Analytics India Magazine](https://analyticsindiamag.com/ai-news-updates/apple-unveils-mmau-a-new-benchmark-for-evaluating-language-model-agents-across-diverse-domains/)

---

Canonical: https://newsio.io/n/db989e64-f0e6-4982-abf0-e15a3f2b04cd/researchers-from-apple-have-unveiled-the-massive-multitask-agent-understanding-m
Summarized by Newsio from Analytics India Magazine. https://newsio.io/how-it-works