{"title": "Gemini 2.X Model Family: Advanced AI Capabilities and Safety Measures", "report_sections": [{"header": "Introduction to the Gemini 2.X Model Family", "content": "

The Gemini 2.X model family includes Gemini 2.5 Pro and Gemini 2.5 Flash, along with earlier Gemini 2.0 Flash and Flash-Lite models. Gemini 2.5 Pro is described as the most capable model to date, achieving state-of-the-art (SoTA) performance on coding and reasoning benchmarks. These models represent the next generation of AI models, designed to power a new era of agentic systems. The Gemini 2.X series are built to be natively multimodal, supporting long context inputs of >1 million tokens, and have native tool use support. This allows them to comprehend vast datasets and handle complex problems from different information sources, including text, audio, images, video, and even entire code repositories. Different models in the series have different strengths and capabilities. Gemini 2.5 Flash provides excellent reasoning abilities at a fraction of the compute and latency requirements and Gemini 2.0 Flash and Flash-Lite provide high performance at low latency and cost. The Gemini 2.X model generation spans the full Pareto frontier of model capability vs cost, allowing users to explore the boundaries of what is possible with complex agentic problem solving.

", "media": []}, {"header": "Key Capabilities of Gemini 2.5 Pro", "content": "

Gemini 2.5 Pro excels at coding tasks, demonstrating a marked improvement over previous models. Performance on LiveCodeBench increased from 30.5% for Gemini 1.5 Pro to 69.0% for Gemini 2.5 Pro, while that for Aider Polyglot went from 16.9% to 82.2%. Performance on SWE-bench Verified went from 34.2% to 67.2%. The model obtained an increase of over 500 Elo over Gemini 1.5 Pro on the LMArena WebDev Arena, resulting in meaningful enhancements in practical applications, including UI and web application development, and the creation of sophisticated agentic workflows. In addition to coding performance, Gemini 2.5 models are noticeably better at math and reasoning tasks than Gemini 1.5 models. Performance on AIME 2025 is 88.0% for Gemini 2.5 Pro compared to 17.5% for Gemini 1.5 Pro, while performance on GPQA (diamond) went from 58.1% for Gemini 1.5 Pro to 86.4%. Image understanding has also increased significantly.

", "media": [{"caption": "Figure 4 | Performance of Gemini 2.X models at coding, math and reasoning tasks in comparison to previous Gemini models. SWE-bench verified numbers correspond to the \u2019multiple attempts\u2019 setting reported in Table 3 .", "url": "https://askpandipro.s3.eu-west-1.amazonaws.com/users/1/documents/414/figures/3.png", "source_url": null, "title": null, "s3_key": "users/1/documents/414/figures/3.png", "type": "image", "id": 210278, "html": null, "carousel_h5": "Figure 4 | Performance of Gemini 2.X models at coding, math and reasoning tasks in comparison to previous Gemini models. SWE-bench verified numbers correspond to the \u2019multiple attempts\u2019 setting reported in Table 3 .", "alt": null}, {"caption": "Figure 11 | Results on the Research Engineer Benchmark (RE-Bench), in which the model must complete simple ML research tasks. Following the original work, scores are normalised against a good quality human-written solution: if a model achieves a score \ud835\udc66 on a challenge, the normalised score is ( \ud835\udc66 \u2212 \ud835\udc66\ud835\udc60 \ud835\udc66\ud835\udc60 )/( \ud835\udc66\ud835\udc5f \ud835\udc66\ud835\udc5f \u2212 \ud835\udc66\ud835\udc60), where \ud835\udc66\ud835\udc60 \ud835\udc66\ud835\udc60 is the \u2019starting score\u2019 of a valid but poor solution provided to the model as an example, and \ud835\udc66\ud835\udc5f \ud835\udc66\ud835\udc5f is the score achieved by a reference solution created by the author of the challenge. Figures for Claude 3.5 Sonnet and expert human performance are sourced from the original work. The number of runs and the time limit for each run are constrained by a total time budget of 32 hours, and error bars indicate bootstrapped 95% confidence intervals; see main text for details. Gemini 2.5 Pro is moderately strong at these challenges, achieving a significant fraction of expert human performance\u2014and in two cases surpassing it.", "url": "https://askpandipro.s3.eu-west-1.amazonaws.com/users/1/documents/414/figures/7.png", "source_url": null, "title": null, "s3_key": "users/1/documents/414/figures/7.png", "type": "image", "id": 210281, "html": null, "carousel_h5": "Figure 11 | Results on the Research Engineer Benchmark (RE-Bench), in which the model must complete simple ML research tasks. Following the original work, scores are normalised against a good quality human-written solution: if a model achieves a score \ud835\udc66 on a challenge, the normalised score is ( \ud835\udc66 \u2212 \ud835\udc66\ud835\udc60 \ud835\udc66\ud835\udc60 )/( \ud835\udc66\ud835\udc5f \ud835\udc66\ud835\udc5f \u2212 \ud835\udc66\ud835\udc60), where \ud835\udc66\ud835\udc60 \ud835\udc66\ud835\udc60 is the \u2019starting score\u2019 of a valid but poor solution provided to the model as an example, and \ud835\udc66\ud835\udc5f \ud835\udc66\ud835\udc5f is the score achieved by a reference solution created by the author of the challenge. Figures for Claude 3.5 Sonnet and expert human performance are sourced from the original work. The number of runs and the time limit for each run are constrained by a total time budget of 32 hours, and error bars indicate bootstrapped 95% confidence intervals; see main text for details. Gemini 2.5 Pro is moderately strong at these challenges, achieving a significant fraction of expert human performance\u2014and in two cases surpassing it.", "alt": null}]}, {"header": "Model Architecture and Training", "content": "

The Gemini 2.5 models are sparse mixture-of-experts (MoE) transformers with native multimodal support for text, vision, and audio inputs. Developments to the model architecture contribute to the significantly improved performance of Gemini 2.5 compared to Gemini 1.5 Pro. The Gemini 2.5 model series makes considerable progress in enhancing large-scale training stability, signal propagation and optimization dynamics, resulting in a considerable boost in performance straight out of pre-training compared to previous Gemini models. The pre-training dataset is a large-scale, diverse collection of data encompassing a wide range of domains and modalities, which includes publicly available web-documents, code (various programming languages), images, audio (including speech and other audio types) and video, with a cutoff date as June 2024 for 2.0 and January 2025 for 2.5. The model family is the first to be trained on TPUv5p architecture, employing synchronous data-parallel training to parallelise over multiple 8960-chip pods of Google\u2019s TPUv5p accelerators, distributed across multiple datacenters.

", "media": [{"caption": "Figure 1 | Cost-performance plot. Gemini 2.5 Pro is a marked improvement over Gemini 1.5 Pro, and has an LMArena score that is over 120 points higher than Gemini 1.5 Pro. Cost is a weighted average of input and output tokens pricing per million tokens. Source: LMArena, imported on 2025-06-16.", "url": "https://askpandipro.s3.eu-west-1.amazonaws.com/users/1/documents/414/figures/0.png", "source_url": null, "title": null, "s3_key": "users/1/documents/414/figures/0.png", "type": "image", "id": 210275, "html": null, "carousel_h5": "Figure 1 | Cost-performance plot. Gemini 2.5 Pro is a marked improvement over Gemini 1.5 Pro, and has an LMArena score that is over 120 points higher than Gemini 1.5 Pro. Cost is a weighted average of input and output tokens pricing per million tokens. Source: LMArena, imported on 2025-06-16.", "alt": null}]}, {"header": "Post-Training Methodologies and Thinking", "content": "

Significant advancements have been made in post-training methodologies, driven by a consistent focus on data quality across the Supervised Fine-Tuning (SFT), Reward Modeling (RM), and Reinforcement Learning (RL) stages. A key focus has been leveraging the model itself to assist in these processes, enabling more efficient and nuanced quality control. Furthermore, the training compute allocated to RL has been increased, allowing deeper exploration and refinement of model behaviors. This has been coupled with a focus on verifiable rewards and model-based generative rewards to provide more sophisticated and scalable feedback signals. Past Gemini models produce an answer immediately following a user query, which constrains the amount of inference-time compute (Thinking) that our models can spend reasoning over a problem. Gemini Thinking models are trained with Reinforcement Learning to use additional compute at inference time to arrive at more accurate answers. The resulting models are able to spend tens of thousands of forward passes during a \u201cthinking\u201d stage, before responding to a question or query.

", "media": [{"caption": "Gemini 2.5 Pro Agent Architecture diagram.", "url": "https://askpandipro.s3.eu-west-1.amazonaws.com/users/1/documents/414/figures/10.png", "source_url": null, "title": null, "s3_key": "users/1/documents/414/figures/10.png", "type": "image", "id": 210285, "html": null, "carousel_h5": "Gemini 2.5 Pro Agent Architecture diagram.", "alt": null}]}, {"header": "Specific Capability Improvements", "content": "

Gemini 2.0 and 2.5 represent a strategic shift of development priorities towards delivering tangible real-world value, empowering users to address practical challenges and achieve development objectives within today\u2019s complex, multimodal software environments. In pre-training, focus was intensified on incorporating a greater volume and diversity of code data from both repository and web sources into the training mixture. With Gemini 2.0 and 2.5, capabilities have been expanded to address multimodal inputs, long-context reasoning, and model-retrieved information. At the same time, the landscape and user expectations for factuality have evolved dramatically, shaped in part by Google\u2019s deployment of AI Overviews and AI Mode. Modeling and data advances helped improve the quality of million-length context, and internal evaluations were reworked to be more challenging to help steer modeling research. Gemini\u2019s multilingual capabilities have also undergone a profound evolution since 1.5, stemming from a holistic strategy, meticulously refining pre- and post-training data quality, advancing tokenization techniques, innovating core modeling, and executing targeted capability hillclimbing.

", "media": [{"caption": "Figure 7 | (Left) Total memorization rates for both exact and approximate memorization. Gemini 2.X model family memorize significantly less than all prior models. (Right) Personal information memorization rates. We observed no instances of personal information being included in outputs classified as memorization for Gemini 2.X, and no instances of high-severity personal data in outputs classified as memorization in prior Gemini models.", "url": "https://askpandipro.s3.eu-west-1.amazonaws.com/users/1/documents/414/figures/5.png", "source_url": null, "title": null, "s3_key": "users/1/documents/414/figures/5.png", "type": "image", "id": 210280, "html": null, "carousel_h5": "Figure 7 | (Left) Total memorization rates for both exact and approximate memorization. Gemini 2.X model family memorize significantly less than all prior models. (Right) Personal information memorization rates. We observed no instances of personal information being included in outputs classified as memorization for Gemini 2.X, and no instances of high-severity personal data in outputs classified as memorization in prior Gemini models.", "alt": null}]}, {"header": "Gemini as an Agent: Deep Research", "content": "

Gemini Deep Research is an agent built on top of the Gemini 2.5 Pro model designed to strategically browse the web and provide informed answers to even the most niche user queries. The agent is optimized to perform task prioritization, and is also able to identify when it reaches a dead-end when browsing. Performance of Gemini Deep Research on the Humanity\u2019s Last Exam benchmark has gone from 7.95% in December 2024 to the SoTA score of 26.9% and 32.4% with higher compute (June 2025).

", "media": [{"caption": "Gemini 2.5 Pro Pok\u00e9mon Progress Timeline graph.", "url": "https://askpandipro.s3.eu-west-1.amazonaws.com/users/1/documents/414/figures/4.png", "source_url": null, "title": null, "s3_key": "users/1/documents/414/figures/4.png", "type": "image", "id": 210279, "html": null, "carousel_h5": "Gemini 2.5 Pro Pok\u00e9mon Progress Timeline graph.", "alt": null}]}, {"header": "Safety, Security, and Responsibility", "content": "

Google is committed to developing Gemini responsibly, innovating on safety and security alongside capabilities. The Gemini safety policies align with Google\u2019s standard framework, which prevents the generative AI models from generating specific types of harmful content, including child sexual abuse and exploitation, hate speech, dangerous content, harassment, sexually explicit content, and medical advice that runs contrary to scientific or medical consensus. Gemini strives to protect users from cyberattacks, for example, by being robust to prompt injection attacks. Compared to Gemini 1.5 models, the 2.0 models are substantially safer. New models are more willing to engage with prompts where previous models may have over-refused, and this nuance can impact automated safety scores.

", "media": []}, {"header": "Automated Red Teaming and Security Measures", "content": "

Automated red teaming (ART) is used to dynamically evaluate Gemini at scale, allowing for significantly increased coverage and understanding of potential risks. Formulated as a multi-agent game between populations of attackers and the target Gemini model, the goal of the attackers is to elicit responses from the target model which satisfy some defined objectives. The generality of this approach has allowed rapid scaling of red teaming to a growing number of areas, including policy violations, tone, helpfulness, and neutrality. There is also an evaluation that measures Gemini\u2019s susceptibility to indirect prompt injection attacks where a third party hides malicious instructions in external retrieved data in order to manipulate Gemini into taking unauthorized actions through function calling.

", "media": [{"caption": "Figure 9 | Results on autonomous cyber offense suite. These benchmarks are based on \u2019capture-theflag\u2019 (CTF) challenges, in which the agent must hack into a simulated server to retrieve a piece of hidden information. Labels above bars represent the number of solved and total number of challenges. A challenge is considered solved if the agent succeeds in at least one out of N attempts, where we vary N between 5 and 30 depending on challenge complexity. Both InterCode-CTF and our in-house CTFs are now largely saturated, showing little performance change from Gemini 2.0 to Gemini 2.5 models. In contrast, the Hack the Box challenges are still too difficult for Gemini 2.5 models, and so also give little signal on capability change.", "url": "https://askpandipro.s3.eu-west-1.amazonaws.com/users/1/documents/414/figures/8.png", "source_url": null, "title": null, "s3_key": "users/1/documents/414/figures/8.png", "type": "image", "id": 210284, "html": null, "carousel_h5": "Figure 9 | Results on autonomous cyber offense suite. These benchmarks are based on \u2019capture-theflag\u2019 (CTF) challenges, in which the agent must hack into a simulated server to retrieve a piece of hidden information. Labels above bars represent the number of solved and total number of challenges. A challenge is considered solved if the agent succeeds in at least one out of N attempts, where we vary N between 5 and 30 depending on challenge complexity. Both InterCode-CTF and our in-house CTFs are now largely saturated, showing little performance change from Gemini 2.0 to Gemini 2.5 models. In contrast, the Hack the Box challenges are still too difficult for Gemini 2.5 models, and so also give little signal on capability change.", "alt": null}]}, {"header": "Memorization and Privacy Considerations", "content": "

The Gemini 2.X model family memorizes long-form text at a much lower rate than prior models. Moreover, a larger proportion of text is characterized as approximately memorized by the Gemini 2.0 Flash-Lite and Gemini 2.5 Flash models in particular, which is a less severe form of memorization. We observed no personal information in the outputs characterized as memorization for Gemini 2.X model family models; this indicates a low rate of personal data in outputs classified as memorization that are below our detection thresholds.

Gemini 2.5 Pro was evaluated against the Critical Capability Levels defined in the Frontier Safety Framework, which examines risk in CBRN, cybersecurity, machine learning R&D, and deceptive alignment. Based on these results, it was found that Gemini 2.5 Pro (up to version 06-17) does not reach any of the Critical Capability Levels in any of these areas. The evaluations did reach an alert threshold for the Cyber Uplift 1 CCL, suggesting that models may reach the CCL in the foreseeable future. Consistent with the FSF, a response plan is being put in place which includes testing models\u2019 cyber capabilities more frequently and accelerating mitigations for them.

", "media": []}, {"header": "The Challenge of Evaluation Benchmarks", "content": "

The staggering performance improvement attained over the space of just one year points to a new challenge in AI research: namely that the development of novel and sufficiently challenging evaluation benchmarks has struggled to keep pace with model capability improvements, especially with the advent of capable reasoning agents. Being able to scale evaluations in both their capability coverage and their difficulty, while also representing tasks that have economic value, will be the key to unlocking the next generation of AI systems.

", "media": []}]}

Fast facts: Research agent performance metrics

Related Content You May Like