Google is committed to developing Gemini responsibly, innovating on safety and security alongside capabilities. This commitment includes training and evaluating models, focusing on automated red teaming, undergoing held-out assurance evaluations on present-day risks, and evaluating the potential for dangerous capabilities to proactively anticipate new and long-term risks.

The Gemini safety policies align with Google’s standard framework, preventing the generation of specific types of harmful content. These policies cover several key areas:

Child sexual abuse and exploitation
Hate speech (e.g., dehumanizing members of protected groups)
Dangerous content (e.g., promoting suicide or instructing in activities that could cause real-world harm)
Harassment (e.g., encouraging violence against people)
Sexually explicit content
Medical advice that runs contrary to scientific or medical consensus

These policies apply across modalities, aiming to minimize harmful outputs irrespective of input type. From a security standpoint, Gemini strives to protect users from cyberattacks, for example, by being robust to prompt injection attacks.

Defining what the model should do is equally important as defining what it should not do. The desiderata, also known as "helpfulness", include:

Helping the user: Fulfilling the user's request and only refusing if it is impossible to find a policy-compliant response.
Assuming good intent: Articulating refusals respectfully without making assumptions about user intent.

Safety is integrated into the models through pre- and post-training approaches. This process starts by constructing metrics based on policies and desiderata, typically turned into automated evaluations that guide model development through successive iterations. Data filtering and conditional pre-training, as well as Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human and Critic Feedback (RL*F), are employed. Dataset filtering applies safety measures to pre-training data for the strictest policies. Pre-training monitoring includes a novel evaluation to capture the model’s ability to be steered towards different viewpoints and values, which helps align the model at post-training time.

For the SFT stage, adversarial prompts are sourced either leveraging existing models and tools to probe Gemini’s attack surface or relying on human interactions to discover potentially harmful behavior. Throughout this process, coverage of the safety policies across common model use cases is strived for. When model behavior needs improvement due to either safety policy violations, or because the model refuses when a helpful, non-policy-violating answer exists, a combination of custom data generation recipes loosely inspired by Constitutional AI, as well as human intervention to revise responses, are used. Automated evaluations on both safety and non-safety metrics are leveraged to monitor impact and potential unintended regressions.

Reward signals during RL come from a combination of a Data Reward Model (DRM), which amortizes human preference data, and a Critic, a prompted model that grades responses according to pre-defined rubrics. Interventions are divided into Reward Model and Critic improvements (RM), and reinforcement learning (RL) improvements. Prompts are sourced through human-model or model-model interactions, striving for coverage of safety policies and use cases. Both DRM training, given a prompt set, use custom data generation recipes to surface a representative sample of model responses. Humans then provide feedback on responses, often comparing multiple potential response candidates for each query, and this preference data is amortized in our Data Reward Model. Critics, on the other hand, do not require additional data, and iteration on the grading rubric can be done offline. Similarly to SFT, RLF steers the model away from undesirable behavior, both in terms of content policy violations, and trains the model to be helpful. RLF is accompanied by a number of evaluations that run continuously during training to monitor for safety and other metrics.

To complement human red teaming and static evaluations, extensive use of automated red teaming (ART) is made to dynamically evaluate Gemini at scale. This allows to significantly increase coverage and understanding of potential risks, as well as rapidly develop model improvements to make Gemini safer and more helpful.

ART is formulated as a multi-agent game between populations of attackers and the target Gemini model being evaluated. Attackers aim to elicit responses from the target model which satisfy some defined objectives (e.g. if the response violates a safety policy, or is unhelpful), and these interactions are scored by various judges (e.g. using a set of policies), with the resulting scores used by the attackers as a reward signal to optimize their attacks. Attackers evaluate Gemini in a black-box setting, using natural language queries without access to the model’s internal parameters, ensuring the automated red teaming is more reflective of real-world use cases and challenges. Attackers are prompted Gemini models, while judges are a mixture of prompted and finetuned Gemini models. This approach has allowed to rapidly scale red teaming to a growing number of areas including policy violations, tone, helpfulness, and neutrality.

Gemini's susceptibility to indirect prompt injection attacks is evaluated, focusing on scenarios in which a third party hides malicious instructions in external retrieved data to manipulate Gemini into taking unauthorized actions through function calling. Function calls available to Gemini allow it to summarize a user’s latest emails and send emails on their behalf. The attacker's objective is to manipulate the model to invoke a send email function call that discreetly exfiltrates sensitive information from conversation history. Several attacks automate the process of generating malicious prompts, including Actor Critic, Beam Search, and Tree of Attacks w/ Pruning (TAP), and after constructing prompt injections using these methods, they're evaluated on a held-out set of synthetic conversation histories containing simulated private user information.

Google DeepMind released its Frontier Safety Framework (FSF) in May 2024 and updated it in February 2025. The FSF comprises a number of processes and evaluations that address risks of severe harm stemming from powerful capabilities of frontier models. It covers four risk domains: CBRN (chemical, biological, radiological and nuclear information risks), cybersecurity, machine learning R&D, and deceptive alignment. The FSF involves the regular evaluation of Google’s frontier models to determine whether they require heightened mitigations, comparing test results against internal alert thresholds (“early warnings”) which are set significantly below the actual Critical Capability Levels (CCLs).

As part of the External Safety Testing Program, independent external groups help identify areas for improvement in model safety by undertaking structured evaluations, qualitative probing, and unstructured red teaming. Testing is carried out on the most capable Gemini models with the largest capability jumps and external testing groups are given black-box testing access to Gemini on AI Studio for a number of weeks, enabling Google DeepMind to gather early insights into the model’s capabilities and understand if and where mitigations were needed. These groups are selected based on their expertise across a range of domain areas, such as autonomous systems, societal, cyber, and CBRN risks, and are by design instructed to develop their own methodology to test topics within a particular domain area, remaining independent from internal Google DeepMind evaluations.

Beyond One Dataset: The Added Value of Linked Health Data

Transcript

Related Content From The Pandipedia