Evaluation Framework for TTD-DR Agents

The framework defines two primary metrics to assess the quality of generated research reports: Helpfulness and Comprehensiveness. Helpfulness is measured according to the ability of the report to satisfy the user’s intent, its ease of understanding (including fluency and coherence), overall accuracy, and the appropriateness of the language used. Comprehensiveness is evaluated on the basis of whether the final report misses any key information. In addition to these metrics, the framework also employs a Correctness metric for multi-hop short-form question-answering tasks. Here, the output is compared directly to a given ground-truth answer by extracting a single response from the generated text. These metrics are critical as they capture both the qualitative and quantitative aspects of the research outputs^[1].

To effectively evaluate long-form outputs and the intricate reasoning paths involved in the research report generation, the framework integrates a side-by-side quality comparison methodology. In this setup, expert human evaluators are provided with two reports, typically one produced by TTD-DR and a baseline system such as OpenAI Deep Research. Evaluators then rate the reports using a 4-point scale. The rating scale is defined as: 'Much Better' when one report is significantly more helpful and comprehensive; 'Better' when one report excels in one metric without lagging in the other; 'Slightly Better' when one report has a small advantage in helpfulness despite being less comprehensive; and 'About The Same' if no substantial differences are observed. This multi-faceted evaluation enables a more nuanced understanding of report quality^[1].

Given the complexity and length of the generated research reports, large language models (LLMs) are used as automated judges to scale the evaluation process. To ensure that the automated evaluations are reliable and align well with human judgments, TTD-DR incorporates a calibration phase. In this phase, approximately 200 reports from TTD-DR agents are compared against those from systems like OpenAI Deep Research. An LLM model, in this case Gemini-1.5-pro, is calibrated using human rating guidelines and instructions. The alignment between the LLM-as-a-judge and human evaluators is quantitatively analyzed, with metrics such as correlation accuracy being reported in additional appendices. This calibration ensures that the self-generated automated feedback mirrors the assessments provided by expert human raters, thereby reinforcing the credibility of the evaluation process^[1].

The evaluation framework leverages a diverse portfolio of benchmark datasets that reflect the varied demands of real-world research. For the generation of comprehensive long-form research reports, datasets like LongForm Research and DeepConsult are used. The LongForm Research dataset comprises 205 real-world queries spanning multiple industry domains, while DeepConsult focuses on business and consulting-related prompts covering topics such as marketing, finance, technology trends, and business planning. To evaluate the capability of TTD-DR in handling shorter, multi-hop, reasoning-intensive queries, the framework also makes use of the Humanity’s Last Exam (HLE) dataset. HLE includes a collection of 2,500 challenging questions, with a curated subset called HLE-search specifically targeting queries that require extensive search and reasoning. In addition, the framework evaluates performance on the GAIA benchmark, which covers real-world questions across three levels of difficulty. Using these varied datasets ensures that the evaluation framework assesses both the depth and breadth of the agent’s research abilities^[1].

The entire evaluation process is integrated with a modular agent system built using the Google Agent Development Kit (ADK). This system enables the orchestration of various workflows, including the planning, search, and report synthesis phases. Each stage in the TTD-DR agent workflow outputs long responses that are subsequently synthesized into a final report. Evaluating these outputs involves verifying long-term logical dependencies and ensuring the consistency of facts through iterative feedback loops. Metrics such as Helpfulness, Comprehensiveness, and Correctness are computed based on both automated LLM feedback and human annotations gathered via a custom-built interface. The evaluation also accounts for latency and efficiency, as demonstrated by the Pareto frontier comparisons with other deep research agents. These comparisons illustrate that TTD-DR, along with its self-evolution and denoising mechanisms, achieves superior performance gains with minimal increases in processing time^[1].

In summary, the evaluation framework for TTD-DR agents is a comprehensive, multi-dimensional system that rigorously tests the ability of deep research agents to produce high-quality, coherent, and comprehensive research reports. By combining clearly defined qualitative metrics with robust side-by-side comparisons, calibrated LLM judgments, and diverse benchmark datasets, the framework offers an in-depth analysis of the agent’s performance in real-world scenarios. This thorough evaluation protocol not only demonstrates the superiority of the TTD-DR system over baseline agents but also provides a transparent and scalable approach for assessing the evolving capabilities of deep research agents^[1].