{"title": "UI-TARS Architecture and Components", "report_sections": [{"header": "Overview of UI-TARS Architecture", "content": "

UI-TARS is a native GUI agent model designed to operate without manual rules or cascaded modules, perceiving screenshots, applying reasoning, and generating actions autonomously. UI-TARS learns from prior experiences, refining its performance using environment feedback. It serves as a model for GUI interaction, with a structure geared towards streamlining operations.

", "media": [{"caption": "New LLMs That Control UIs: Meet UI-TARS by ByteDance", "url": "https://miro.medium.com/v2/resize:fit:1200/1*n4qGFRCOVBINK9e_VmFkEw.jpeg", "source_url": "https://medium.com/@ezzekielnjuguna.en/new-llms-that-control-uis-meet-ui-tars-by-bytedance-182f11fce3ed", "title": "New LLMs That Control UIs: Meet UI-TARS by ByteDance", "s3_key": null, "type": "image", "id": 202276, "html": null, "alt": null}]}, {"header": "Sequential Process and Core Capabilities", "content": "

The architecture of UI-TARS involves a sequential process that includes observations and actions to accomplish tasks. This process can be expressed as (instruction ,(o1, a1),(o2, a2),\u00b7\u00b7\u00b7,(on, an)), where 'o' denotes the observation (device screenshot) at time step i, and 'a' represents the action executed by the agent. At each time step, UI-TARS takes as input the task instruction, the history of prior interactions (o1, a1,\u00b7\u00b7\u00b7, oi\u22121, ai\u22121), and the current observation oi. Based on this input, the model outputs an action ai from the predefined action space.

UI-TARS incorporates core capabilities including perception, action, reasoning (System 1 & 2 thinking), and memory. This design supports effective interaction with graphical user interfaces.

", "media": [{"caption": "Top AI Web Browsing Agents", "url": "https://meetcody.ai/wp-content/uploads/2024/07/344e3bff-4cb8-4a95-b416-e0dadbd9c72d.png", "source_url": "https://meetcody.ai/blog/top-ai-web-browsing-agents/", "title": "Top AI Web Browsing Agents", "s3_key": null, "type": "image", "id": 202279, "html": null, "alt": null}]}, {"header": "Enhancing Reasoning with 'Thoughts'", "content": "

To enhance the agent\u2019s reasoning and enable more deliberate decision-making, UI-TARS integrates a reasoning component in the form of 'thoughts' generated before each action, reflecting the reflective nature of 'System 2' thinking. These thoughts act as an intermediary step, guiding the agent to reconsider previous actions and observations before moving forward. Therefore, providing enhanced decision-making in ambiguous situations. The process can then be formalized as (instruction ,(o1, t1, a1),(o2, t2, a2),\u00b7\u00b7\u00b7,(on, tn, an)), where 't' represents the reasoning thought.

", "media": []}, {"header": "Key Architectural Components", "content": "

The core contributions of UI-TARS are:

Enhanced Perception for GUI Screenshots: UI-TARS uses a large-scale dataset of GUI screenshots for context-aware understanding of UI elements and precise captioning.
Unified Action Modeling: Actions are standardized into a unified space across platforms, achieving precise grounding and interaction through large-scale action traces.
System-2 Reasoning: UI-TARS incorporates deliberate reasoning into multi-step decision making, involving multiple reasoning patterns such as task decomposition, reflection thinking, and milestone recognition. This design makes decision making more logical and well-thoughtout in its execution.
Iterative Training with Reflective Online Traces: UI-TARS addresses the data bottleneck by automatically collecting, filtering, and reflectively refining new interaction traces on virtual machines, continuously learning from mistakes and adapting to unforeseen situations with minimal human intervention.

", "media": []}, {"header": "GUI Perception", "content": "

GUI perception involves interpreting graphical user interfaces in real-time, adapting to changes as the interface evolves. UI-TARS uses structured text from HTML, visual screenshots, and semantic outlines to achieve a holistic understanding. It is trained to identify elements and generate descriptions, enabling the model to understand the interface layout. It is important to have the ability to interact with the current environment in real-time. GUIs are dynamic, so the agent model must be able to adapt to the current environment.

", "media": []}, {"header": "Action Mechanisms", "content": "

Effective action mechanisms within UI-TARS must be versatile, precise, and adaptable to various GUI contexts.

UI-TARS uses a unified action space to standardize actions across platforms such as 'click', 'type', 'scroll', and 'drag'. Actions can be categorized into atomic actions to execute single operations, and compositional actions to sequence multiple atomic actions. UI-TARS predicts the coordinates of elements it needs to interact with, normalizing coordinates to maintain consistency across different devices.

", "media": [{"caption": "How AI Agents Like Amazon\u2019s Nova Are Changing the Way We Browse the Web", "url": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fps0xfjox45jp9yal3a7c.png", "source_url": "https://dev.to/epicx/how-ai-agents-like-amazons-nova-are-changing-the-way-we-browse-the-web-5ckf", "title": "How AI Agents Like Amazon\u2019s Nova Are Changing the Way We Browse the Web", "s3_key": null, "type": "image", "id": 202281, "html": null, "alt": null}]}, {"header": "Reasoning with System 1 & 2 Thinking", "content": "

Reasoning is a key capability that integrates a variety of cognitive functions, that are separated into two systems. First, is System 1, which refers to fast, automatic, and intuitive thinking, typically employed for simple and routine tasks. Second, is System 2, which encompasses slow, deliberate, and analytical thinking, crucial for solving complex tasks.

System 2 Reasoning represents the agent\u2019s ability to handle complex, multi-step tasks through explicit intermediate thinking processes. The System 2 mechanism of UI-TARS uses techniques like task decomposition, long-term consistency, milestone recognition, and reflection.

", "media": [{"caption": "Build Your Own AI Browser Agent: A Step-by-Step Guide with Browser Use & Python - Diffused Creations", "url": "https://i3.wp.com/diffusedcreations.com/wp-content/uploads/Browser-Use-AI-Agent-1024x579.png?ssl=1", "source_url": "https://diffusedcreations.com/build-your-own-ai-browser-agent-a-step-by-step-guide-with-browser-use-python/", "title": "Build Your Own AI Browser Agent: A Step-by-Step Guide with Browser Use & Python - Diffused Creations", "s3_key": null, "type": "image", "id": 202282, "html": null, "alt": null}]}, {"header": "Memory Utilization", "content": "

The memory components of UI-TARS are used to store knowledge and historical experience that the agent refers to when making decisions. It uses short-term memory for task-specific information and long-term memory for background knowledge. Native agent models encode long-term operational experience within their internal parameters, converting the interaction process into implicit, parameterized storage.

", "media": []}]}