目录：

UI Agents 知识星球UI Agents 综述资料【Update 2025.03.22】UI Agents 论文列表【Update 2025.03.22】Policy ModelsTraining-based ModelsTraining-free ModelsEnhanced KnowledgesData SynthesisDatasets / BenchmarksTools / EnvironmentsOthersUI Agents 其他汇总信息

UI Agents 知识星球

加入知识星球，每周获取会员专享视频👇

 

扫码加微信小助手为好友，备注「agent」，小助手会定期邀请入群👇

UI Agents 综述资料【Update 2025.03.22】

• UI Agents（智能体）技术综述, Bilibili, Youtube, Breezedeus, 2024.11

• [2503.11069] API Agents vs. GUI Agents: Divergence and Convergence

• [2501.16150] AI Agents for Computer Use: A Review of Instruction-based Computer Control, GUI Automation, and Operator Assistants

• [2501] OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use, OPPO

• [2412.13501] GUI Agents: A Survey

• [2412.10047] Large Action Models: From Inception to Implementation, Microsoft

• [2411.10943] Generalist Virtual Agents: A Survey on Autonomous Agents Across Digital Platforms

• [2411.18279] Large Language Model-Brained GUI Agents: A Survey, Microsoft

• [2411.10323] The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use

• [2411.04890] GUI Agents with Foundation Models: A Comprehensive Survey, Huawei Noah’s Ark Lab

• [2411.02006] Foundations and Recent Trends in Multimodal Mobile Agents: A Survey

UI Agents 论文列表【Update 2025.03.22】

Policy Models

Training-based Models

• [2503.02197] ATLaS: Agent Tuning via Learning Critical Steps

• [2502.13130] Magma: A Foundation Model for Multimodal AI Agents

• [2502.02955] ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation

• [2501.12326] UI-TARS: Pioneering Automated GUI Interaction with Native Agents, ByteDance

• [2501.04575] InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection

• [2412.17589] PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital World

• [2412.16256] Aria-UI: Visual Grounding for GUI Instructions

• [2412.09362] Falcon-UI: Understanding GUI Before Following User Instructions

• [2412.04454] Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction

• [2412.01268] Ponder & Press: Advancing Visual GUI Agent towards General Computer Control

• [2411.17465] ShowUI: One Vision-Language-Action Model for GUI Visual Agent, Github

• [2402.15506] AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning

• [2411.13451] AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations

• [2411.00820] AutoGLM: Autonomous Foundation Agents for GUIs, Zhipu

• [2410.19461] EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data

• [2410.22916] Explainable Behavior Cloning: Teaching Large Language Model Agents through Learning by Demonstration

• [2406.19263] Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding

• [2312.08914] CogAgent: A Visual Language Model for GUI Agents

• [2312.15820] WebVLN: Vision-and-Language Navigation on Websites

• [2502.18906] VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Model

• [2502.15760] Digi-Q: Learning Q-Value Functions for Training Device-Control Agents

• [2502.12130] Scaling Autonomous Agents via Automatic Reward Modeling And Planning

• [2502.07949] VSC-RL: Advancing Autonomous Vision-Language Agents with Variational Subgoal-Conditioned Reinforcement Learning

• [2412.10742] WEPO: Web Element Preference Optimization for LLM-based Web Navigation

• [2412.06313] Vision-Based Deep Reinforcement Learning of UAV Autonomous Navigation Using Privileged Information

• [2411.03817] From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning

• [2411.02337] WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning, Zhipu

• [2410.24218] Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use

• [2406.11896] DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning

• [2406.04151] AgentGym: Evolving Large Language Model-based Agents across Diverse Environments

Training-free Models

• [2503.09572] Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks long-horizon

• [2503.03459] Unified Mind Model: Reimagining Autonomous Agents in the LLM Era

• [2502.16796] MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructions

• [2502.14282] PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC

• [2501.11733] Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks

• [2503.15937] Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deployment

• [2503.03196] SpiritSight Agent: Advanced GUI Agent with One Look

• [2503.03743] CHOP: Mobile Operating Assistant with Constrained High-frequency Optimized Subtask Planning

• [2503.13843] WebNav: An Intelligent Agent for Voice-Controlled Web Navigation

• [2503.10689] Learning to Contextualize Web Pages for Enhanced Decision Making by LLM Agents

• [2503.06580] Agent models: Internalizing Chain-of-Action Generation into Reasoning models

• [2503.02950] LiteWebAgent: The Open-Source Suite for VLM-Based Web-Agent Applications

• [2503.02268] AppAgentX: Evolving GUI Agents as Proficient Smartphone Users

• [2502.09215] Architecture for Simulating Behavior Mode Changes in Norm-Aware Autonomous Agents

• [2502.08226] TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents

• [2502.07056] Autonomous Deep Agent

• [2501.12485] R2D2: Remembering, Reflecting and Dynamic Decision Making for Web Agents

• [2412.18116] AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation

• [2412.10840] Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning

• [2412.07472] SmartAgent: Chain-of-User-Thought for Embodied Personalized Agent in Cyber World

• [2411.15004] ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data

• [2411.06559] Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents

• [2408.06458] Towards Autonomous Agents: Adaptive-planning, Reasoning, and Acting in Language Models

• [2411.13591] Improved GUI Grounding via Iterative Narrowing

• [2410.19609] OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization

• [2408.00203] OmniParser for Pure Vision Based GUI Agent

• [2407.03913] MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices

• [2406.06947] CAAP: Context-Aware Action Planning Prompting to Solve Computer Tasks with Front-End UI Only

• [2405.15341] V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal LLM

• [2312.11190] VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning

Enhanced Knowledges

• [2501.11425] Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training

• [2410.13232][WMA] Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation

• [2410.23555] From Context to Action: Analysis of the Impact of State Representation and Context on the Generalization of Multi-Turn Web Navigation Agents

• [2410.22552] Auto-Intent: Automated Intent Discovery and Self-Exploration for Large Language Model Web Agents

• [2406.14596] VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought

• [2405.16247] AutoManual: Generating Instruction Manuals by LLM Agents via Interactive Environmental Learning

Data Synthesis

• [2502.11357] Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents

• [2502.07942] Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMs

• [2502.02982] FedMobileAgent: Training Mobile Agents Using Decentralized Self-Sourced Data from Diverse Users

• [2501.13896] GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration

• [2501.10893] Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments

• [2412.19723] OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis

• [2412.13194] Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents

• [2412.10342] Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining

• [2412.09605] AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials

• [2410.02907] NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wild

Datasets / Benchmarks

• [2503.09780] AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents

• [2503.04957] SafeArena: Evaluating the Safety of Autonomous Web Agents

• [2503.03056] A2Perf: Real-World Autonomous Agents Benchmark

• [2503.02403] AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agents

• [2502.18356] WebGames: Challenging General-Purpose Web-Browsing AI Agents

• [2502.15840] Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents long-horizon

• [2501.02863] Beyond Pass or Fail: Multi-Dimensional Benchmarking of Foundation Models for Goal-based Mobile UI Navigation

• [2403.11905] Tur[k]ingBench: A Challenge Benchmark for Web Agents

• [2410.19100] VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks

• [2502.13053] AEIA-MN: Evaluating the Robustness of Multimodal LLM-Powered Mobile Agents Against Active Environmental Injection Attacks

• [2502.08047] WorldGUI: Dynamic Testing for Comprehensive Desktop GUI Automation

• [2501.16609] CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation

• [2410.23252] Evaluating Cultural and Social Awareness of LLM Web Agents

• ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use

• [2501.01149] A3: Android Agent Arena for Mobile GUI Agents

• [2412.18426] GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent

• [2410.17520] MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control

• [2412.04531] MageBench: Bridging Large Multimodal Models to Agents

• [2412.05789] InfiniteWorld: A Unified Scalable Simulation Framework for General Visual-Language Robot Interaction

• [2410.06703] ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents

• [2410.24024] AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents, Zhipu

• [2410.15164] SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation

• [2410.17520] MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control

• [2406.03679][AndroidControl] On the Effects of Data Scale on UI Control Agents

• [2406.14250] E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion

• [2405.14573] AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

• [2209.08199] ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots

Tools / Environments

• browser-use/browser-use: Make websites accessible for AI agents
• browser-use/web-ui: Run AI Agent in your browser.
• browser-use/macOS-use: Make Mac apps accessible for AI agents

• maitrix-org/llm-reasoners: A library for advanced large language model reasoning

• web-infra-dev/midscene: Let AI be your browser operator.

• OmniParser/omnitool at master · microsoft/OmniParser

• bytedance/UI-TARS-desktop: A GUI Agent application based on UI-TARS(Vision-Lanuage Model) that allows you to control your computer using natural language.

• hrithikkoduri/WebRover: WebRover is an autonomous AI agent designed to interpret user input and execute actions by interacting with web elements to accomplish tasks or answer questions. It leverages advanced language models and web automation tools to navigate the web, gather information, and provide structured responses based on the user's needs.

• [2412.05467] The BrowserGym Ecosystem for Web Agent Research

• [2403.17918] AgentStudio: A Toolkit for Building General Virtual Agents

Others

• [2503.09385] PCLA: A Framework for Testing Autonomous Agents in the CARLA Simulator

• [2503.08464] An Autonomous RL Agent Methodology for Dynamic Web UI Testing in a BDD Framework

• [2502.20383] Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysis

• [2502.17903] Towards Sustainable Web Agents: A Plea for Transparency and Dedicated Metrics for Energy Consumption

UI Agents 其他汇总信息

• https://github.com/OSU-NLP-Group/GUI-Agents-Paper-List

• https://github.com/showlab/Awesome-GUI-Agent

• https://github.com/opendilab/awesome-ui-agents