README

[CHI 2025] AppAgent 



      
       
       
       
 
 

Chi Zhang†, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang,

Bin Fu, Gang Yu✦

( equal contribution, † Project Leader, ✦ Corresponding Author )



!

ℹ️ 🔥🔥🔥 AppAgentX is released, the next-generatation GUI Agent with evolving mechanism.

ℹ️Should you encounter any issues⚠️ while using our project, please feel free to report them on GitHub Issues or reach out to Dr. Chi Zhang via email at [email protected].

ℹ️This project will be synchronously updated on the official TencentQQGYLab Github Page.

 📝 Changelog

- __[2025.3.5]__: 🔥🔥🔥AppAgentX is released, the next-generatation GUI Agent with evloving mechanism.
- __[2024.2.8]__: Added qwen-vl-max (通义千问-VL) as an alternative multi-modal model. The model is currently free to use but has a relatively poorer performance compared with GPT-4V.
- __[2024.1.31]__: Released the evaluation benchmark used during our testing of AppAgent
- __[2024.1.2]__: Added an optional method for the agent to bring up a grid overlay on the screen to tap/swipe anywhere on the screen.
- __[2023.12.26]__: Added Tips section for better use experience; added instruction for using the Android Studio emulator for
  users who do not have Android devices.
- __[2023.12.21]__:  Open-sourced the git repository, including the detailed configuration steps to implement our AppAgent!


 🔆 Introduction

We introduce a novel LLM-based multimodal agent framework designed to operate smartphone applications. 

Our framework enables the agent to operate smartphone applications through a simplified action space, mimicking human-like interactions such as tapping and swiping. This novel approach bypasses the need for system back-end access, thereby broadening its applicability across diverse apps.

Central to our agent's functionality is its innovative learning method. The agent learns to navigate and use new apps either through autonomous exploration or by observing human demonstrations. This process generates a knowledge base that the agent refers to for executing complex tasks across different applications.


 ✨ Demo

The demo video shows the process of using AppAgent to follow a user on X (Twitter) in the deployment phase.

https://github.com/mnotgod96/AppAgent/assets/40715314/db99d650-dec1-4531-b4b2-e085bfcadfb7

An interesting experiment showing AppAgent's ability to pass CAPTCHA.

https://github.com/mnotgod96/AppAgent/assets/27103154/5cc7ba50-dbab-42a0-a411-a9a862482548

An example of using the grid overlay to locate a UI element that is not labeled with a numeric tag.

https://github.com/mnotgod96/AppAgent/assets/27103154/71603333-274c-46ed-8381-2f9a34cdfc53

 🚀 Quick Start

This section will guide you on how to quickly use gpt-4-vision-preview (or qwen-vl-max) as an agent to complete specific tasks for you on
your Android app.

 ⚙️ Step 1. Prerequisites

1. On you