![\"GUI](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/smol2operator/google.png\")
\\nclick(x=0.41, y=0.178)\\n\",\n}\n```\n\nEach sample links a screenshot with a system/user/assistant turn. During fine-tuning, the data collator masks everything except the assistant’s answers when computing the loss.\n\n### Phase 2 Results\n\nStarting from the Phase 1 checkpoint (1152 px resolution, normalized coordinates), we fine-tuned the model for two epochs on [smolagents/aguvis-stage-2](https://huggingface.co/datasets/smolagents/aguvis-stage-2). The accuracy on **ScreenSpot-v2 increased from 41% to 61%**, indicating that explicit reasoning improves GUI grounding performance.\n\n| Configuration (coords / image size) | Screenspot-v2 (%) |\n| ----------------------------------- | ----------------- |\n| Normalized coordinates / 1152 | 61.71 |\n\n**Table 2:** *Baseline on HuggingFaceTB/SmolVLM2-2.2B-Instruct after Phase 1 finetuning (2 epochs, aguvis-stage-1).*\n\n💡 *We also reproduced the two-phase training on a much smaller VLM (nanoVLM-460M). Despite its reduced capacity, the model achieved **\\~58% on ScreenSpot-v2**, demonstrating that the training strategy scales down effectively, **making it SOTA on ScreenSpot-v2 for this model size (460M parameters)**. In addition, aguvis-stage-1 is already included in [FineVision Dataset](https://huggingface.co/datasets/HuggingFaceM4/FineVision)!*\n\n## 4. All you need is Open Source\n\nAll training code, data processing pipelines, datasets and model are open-source!\n\n1. **Training Recipe** ([`recipe.ipynb`](https://github.com/huggingface/smol2operator/blob/main/recipe.ipynb)): Complete training pipeline for both Phase 1 and Phase 2, including dataset mixture configurations and training orchestration. We leverage the [TRL](https://huggingface.co/docs/trl/en/index) library to train our models.\n2. **Datasets** ([`smolagents/aguvis-stage-1`](https://huggingface.co/datasets/smolagents/aguvis-stage-1), [`smolagents/aguvis-stage-2`](https://huggingface.co/datasets/smolagents/aguvis-stage-2)): all datasets used are open-source.\n3. **Model** ([`smolagents/SmolVLM2-2.2B-Instruct-Agentic-GUI`](https://huggingface.co/smolagents/SmolVLM2-2.2B-Instruct-Agentic-GUI)): the model produced by applying the training recipe described above.\n4. **Preprocessing Tools:**\n - **Function Parser** ([`utils/function_parser.py`](https://github.com/huggingface/smol2operator/blob/main/utils/function_parser.py)): Utilities for parsing, normalizing, and reconstructing function calls from diverse dataset formats. Supports complex parameter structures, positional arguments, and multiple function call extraction.\n - **Action Conversion System** ([`preprocessing/action_conversion.py`](https://github.com/huggingface/smol2operator/blob/main/preprocessing/action_conversion.py)): Core unification engine transforming mobile and PyAutoGUI desktop actions into a standardized API format. Features smart coordinate handling, direction detection for scroll actions, and comprehensive parameter normalization.\n - **Action Space Converter** ([`utils/action_space_converter.py`](https://github.com/huggingface/smol2operator/blob/main/utils/action_space_converter.py)): Flexible tool for adapting the unified action space to custom vocabularies and naming conventions. Enables domain-specific customization through configurable parameter mappings.\n\n💡 We’ve also released a Space to try the model’s agentic grounding capabilities: [A-Mahla/Smol2Operator](https://huggingface.co/spaces/A-Mahla/Smol2Operator)\n\n## 5. Conclusion\n\nOur experiments demonstrate that high-quality, reasoning-oriented data can substantially improve GUI grounding, even for small VLMs, using only supervised fine-tuning (SFT). Beyond raw performance gains, these results show that the GUI grounding capabilities are largely determined by the quality of the data. Carefully curated datasets teach models the structure and semantics of user interfaces, providing the grounding needed for accurate action prediction.\n\nTo support the development of GUI agents, we’re open-sourcing everything: our complete pipeline, datasets, and trained model. You can reproduce our results, experiment with different models and architectures, or adapt our approach to new domains. The future of agentic AI depends on researchers like you pushing these boundaries further!\n\n## What's Next?\n\nWhile SFT excels at supervised tasks, emerging methods such as Reinforcement Learning (RL) or Direct Preference Optimization (DPO) help develop stronger reasoning capabilities and enable real-time adaptation. These advances point toward a new generation of GUI agents that learn and improve through interaction rather than relying solely on static datasets.\n\nLet’s build the future of GUI agents together 🤗\n",
"authors": [],
"captured_at": "2026-10-04T12:50:30.804479567+00:00",
"category": "ai",
"content_html": "TL;DR: This work shows how a lightweight vision–language model can acquire GUI-grounded skills and evolve into an agentic GUI coder. We release all training recipes, data-processing tools, resulting model, demo and datasets to enable full reproducibility and foster further research 🫡. Find the collection here.
\nThis video demonstrates the model obtained through the recipe described below, executing a task end-to-end.
\nGraphical User Interface (GUI) automation is one of the most challenging frontiers in computer vision. Developing models that see and interact with user interfaces enables AI agents to navigate mobile, desktop, and web platforms. This will reshape the future of digital interaction.
\nIn this blog post, we present a comprehensive approach to training vision-language models for GUI automation through a multi-phase training strategy. We demonstrate how to transform a model with zero grounding capabilities into an agentic coder capable of understanding and interacting with graphical interfaces.
\nRather than aiming for a SOTA model, our goal is to demonstrate the entire process, from data processing to model training, and, in doing so, show how to unlock GUI-grounding capabilities in VLMs.
\n
GUI capabilities combine understanding of the interface and precise element localization. These abilities enable the model to translate high-level tasks into low-level GUI actions such as clicking, typing, …
\nOur approach leverages SmolVLM2-2.2B-Instruct as the baseline model, a small powerful vision-language model that initially has no grounding capabilities for GUI tasks. This makes it an ideal candidate to demonstrate the effectiveness of our training methodology. Through our two-phase training process, we first instill grounding capabilities in the model, then enhance it with agentic reasoning abilities using Supervised Fine-Tuning (SFT).
\nWe evaluate our approach on an established perception benchmark: ScreenSpot-v2, which tests the model’s ability to understand and locate elements within screenshots. Our process is inspired by the AGUVIS paper, and we leverage their carefully curated datasets to build upon their foundational work.
\n![\"Evolution](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/smol2operator/screenspot-v2.png\")
Evolution of ScreenSpot-v2 performance during the training phase of the base model SmolVLM2-2.2B-Instruct.
\nThis section explains how we convert heterogeneous GUI actions format from multiple datasets into a single unified format. By standardizing function names, signatures, and parameters, we create consistent, high-quality data that forms the foundation for effective model training.
\nOne of the primary challenges when working with multiple GUI automation datasets is the lack of standardization in action representations. Different datasets use varying function signatures, parameter naming conventions, and action taxonomies, making it difficult to train a unified model across diverse data sources.
\nWe took the open-source datasets (xlangai/aguvis-stage1, xlangai/aguvis-stage2), originally used by AGUVIS, and implemented a comprehensive data transformation pipeline to create a unified action space. Our approach involved:
\nutils/function_parser.py) that can extract and parse function calls from various formats across all datasets. This parser supports any function signature format, handles complex parameter structures, and can reconstruct function calls with proper parameter ordering.preprocessing/action_conversion.py) that transforms all original action representations into a standardized function naming and argument structure. This process highlighted the significant inconsistencies in function signatures across different datasets and allowed us to:\nutils/action_space_converter.py toolHere are real examples from our action conversion system (preprocessing/action_conversion.py) showing how we transform heterogeneous action representations into our unified format (grounding coordinates normalized to [0,1]):
Before (Original Action Dataset Formats):
\n# Mobile Actions\n\n\n\n\n\n# Desktop Actions\n\n\n\n\n\n\nAfter (Unified Action Dataset Formats):
\n# Unified Mobile Actions\n\n\n\n\n\n# Unified Desktop Actions\n\n\n\n # Smart direction detection\n\n\nThis unification process was essential for creating coherent training data that allows the model to learn consistent action patterns across diverse GUI environments.
\nđź’ˇ Why Normalized Coordinates?
\nUsing raw pixel coordinates in text-action datapoint (e.g. click(x=302, y=63)) ties them to a single image size. Vision Language Models (VLMs) often resize images, causing pixel coordinates to break and require adjustment. Normalized coordinates (relative to image size) remain valid at any resolution and keep the dataset consistent.
To maximize flexibility for different use cases, we developed the Action Space Converter (utils/action_space_converter.py), a tool that allows users to easily adapt from an action space to their own custom action vocabularies and naming conventions.
You can use this tool to transform one action signature (function names, parameter names, and parameter value changes, ...) into another:
\nBefore
\n: \nAfter
\n: \nThe Action Space Converter provides:
\n\n\n\n# Create custom mappings\n = \n\n = \n\n# Parse function calls\n = \n\n# Initialize converter\n = \n\n# Convert actions\n = \n\n = \n\n\n# Output: I'll interact at those coordinates for you. touch(x_coord=0.5, y_coord=0.3) Now I'll input the text. write(content='hello world')\nThis tool enables researchers and practitioners to:
\nThe Action Space Converter is particularly valuable for preparing datasets for training, as it ensures consistent action vocabularies across different deployment environments while maintaining compatibility with existing automation frameworks.
\nThrough this pipeline, we transform the open-source datasets xlangai/aguvis-stage1, xlangai/aguvis-stage2 into our unified action space (see here). The output of this process is released as two new fully formatted datasets: smolagents/aguvis-stage-1 and smolagents/aguvis-stage-2.
\nPhase 1 leverages the smolagents/aguvis-stage-1 dataset, which introduces GUI grounding by pairing low-level instructions with diverse executable actions (expressed in code form). For example, a user/assistant turn in smolagents/aguvis-stage-1 follows the structure:
\n\nEach sample links a screenshot with multi-turn user/assistant interactions, enabling the model to learn fine-grained action grounding across dialogue turns. During fine-tuning, the data collator masks everything except the assistant’s answers when computing the loss.
\nBefore proceeding with full-scale Phase 1 training, we conducted comprehensive ablation studies to determine optimal training configurations
\nWe experimented with different image sizes and coordinate representation systems to identify the optimal configuration for SmolVLM2:
\n\n\nSome SOTA GUI VLMs (e.g., Qwen-VL) appear also to use a different normalized range (0–1000), which was not tested in this experiment.
\n
| Configuration (coords / image size) | \nScreenspot-v2 (%) | \n
|---|---|
| Normalized coordinates | \n\n |
| Base / – | \n0.47 | \n
| 384 | \n31.28 | \n
| 764 | \n32.32 | \n
| 1152 | \n33.72 | \n
| Pixel coordinates | \n\n |
| Base / – | \n0.55 | \n
| 384 | \n1.17 | \n
| 764 | \n2.67 | \n
| 1152 | \n4.32 | \n
Table 1: Baseline on HuggingFaceTB/SmolVLM2-2.2B-Instruct (400k samples, aguvis-stage-1). Higher is better.
\nAs demonstrated in our benchmark results, SmolVLM2-2.2B-Instruct base initially achieved 0% performance on perception benchmarks like ScreenSpot-v2. This complete lack of grounding capability provided us with a clean slate to evaluate the effectiveness of our training methodology.
\nFrom our experiments, we determined that:
\nUsing the optimal configuration (1152px resolution with normalized coordinates), we trained for 2 epochs on the smolagents/aguvis-stage-1 dataset. The results were remarkable, +41% improvement over baseline on ScreenSpot-v2
\nThis dramatic improvement demonstrates that our Phase 1 training successfully instilled fundamental grounding capabilities in the model, enabling it to understand and locate visual elements within screenshots.
\n| Configuration (coords / image size) | \nScreenspot-v2 (%) | \n
|---|---|
| Normalized coordinates / 1152 | \n41.27 | \n
Table 2: Baseline on HuggingFaceTB/SmolVLM2-2.2B-Instruct (2 epochs, aguvis-stage-1).
\nWhereas Phase 1 provided grounding capabilities, Phase 2 targets agentic reasoning, the ability to deliberate and plan before acting. This stage transforms the model from a reactive system identifying GUI elements into a proactive agent capable of executing complex, multi-step interactions.
\nPhase 2 uses the smolagents/aguvis-stage-2 dataset, which introduces agentic scenarios:
\nExplicit reasoning about upcoming actions
\nContext consistency across multiple interaction steps
\nHigh-level instructions require multi-step, low-level actions.
\nFor example, the smolagents/aguvis-stage-2 chat message is like this:
\n\nEach sample links a screenshot with a system/user/assistant turn. During fine-tuning, the data collator masks everything except the assistant’s answers when computing the loss.
\nStarting from the Phase 1 checkpoint (1152 px resolution, normalized coordinates), we fine-tuned the model for two epochs on smolagents/aguvis-stage-2. The accuracy on ScreenSpot-v2 increased from 41% to 61%, indicating that explicit reasoning improves GUI grounding performance.
\n| Configuration (coords / image size) | \nScreenspot-v2 (%) | \n
|---|---|
| Normalized coordinates / 1152 | \n61.71 | \n
Table 2: Baseline on HuggingFaceTB/SmolVLM2-2.2B-Instruct after Phase 1 finetuning (2 epochs, aguvis-stage-1).
\nđź’ˇ We also reproduced the two-phase training on a much smaller VLM (nanoVLM-460M). Despite its reduced capacity, the model achieved ~58% on ScreenSpot-v2, demonstrating that the training strategy scales down effectively, making it SOTA on ScreenSpot-v2 for this model size (460M parameters). In addition, aguvis-stage-1 is already included in FineVision Dataset!
\nAll training code, data processing pipelines, datasets and model are open-source!
\nrecipe.ipynb): Complete training pipeline for both Phase 1 and Phase 2, including dataset mixture configurations and training orchestration. We leverage the TRL library to train our models.smolagents/aguvis-stage-1, smolagents/aguvis-stage-2): all datasets used are open-source.smolagents/SmolVLM2-2.2B-Instruct-Agentic-GUI): the model produced by applying the training recipe described above.utils/function_parser.py): Utilities for parsing, normalizing, and reconstructing function calls from diverse dataset formats. Supports complex parameter structures, positional arguments, and multiple function call extraction.preprocessing/action_conversion.py): Core unification engine transforming mobile and PyAutoGUI desktop actions into a standardized API format. Features smart coordinate handling, direction detection for scroll actions, and comprehensive parameter normalization.utils/action_space_converter.py): Flexible tool for adapting the unified action space to custom vocabularies and naming conventions. Enables domain-specific customization through configurable parameter mappings.💡 We’ve also released a Space to try the model’s agentic grounding capabilities: A-Mahla/Smol2Operator
\nOur experiments demonstrate that high-quality, reasoning-oriented data can substantially improve GUI grounding, even for small VLMs, using only supervised fine-tuning (SFT). Beyond raw performance gains, these results show that the GUI grounding capabilities are largely determined by the quality of the data. Carefully curated datasets teach models the structure and semantics of user interfaces, providing the grounding needed for accurate action prediction.
\nTo support the development of GUI agents, we’re open-sourcing everything: our complete pipeline, datasets, and trained model. You can reproduce our results, experiment with different models and architectures, or adapt our approach to new domains. The future of agentic AI depends on researchers like you pushing these boundaries further!
\nWhile SFT excels at supervised tasks, emerging methods such as Reinforcement Learning (RL) or Direct Preference Optimization (DPO) help develop stronger reasoning capabilities and enable real-time adaptation. These advances point toward a new generation of GUI agents that learn and improve through interaction rather than relying solely on static datasets.
\nLet’s build the future of GUI agents together 🤗
\n", "content_markdown": "**TL;DR:** This work shows how a lightweight vision–language model can acquire GUI-grounded skills and evolve into an agentic GUI coder. We release all training recipes, data-processing tools, resulting model, demo and datasets to enable full reproducibility and foster further research 🫡. Find the collection [here](https://huggingface.co/collections/smolagents/smol2operator-release-68d288e87d3fa8f551d2ce2e).\n\n* * *\n\nThis video demonstrates the model obtained through the recipe described below, executing a task end-to-end.\n\n* * *\n\n## Table of Contents\n\n- [Introduction](https://huggingface.co/blog/smol2operator#introduction)\n- [1\\. Data Transformation and Unified Action Space](https://huggingface.co/blog/smol2operator#1-data-transformation-and-unified-action-space)\n - [The Challenge of Inconsistent Action Spaces](https://huggingface.co/blog/smol2operator#the-challenge-of-inconsistent-action-spaces)\n - [Our Unified Approach](https://huggingface.co/blog/smol2operator#our-unified-approach)\n - [Example Data Transformation](https://huggingface.co/blog/smol2operator#example-data-transformation)\n - [Custom Action Space Adaptation with Action Space Converter](https://huggingface.co/blog/smol2operator#bonus-custom-action-space-adaptation-with-action-space-converter)\n - [Key Features](https://huggingface.co/blog/smol2operator#key-features)\n - [Usage Example](https://huggingface.co/blog/smol2operator#usage-example)\n - [Transformed and Released Datasets](https://huggingface.co/blog/smol2operator#transformed-and-released-datasets)\n- [2\\. Phase 1: From Zero to Perception](https://huggingface.co/blog/smol2operator#2-phase-1-from-zero-to-perception)\n - [Training Data](https://huggingface.co/blog/smol2operator#training-data)\n - [Optimization Experiments](https://huggingface.co/blog/smol2operator#optimization-experiments)\n - [Image Resolution and Coordinate System Analysis](https://huggingface.co/blog/smol2operator#image-resolution-and-coordinate-system-analysis)\n - [Key Findings](https://huggingface.co/blog/smol2operator#key-findings)\n - [Phase 1 Results](https://huggingface.co/blog/smol2operator#phase-1-results)\n- [3\\. Phase 2: From Perception to Cognition](https://huggingface.co/blog/smol2operator#3-phase-2-from-perception-to-cognition)\n - [Training Data](https://huggingface.co/blog/smol2operator#training-data-1)\n - [Phase 2 Results](https://huggingface.co/blog/smol2operator#phase-2-results)\n- [4\\. All you need is Open Source](https://huggingface.co/blog/smol2operator#4-all-you-need-is-open-source)\n- [5\\. Conclusion](https://huggingface.co/blog/smol2operator#5-conclusion)\n- [What's Next?](https://huggingface.co/blog/smol2operator#whats-next)\n\n## Introduction\n\nGraphical User Interface (GUI) automation is one of the most challenging frontiers in computer vision. Developing models that see and interact with user interfaces enables AI agents to navigate mobile, desktop, and web platforms. This will reshape the future of digital interaction.\n\nIn this blog post, we present a comprehensive approach to training vision-language models for GUI automation through a multi-phase training strategy. We demonstrate how to transform a model with zero grounding capabilities into an agentic coder capable of understanding and interacting with graphical interfaces.\n\nRather than aiming for a SOTA model, our goal is to demonstrate the entire process, from data processing to model training, and, in doing so, show how to unlock GUI-grounding capabilities in VLMs.\n\n[](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/smol2operator/google.png)\n\n*GUI capabilities combine understanding of the interface and precise element localization. These abilities enable the model to translate high-level tasks into low-level GUI actions such as clicking, typing, …*\n\nOur approach leverages [**SmolVLM2-2.2B-Instruct**](https://huggingface.co/HuggingFaceTB/SmolVLM2-2.2B-Instruct) as the baseline model, a small powerful vision-language model that initially has no grounding capabilities for GUI tasks. This makes it an ideal candidate to demonstrate the effectiveness of our training methodology. Through our two-phase training process, we first instill grounding capabilities in the model, then enhance it with agentic reasoning abilities using Supervised Fine-Tuning (SFT).\n\nWe evaluate our approach on an established perception benchmark: [**ScreenSpot-v2**](https://huggingface.co/datasets/HongxinLi/ScreenSpot_v2), which tests the model’s ability to understand and locate elements within screenshots. Our process is inspired by the [AGUVIS](https://huggingface.co/papers/2412.04454) paper, and we leverage their carefully curated datasets to build upon their foundational work.\n\n[](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/smol2operator/screenspot-v2.png)\n\n*Evolution of ScreenSpot-v2 performance during the training phase of the base model **SmolVLM2-2.2B-Instruct**.*\n\n## 1. Data Transformation and Unified Action Space\n\n*This section explains how we **convert heterogeneous GUI actions format from multiple datasets into a single unified format**. By standardizing function names, signatures, and parameters, we create consistent, high-quality data that forms the foundation for effective model training.*\n\n### The Challenge of Inconsistent Action Spaces\n\nOne of the primary challenges when working with multiple GUI automation datasets is the lack of standardization in action representations. Different datasets use varying function signatures, parameter naming conventions, and action taxonomies, making it difficult to train a unified model across diverse data sources.\n\n### Our Unified Approach\n\nWe took the open-source datasets ([xlangai/aguvis-stage1](https://huggingface.co/datasets/xlangai/aguvis-stage1), [xlangai/aguvis-stage2](https://huggingface.co/datasets/xlangai/aguvis-stage2)), originally used by [AGUVIS](https://huggingface.co/papers/2412.04454), and implemented a comprehensive data transformation pipeline to create a unified action space. Our approach involved:\n\n1. **Function Parsing and Normalization**: We developed a function parser (see `utils/function_parser.py`) that can extract and parse function calls from various formats across all datasets. This parser supports any function signature format, handles complex parameter structures, and can reconstruct function calls with proper parameter ordering.\n2. **Action Space Unification**: We implemented a comprehensive action conversion system (see `preprocessing/action_conversion.py`) that transforms all original action representations into a standardized function naming and argument structure. This process highlighted the significant inconsistencies in function signatures across different datasets and allowed us to:\n - Remove undesired or redundant actions\n - Standardize parameter naming conventions\n - Create a cohesive action vocabulary\n3. **(Bonus) Flexible Adaptation Framework**: Our transformation pipeline includes utilities that allow users to:\n - Adapt the entire dataset to their own action space naming conventions using the `utils/action_space_converter.py` tool\n - Extract and analyze the current action space structure\n\n### Example Data Transformation\n\nHere are real examples from our action conversion system (`preprocessing/action_conversion.py`) showing how we transform heterogeneous action representations into our unified format (grounding coordinates normalized to \\[0,1\\]):\n\n**Before (Original Action Dataset Formats):**\n\n```python\n# Mobile Actions\nmobile.home()\nmobile.open_app(app_name='drupe')\nmobile.swipe(from_coord=[0.581, 0.898], to_coord=[0.601, 0.518])\nmobile.long_press(x=0.799, y=0.911)\nmobile.terminate(status='success')\n# Desktop Actions\npyautogui.click(x=0.8102, y=0.9463)\npyautogui.doubleClick(x=0.8102, y=0.9463)\npyautogui.hotkey(keys=['ctrl', 'c'])\npyautogui.scroll(page=-0.1)\npyautogui.write(message='bread buns')\npyautogui.dragTo(from_coord=[0.87, 0.423], to_coord=[0.8102, 0.9463])\n```\n\n**After (Unified Action Dataset Formats):**\n\n```python\n# Unified Mobile Actions\nnavigate_home()\nopen_app(app_name='drupe')\nswipe(from_coord=[0.581, 0.898], to_coord=[0.601, 0.518])\nlong_press(x=0.799, y=0.911)\nfinal_answer('success')\n# Unified Desktop Actions\nclick(x=0.8102, y=0.9463)\ndouble_click(x=0.8102, y=0.9463)\npress(keys=['ctrl', 'c'])\nscroll(direction='up', amount=10) # Smart direction detection\ntype(text='bread buns')\ndrag(from_coord=[0.87, 0.423], to_coord=[0.8102, 0.9463])\n```\n\nThis unification process was essential for creating coherent training data that allows the model to learn consistent action patterns across diverse GUI environments.\n\n**💡 Why Normalized Coordinates?** \\\n Using raw pixel coordinates in text-action datapoint (e.g. `click(x=302, y=63)`) ties them to a single image size. Vision Language Models (VLMs) often resize images, causing pixel coordinates to break and require adjustment. Normalized coordinates (relative to image size) remain valid at any resolution and keep the dataset consistent.\n\n### (Bonus) Custom Action Space Adaptation with Action Space Converter\n\nTo maximize flexibility for different use cases, we developed the **Action Space Converter** (`utils/action_space_converter.py`), a tool that allows users to easily adapt from an action space to their own custom action vocabularies and naming conventions.\n\nYou can use this tool to transform one action signature (function names, parameter names, and parameter value changes, ...) into another:\n\n**Before**\n\n```python\nassistant_message: \"Action: click(x=0.5, y=0.3)\"\n```\n\n**After**\n\n```python\nassistant_message: \"Action: touch(x_coord=200, y_coord=300)\"\n```\n\n### Key Features\n\nThe Action Space Converter provides:\n\n1. **Configurable Mappings**: Define custom mappings between unified actions and your preferred action names\n2. **Parameter Transformation**: Rename parameters, apply value transformations, and set default values\n3. **Flexible Architecture**: Support for both simple parameter mappings and complex custom transformation functions\n4. **Validation**: Built-in validation to ensure mapping configurations are valid\n\n### Usage Example\n\n```python\nfrom utils.action_space_converter import ActionSpaceConverter, ActionMapping, ParameterMapping\nfrom utils.function_parser import parse_function_call\n\n# Create custom mappings\nmappings = [\n ActionMapping(\n source_function=\"click\",\n target_function=\"touch\",\n parameter_mappings=[\n ParameterMapping(source_name=\"x\", target_name=\"x_coord\"),\n ParameterMapping(source_name=\"y\", target_name=\"y_coord\")\n ],\n description=\"Touch screen at coordinates\" ),\n ActionMapping(\n source_function=\"type\", # source_function is the name of the function in the original function call\n target_function=\"write\", # target_function is the name of the function in the target function call \n parameter_mappings=[\n ParameterMapping(source_name=\"text\", target_name=\"content\")\n # source_name is the name of the parameter in the original function call \n # target_name is the name of the parameter in the target function call \n ],\n description=\"Input text\" \n )\n]\n\nassistant_message = \"I'll interact at those coordinates for you. click(x=0.5, y=0.3) Now I'll input the text. type(text='hello world')\"\n\n# Parse function calls\nparsed_function_calls = parse_function_call(text)\n\n# Initialize converter\nconverter = ActionSpaceConverter(mappings)\n\n# Convert actions\nconverted_actions = converter.convert_actions(parsed_function_calls)\nfor new_function_call, old_function_call in zip(converted_actions, parsed_function_calls):\n text = text.replace(old_function_call.to_string(), new_function_call.to_string())\n\nprint(text)\n# Output: I'll interact at those coordinates for you. touch(x_coord=0.5, y_coord=0.3) Now I'll input the text. write(content='hello world')\n```\n\nThis tool enables researchers and practitioners to:\n\n- **Customize Training Data**: Adapt the dataset to match their specific action vocabulary requirements\n- **Domain Adaptation**: Transform actions for different platforms (mobile vs. desktop vs. web)\n- **Framework Integration**: Easily align training data with existing automation frameworks\n- **Rapid Experimentation**: Quickly test different action space configurations\n- **Release Preparation**: Standardize action spaces for production deployment with consistent naming conventions\n\nThe Action Space Converter is particularly valuable for preparing datasets for training, as it ensures consistent action vocabularies across different deployment environments while maintaining compatibility with existing automation frameworks.\n\n### Transformed and Released Datasets\n\nThrough this pipeline, we transform the open-source datasets [xlangai/aguvis-stage1](https://huggingface.co/datasets/xlangai/aguvis-stage1), [xlangai/aguvis-stage2](https://huggingface.co/datasets/xlangai/aguvis-stage2) into our unified action space (see [here](https://www.notion.so/Smol2Operator-Post-Training-GUI-Agents-for-Computer-Use-Draft-Blog-Post-2701384ebcac8035bbaad69b5b32ed99?pvs=21)). The output of this process is released as two new fully formatted datasets: [smolagents/aguvis-stage-1](https://huggingface.co/datasets/smolagents/aguvis-stage-1) and [smolagents/aguvis-stage-2](https://huggingface.co/datasets/smolagents/aguvis-stage-2).\n\n## 2. Phase 1: From Zero to Perception\n\n### Training Data\n\nPhase 1 leverages the [smolagents/aguvis-stage-1](https://huggingface.co/datasets/smolagents/aguvis-stage-1) dataset, which introduces **GUI grounding** by pairing low-level instructions with diverse executable actions (expressed in code form). For example, a user/assistant turn in [smolagents/aguvis-stage-1](https://huggingface.co/datasets/smolagents/aguvis-stage-1) follows the structure:\n\n```json\n{\n \"user\": \"click on more button\",\n \"assistant\": \"click(x=0.8875, y=0.2281)\",\n}\n```\n\nEach sample links a screenshot with multi-turn user/assistant interactions, enabling the model to learn fine-grained action grounding across dialogue turns. During fine-tuning, the data collator masks everything except the assistant’s answers when computing the loss.\n\n### Optimization Experiments\n\nBefore proceeding with full-scale Phase 1 training, we conducted comprehensive ablation studies to determine optimal training configurations\n\n### Image Resolution and Coordinate System Analysis\n\nWe experimented with different image sizes and coordinate representation systems to identify the optimal configuration for SmolVLM2:\n\n- **Image Sizes Tested**: 384px, 768px, 1152px\n- **Coordinate Systems**: Pixel coordinates vs. normalized coordinates (0-1 range)\n- **Training Data**: 400K samples from Aguvis datasets\n\n> Some SOTA GUI VLMs (e.g., Qwen-VL) appear also to use a different normalized range (0–1000), which was not tested in this experiment.\n\n| Configuration (coords / image size) | Screenspot-v2 (%) |\n| ----------------------------------- | ----------------- |\n| *Normalized coordinates* | |\n| Base / – | 0.47 |\n| 384 | 31.28 |\n| 764 | 32.32 |\n| 1152 | **33.72** |\n| *Pixel coordinates* | |\n| Base / – | 0.55 |\n| 384 | 1.17 |\n| 764 | 2.67 |\n| 1152 | 4.32 |\n\n**Table 1:** *Baseline on HuggingFaceTB/SmolVLM2-2.2B-Instruct (400k samples, aguvis-stage-1). Higher is better.*\n\n*As demonstrated in our benchmark results, SmolVLM2-2.2B-Instruct base initially achieved 0% performance on perception benchmarks like ScreenSpot-v2. This complete lack of grounding capability provided us with a clean slate to evaluate the effectiveness of our training methodology.*\n\n### Key Findings\n\nFrom our experiments, we determined that:\n\n- **Image Size**: 1152px\n- **Coordinate System**: Normalized coordinates (0-1 range) proved most effective for SmolVLM2\n- Note: The optimal choice between pixel and normalized coordinates may vary depending on the base model’s pre-training approach\n\n### Phase 1 Results\n\nUsing the optimal configuration (1152px resolution with normalized coordinates), we trained for 2 epochs on the smolagents/aguvis-stage-1 dataset. The results were remarkable, **+41% improvement over baseline on ScreenSpot-v2**\n\nThis dramatic improvement demonstrates that our Phase 1 training successfully instilled fundamental grounding capabilities in the model, enabling it to understand and locate visual elements within screenshots.\n\n| Configuration (coords / image size) | Screenspot-v2 (%) |\n| ----------------------------------- | ----------------- |\n| Normalized coordinates / 1152 | 41.27 |\n\n**Table 2:** *Baseline on HuggingFaceTB/SmolVLM2-2.2B-Instruct (2 epochs, aguvis-stage-1).*\n\n## 3. Phase 2: From Perception to Cognition\n\nWhereas Phase 1 provided grounding capabilities, Phase 2 targets **agentic reasoning,** the ability to deliberate and plan before acting. This stage transforms the model from a reactive system identifying GUI elements into a proactive agent capable of executing complex, multi-step interactions.\n\n### Training Data\n\nPhase 2 uses the [smolagents/aguvis-stage-2](https://huggingface.co/datasets/smolagents/aguvis-stage-2) dataset, which introduces agentic scenarios:\n\n- **Explicit reasoning** about upcoming actions\n\n- **Context consistency** across multiple interaction steps\n\n- **High-level instructions** require multi-step, low-level actions.\n\nFor example, the [smolagents/aguvis-stage-2](https://huggingface.co/datasets/smolagents/aguvis-stage-2) chat message is like this:\n\n```json\n{\n \"system\": \"You are a helpful GUI agent. ...\",\n \"user\": \"Please generate the next move according to the UI screenshot, instruction and previous actions.\\n\\nInstruction: What information does the site provide about Judith Lauand's career, works and exhibitions?\\n\\nPrevious actions:\\nNone\",\n \"assistant\": \"\\nclick(x=0.41, y=0.178)\\n\",\n}\n```\n\nEach sample links a screenshot with a system/user/assistant turn. During fine-tuning, the data collator masks everything except the assistant’s answers when computing the loss.\n\n### Phase 2 Results\n\nStarting from the Phase 1 checkpoint (1152 px resolution, normalized coordinates), we fine-tuned the model for two epochs on [smolagents/aguvis-stage-2](https://huggingface.co/datasets/smolagents/aguvis-stage-2). The accuracy on **ScreenSpot-v2 increased from 41% to 61%**, indicating that explicit reasoning improves GUI grounding performance.\n\n| Configuration (coords / image size) | Screenspot-v2 (%) |\n| ----------------------------------- | ----------------- |\n| Normalized coordinates / 1152 | 61.71 |\n\n**Table 2:** *Baseline on HuggingFaceTB/SmolVLM2-2.2B-Instruct after Phase 1 finetuning (2 epochs, aguvis-stage-1).*\n\n💡 *We also reproduced the two-phase training on a much smaller VLM (nanoVLM-460M). Despite its reduced capacity, the model achieved **\\~58% on ScreenSpot-v2**, demonstrating that the training strategy scales down effectively, **making it SOTA on ScreenSpot-v2 for this model size (460M parameters)**. In addition, aguvis-stage-1 is already included in [FineVision Dataset](https://huggingface.co/datasets/HuggingFaceM4/FineVision)!*\n\n## 4. All you need is Open Source\n\nAll training code, data processing pipelines, datasets and model are open-source!\n\n1. **Training Recipe** ([`recipe.ipynb`](https://github.com/huggingface/smol2operator/blob/main/recipe.ipynb)): Complete training pipeline for both Phase 1 and Phase 2, including dataset mixture configurations and training orchestration. We leverage the [TRL](https://huggingface.co/docs/trl/en/index) library to train our models.\n2. **Datasets** ([`smolagents/aguvis-stage-1`](https://huggingface.co/datasets/smolagents/aguvis-stage-1), [`smolagents/aguvis-stage-2`](https://huggingface.co/datasets/smolagents/aguvis-stage-2)): all datasets used are open-source.\n3. **Model** ([`smolagents/SmolVLM2-2.2B-Instruct-Agentic-GUI`](https://huggingface.co/smolagents/SmolVLM2-2.2B-Instruct-Agentic-GUI)): the model produced by applying the training recipe described above.\n4. **Preprocessing Tools:**\n - **Function Parser** ([`utils/function_parser.py`](https://github.com/huggingface/smol2operator/blob/main/utils/function_parser.py)): Utilities for parsing, normalizing, and reconstructing function calls from diverse dataset formats. Supports complex parameter structures, positional arguments, and multiple function call extraction.\n - **Action Conversion System** ([`preprocessing/action_conversion.py`](https://github.com/huggingface/smol2operator/blob/main/preprocessing/action_conversion.py)): Core unification engine transforming mobile and PyAutoGUI desktop actions into a standardized API format. Features smart coordinate handling, direction detection for scroll actions, and comprehensive parameter normalization.\n - **Action Space Converter** ([`utils/action_space_converter.py`](https://github.com/huggingface/smol2operator/blob/main/utils/action_space_converter.py)): Flexible tool for adapting the unified action space to custom vocabularies and naming conventions. Enables domain-specific customization through configurable parameter mappings.\n\n💡 We’ve also released a Space to try the model’s agentic grounding capabilities: [A-Mahla/Smol2Operator](https://huggingface.co/spaces/A-Mahla/Smol2Operator)\n\n## 5. Conclusion\n\nOur experiments demonstrate that high-quality, reasoning-oriented data can substantially improve GUI grounding, even for small VLMs, using only supervised fine-tuning (SFT). Beyond raw performance gains, these results show that the GUI grounding capabilities are largely determined by the quality of the data. Carefully curated datasets teach models the structure and semantics of user interfaces, providing the grounding needed for accurate action prediction.\n\nTo support the development of GUI agents, we’re open-sourcing everything: our complete pipeline, datasets, and trained model. You can reproduce our results, experiment with different models and architectures, or adapt our approach to new domains. The future of agentic AI depends on researchers like you pushing these boundaries further!\n\n## What's Next?\n\nWhile SFT excels at supervised tasks, emerging methods such as Reinforcement Learning (RL) or Direct Preference Optimization (DPO) help develop stronger reasoning capabilities and enable real-time adaptation. These advances point toward a new generation of GUI agents that learn and improve through interaction rather than relying solely on static datasets.\n\nLet’s build the future of GUI agents together 🤗\n",
"dateCreated": "2026-10-04T12:50:30.804479567+00:00",
"date_created": "2025-09-23T00:00:00+00:00",
"date_modified": "2025-09-23T00:00:00+00:00",
"date_published": "2025-09-23T00:00:00+00:00",
"description": "TL;DR: This work shows how a lightweight vision–language model can acquire GUI-grounded skills and evolve into an agentic GUI coder. We release all training recipes, data-processing tools, resulting model, demo and datasets to enable full…",
"external_url": "https://huggingface.co/blog/smol2operator",
"git": {
"commit": "ebb0941c5c06a2ab2377342d607535262a3e969e",
"path": "items/huggingface-co/2025/09/2025-09-23-smol2operator-post-training-gui-agents-for-computer-use.md",
"permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2025/09/2025-09-23-smol2operator-post-training-gui-agents-for-computer-use.md"
},
"id": "urn:aggr:item:0c229480e01259c2210ed1cc6575f5b23c770c5d",
"isBasedOn": {
"@id": "https://huggingface.co/blog/smol2operator"
},
"mainEntity": {
"@id": "https://huggingface.co/blog/smol2operator",
"@type": "CreativeWork",
"archivedAt": {
"@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-09-23-smol2operator-post-training-gui-agents-for-computer-use/#webpage"
},
"dateModified": "2025-09-23T00:00:00+00:00",
"datePublished": "2025-09-23T00:00:00+00:00",
"headline": "Smol2Operator: Post-Training GUI Agents for Computer Use",
"timeRequired": "PT11M",
"url": "https://huggingface.co/blog/smol2operator",
"wordCount": 2451
},
"name": "Archived snapshot: Smol2Operator: Post-Training GUI Agents for Computer Use",
"preview": null,
"reading_minutes": 11,
"replicated_at": null,
"source": "huggingface-co",
"source_name": "Hugging Face - Blog",
"summary": null,
"tags": [],
"temporalCoverage": "2026-10-04T12:50:30.804479567+00:00",
"title": "Smol2Operator: Post-Training GUI Agents for Computer Use",
"url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-09-23-smol2operator-post-training-gui-agents-for-computer-use/",
"word_count": 2451
}