{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-02-13-multimodal-function-calling-with-gemini-3-and-interactions/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "Building AI agents that can truly \"see\" requires more than just passing images in prompts. Coding agents need to read screenshots of UIs they're building. Computer use agents need to capture and analyze browser screenshots. Research agents need to process charts and diagrams from documents.\n\nMultimodal function calling allows tools to return images the model can process natively, similar to how you pass images in prompts. Instead of describing what's in a file, your tool returns the actual image and Gemini 3 processes it natively.\n\nThis guide shows you how Multimodal Function Calling works using the [Interactions API](https://ai.google.dev/gemini-api/docs/interactions). The Interactions API is a unified interface for interacting with Gemini models and agents. It simplifies state management and tool orchestration, making it easy to build multi-turn agentic workflows.\n\n![chart](https://www.philschmid.de/static/blog/interactions-multimodal-fc/chart.png)\n\nThe key difference from standard function calling is in the `function_result` returns the actual image data. Gemini 3 processes this content natively—describing images, analyzing documents, or using visual information to make decisions.\n\nJSON\n\n```json\n{\n \"type\": \"function_result\",\n \"call_id\": \"abc123\",\n \"name\": \"read_image\",\n \"result\": [\n {\"type\": \"text\", \"text\": \"Additional context...\"},\n {\"type\": \"image\", \"data\": \"\", \"mime_type\": \"image/png\"}\n ]\n}\n```\n\n## Example: Reading images from Fileystem\n\nLet's build a tool that reads images from disk and returns them to the model for description. Get a [Gemini API key](https://aistudio.google.com/apikey) and install the Google GenAI SDK:\n\nBash\n\n```bash\npip install google-genai\n```\n\nSet your API key:\n\nBash\n\n```bash\nexport GEMINI_API_KEY=\"your-api-key\"\n```\n\n### 1\\. Define the tool\n\nPython\n\n```python\nread_image = {\n \"type\": \"function\",\n \"name\": \"read_image\",\n \"description\": \"Reads an image file from disk and returns its contents.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"path\": {\n \"type\": \"string\", \n \"description\": \"The path to the image file to read\"\n }\n },\n \"required\": [\"path\"]\n }\n}\n```\n\n### 2\\. Implement the tool execution\n\nPython\n\n```python\nimport base64\nfrom pathlib import Path\n \ndef execute_read_image(path: str) -> list[dict]:\n \"\"\"Read image from disk and return as function result content.\"\"\"\n image_path = Path(path)\n \n if not image_path.exists():\n return [{\"type\": \"text\", \"text\": f\"Error: Image not found at {path}\"}]\n \n mime_types = {\n \".jpg\": \"image/jpeg\",\n \".jpeg\": \"image/jpeg\",\n \".png\": \"image/png\",\n \".webp\": \"image/webp\",\n }\n mime_type = mime_types.get(image_path.suffix.lower(), \"image/png\")\n \n image_data = base64.b64encode(image_path.read_bytes()).decode()\n return [{\"type\": \"image\", \"data\": image_data, \"mime_type\": mime_type}]\n```\n\n### 3\\. Create the agentic loop\n\nPython\n\n```python\nfrom google import genai\n \nclient = genai.Client()\n \nimage_path = \"path/to/your/image.png\"\nprompt=f\"Use the read_image tool to read '{image_path}' and describe what you see.\"\n \n# Step 1: Send the initial request with our tool\ninteraction = client.interactions.create(\n model=\"gemini-3-flash-preview\",\n input=prompt,\n tools=[read_image],\n)\n \n# Step 2: Handle the function call\nfor output in interaction.outputs:\n if output.type == \"function_call\" and output.name == \"read_image\":\n \n image_result = execute_read_image(output.arguments.get(\"path\"))\n \n # Step 3: Send the image back to the model\n interaction = client.interactions.create(\n model=\"gemini-3-flash-preview\",\n input=[{\n \"type\": \"function_result\",\n \"call_id\": output.id,\n \"name\": output.name,\n \"result\": image_result, # Contains the image!\n }],\n tools=[read_image],\n previous_interaction_id=interaction.id,\n )\n \n # Step 4: Get the model's description\n for out in interaction.outputs:\n if out.type == \"text\":\n print(out.text)\n```\n\n## Concluding how it works\n\nThe interaction flow has four steps:\n\n1. **User request** → You send a prompt asking the model to use the tool\n2. **Function call** → Model responds with a `function_call` specifying which tool to use and arguments\n3. **Tool execution** → You execute the tool and return the result (including the image)\n4. **Model response** → Model processes the image and generates a text description\n\nPython\n\n```python\n# The function_call output from the model looks like:\n{\n \"type\": \"function_call\",\n \"id\": \"call_abc123\",\n \"name\": \"read_image\",\n \"arguments\": {\"path\": \"photo.jpg\"}\n}\n \n# Your function_result response looks like:\n{\n \"type\": \"function_result\",\n \"call_id\": \"call_abc123\",\n \"name\": \"read_image\",\n \"result\": [\n {\"type\": \"image\", \"data\": \"iVBORw0KGgo...\", \"mime_type\": \"image/png\"}\n ]\n}\n```\n\n## Conclusion\n\nMultimodal function calling unlocks multimodal agentic use cases. By returning images directly in your `function_result`, you give Gemini 3 the ability to see what your tools see.\n\nThis pattern extends beyond file reading. You can build tools that capture screenshots, fetch images from APIs, render charts, or process scanned documents. Combined with the Interactions API's stateful sessions, you have everything you need to build sophisticated visual agents.\n\n**Next steps:**\n\n- Try the [Interactions API quickstart](https://ai.google.dev/gemini-api/docs/interactions) notebook\n- Explore [Computer Use](https://ai.google.dev/gemini-api/docs/computer-use) for browser automation with visual feedback\n- Build tools that return PDFs or multiple images for complex document workflows\n\n* * *\n\n**This API is in Beta, and we want your feedback!** We're actively listening to developers to shape the future of this API. What features would help your agent workflows? What pain points are you experiencing? Please let me know on [Twitter](https://twitter.com/_philschmid) or [LinkedIn](https://www.linkedin.com/in/philipp-schmid-a6a2bb196/).\n", "authors": [], "captured_at": "2026-09-27T19:29:15.293681927+00:00", "category": "programming", "content_html": "

Building AI agents that can truly "see" requires more than just passing images in prompts. Coding agents need to read screenshots of UIs they're building. Computer use agents need to capture and analyze browser screenshots. Research agents need to process charts and diagrams from documents.

\n

Multimodal function calling allows tools to return images the model can process natively, similar to how you pass images in prompts. Instead of describing what's in a file, your tool returns the actual image and Gemini 3 processes it natively.

\n

This guide shows you how Multimodal Function Calling works using the Interactions API. The Interactions API is a unified interface for interacting with Gemini models and agents. It simplifies state management and tool orchestration, making it easy to build multi-turn agentic workflows.

\n

\"chart\"

\n

The key difference from standard function calling is in the function_result returns the actual image data. Gemini 3 processes this content natively—describing images, analyzing documents, or using visual information to make decisions.

\n

JSON

\n
{\n  "type": "function_result",\n  "call_id": "abc123",\n  "name": "read_image",\n  "result": [\n    {"type": "text", "text": "Additional context..."},\n    {"type": "image", "data": "<base64>", "mime_type": "image/png"}\n  ]\n}\n
\n

Example: Reading images from Fileystem

\n

Let's build a tool that reads images from disk and returns them to the model for description. Get a Gemini API key and install the Google GenAI SDK:

\n

Bash

\n
pip install google-genai\n
\n

Set your API key:

\n

Bash

\n
export GEMINI_API_KEY="your-api-key"\n
\n

1. Define the tool

\n

Python

\n
read_image = {\n    "type": "function",\n    "name": "read_image",\n    "description": "Reads an image file from disk and returns its contents.",\n    "parameters": {\n        "type": "object",\n        "properties": {\n            "path": {\n                "type": "string", \n                "description": "The path to the image file to read"\n            }\n        },\n        "required": ["path"]\n    }\n}\n
\n

2. Implement the tool execution

\n

Python

\n
import base64\nfrom pathlib import Path\n \ndef execute_read_image(path: str) -> list[dict]:\n    """Read image from disk and return as function result content."""\n    image_path = Path(path)\n    \n    if not image_path.exists():\n        return [{"type": "text", "text": f"Error: Image not found at {path}"}]\n    \n    mime_types = {\n        ".jpg": "image/jpeg",\n        ".jpeg": "image/jpeg",\n        ".png": "image/png",\n        ".webp": "image/webp",\n    }\n    mime_type = mime_types.get(image_path.suffix.lower(), "image/png")\n    \n    image_data = base64.b64encode(image_path.read_bytes()).decode()\n    return [{"type": "image", "data": image_data, "mime_type": mime_type}]\n
\n

3. Create the agentic loop

\n

Python

\n
from google import genai\n \nclient = genai.Client()\n \nimage_path = "path/to/your/image.png"\nprompt=f"Use the read_image tool to read '{image_path}' and describe what you see."\n \n# Step 1: Send the initial request with our tool\ninteraction = client.interactions.create(\n    model="gemini-3-flash-preview",\n    input=prompt,\n    tools=[read_image],\n)\n \n# Step 2: Handle the function call\nfor output in interaction.outputs:\n    if output.type == "function_call" and output.name == "read_image":\n      \n        image_result = execute_read_image(output.arguments.get("path"))\n        \n        # Step 3: Send the image back to the model\n        interaction = client.interactions.create(\n            model="gemini-3-flash-preview",\n            input=[{\n                "type": "function_result",\n                "call_id": output.id,\n                "name": output.name,\n                "result": image_result,  # Contains the image!\n            }],\n            tools=[read_image],\n            previous_interaction_id=interaction.id,\n        )\n        \n        # Step 4: Get the model's description\n        for out in interaction.outputs:\n            if out.type == "text":\n                print(out.text)\n
\n

Concluding how it works

\n

The interaction flow has four steps:

\n
    \n
  1. User request → You send a prompt asking the model to use the tool
    2. \n
  2. Function call → Model responds with a function_call specifying which tool to use and arguments
    3. \n
  3. Tool execution → You execute the tool and return the result (including the image)
    4. \n
  4. Model response → Model processes the image and generates a text description
    5. \n
\n

Python

\n
# The function_call output from the model looks like:\n{\n    "type": "function_call",\n    "id": "call_abc123",\n    "name": "read_image",\n    "arguments": {"path": "photo.jpg"}\n}\n \n# Your function_result response looks like:\n{\n    "type": "function_result",\n    "call_id": "call_abc123",\n    "name": "read_image",\n    "result": [\n        {"type": "image", "data": "iVBORw0KGgo...", "mime_type": "image/png"}\n    ]\n}\n
\n

Conclusion

\n

Multimodal function calling unlocks multimodal agentic use cases. By returning images directly in your function_result, you give Gemini 3 the ability to see what your tools see.

\n

This pattern extends beyond file reading. You can build tools that capture screenshots, fetch images from APIs, render charts, or process scanned documents. Combined with the Interactions API's stateful sessions, you have everything you need to build sophisticated visual agents.

\n

Next steps:

\n\n
\n

This API is in Beta, and we want your feedback! We're actively listening to developers to shape the future of this API. What features would help your agent workflows? What pain points are you experiencing? Please let me know on Twitter or LinkedIn.

\n", "content_markdown": "Building AI agents that can truly \"see\" requires more than just passing images in prompts. Coding agents need to read screenshots of UIs they're building. Computer use agents need to capture and analyze browser screenshots. Research agents need to process charts and diagrams from documents.\n\nMultimodal function calling allows tools to return images the model can process natively, similar to how you pass images in prompts. Instead of describing what's in a file, your tool returns the actual image and Gemini 3 processes it natively.\n\nThis guide shows you how Multimodal Function Calling works using the [Interactions API](https://ai.google.dev/gemini-api/docs/interactions). The Interactions API is a unified interface for interacting with Gemini models and agents. It simplifies state management and tool orchestration, making it easy to build multi-turn agentic workflows.\n\n![chart](https://www.philschmid.de/static/blog/interactions-multimodal-fc/chart.png)\n\nThe key difference from standard function calling is in the `function_result` returns the actual image data. Gemini 3 processes this content natively—describing images, analyzing documents, or using visual information to make decisions.\n\nJSON\n\n```json\n{\n \"type\": \"function_result\",\n \"call_id\": \"abc123\",\n \"name\": \"read_image\",\n \"result\": [\n {\"type\": \"text\", \"text\": \"Additional context...\"},\n {\"type\": \"image\", \"data\": \"\", \"mime_type\": \"image/png\"}\n ]\n}\n```\n\n## Example: Reading images from Fileystem\n\nLet's build a tool that reads images from disk and returns them to the model for description. Get a [Gemini API key](https://aistudio.google.com/apikey) and install the Google GenAI SDK:\n\nBash\n\n```bash\npip install google-genai\n```\n\nSet your API key:\n\nBash\n\n```bash\nexport GEMINI_API_KEY=\"your-api-key\"\n```\n\n### 1\\. Define the tool\n\nPython\n\n```python\nread_image = {\n \"type\": \"function\",\n \"name\": \"read_image\",\n \"description\": \"Reads an image file from disk and returns its contents.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"path\": {\n \"type\": \"string\", \n \"description\": \"The path to the image file to read\"\n }\n },\n \"required\": [\"path\"]\n }\n}\n```\n\n### 2\\. Implement the tool execution\n\nPython\n\n```python\nimport base64\nfrom pathlib import Path\n \ndef execute_read_image(path: str) -> list[dict]:\n \"\"\"Read image from disk and return as function result content.\"\"\"\n image_path = Path(path)\n \n if not image_path.exists():\n return [{\"type\": \"text\", \"text\": f\"Error: Image not found at {path}\"}]\n \n mime_types = {\n \".jpg\": \"image/jpeg\",\n \".jpeg\": \"image/jpeg\",\n \".png\": \"image/png\",\n \".webp\": \"image/webp\",\n }\n mime_type = mime_types.get(image_path.suffix.lower(), \"image/png\")\n \n image_data = base64.b64encode(image_path.read_bytes()).decode()\n return [{\"type\": \"image\", \"data\": image_data, \"mime_type\": mime_type}]\n```\n\n### 3\\. Create the agentic loop\n\nPython\n\n```python\nfrom google import genai\n \nclient = genai.Client()\n \nimage_path = \"path/to/your/image.png\"\nprompt=f\"Use the read_image tool to read '{image_path}' and describe what you see.\"\n \n# Step 1: Send the initial request with our tool\ninteraction = client.interactions.create(\n model=\"gemini-3-flash-preview\",\n input=prompt,\n tools=[read_image],\n)\n \n# Step 2: Handle the function call\nfor output in interaction.outputs:\n if output.type == \"function_call\" and output.name == \"read_image\":\n \n image_result = execute_read_image(output.arguments.get(\"path\"))\n \n # Step 3: Send the image back to the model\n interaction = client.interactions.create(\n model=\"gemini-3-flash-preview\",\n input=[{\n \"type\": \"function_result\",\n \"call_id\": output.id,\n \"name\": output.name,\n \"result\": image_result, # Contains the image!\n }],\n tools=[read_image],\n previous_interaction_id=interaction.id,\n )\n \n # Step 4: Get the model's description\n for out in interaction.outputs:\n if out.type == \"text\":\n print(out.text)\n```\n\n## Concluding how it works\n\nThe interaction flow has four steps:\n\n1. **User request** → You send a prompt asking the model to use the tool\n2. **Function call** → Model responds with a `function_call` specifying which tool to use and arguments\n3. **Tool execution** → You execute the tool and return the result (including the image)\n4. **Model response** → Model processes the image and generates a text description\n\nPython\n\n```python\n# The function_call output from the model looks like:\n{\n \"type\": \"function_call\",\n \"id\": \"call_abc123\",\n \"name\": \"read_image\",\n \"arguments\": {\"path\": \"photo.jpg\"}\n}\n \n# Your function_result response looks like:\n{\n \"type\": \"function_result\",\n \"call_id\": \"call_abc123\",\n \"name\": \"read_image\",\n \"result\": [\n {\"type\": \"image\", \"data\": \"iVBORw0KGgo...\", \"mime_type\": \"image/png\"}\n ]\n}\n```\n\n## Conclusion\n\nMultimodal function calling unlocks multimodal agentic use cases. By returning images directly in your `function_result`, you give Gemini 3 the ability to see what your tools see.\n\nThis pattern extends beyond file reading. You can build tools that capture screenshots, fetch images from APIs, render charts, or process scanned documents. Combined with the Interactions API's stateful sessions, you have everything you need to build sophisticated visual agents.\n\n**Next steps:**\n\n- Try the [Interactions API quickstart](https://ai.google.dev/gemini-api/docs/interactions) notebook\n- Explore [Computer Use](https://ai.google.dev/gemini-api/docs/computer-use) for browser automation with visual feedback\n- Build tools that return PDFs or multiple images for complex document workflows\n\n* * *\n\n**This API is in Beta, and we want your feedback!** We're actively listening to developers to shape the future of this API. What features would help your agent workflows? What pain points are you experiencing? Please let me know on [Twitter](https://twitter.com/_philschmid) or [LinkedIn](https://www.linkedin.com/in/philipp-schmid-a6a2bb196/).\n", "dateCreated": "2026-09-27T19:29:15.293681927+00:00", "date_created": "2026-02-13T00:00:00+00:00", "date_modified": "2026-02-13T00:00:00+00:00", "date_published": "2026-02-13T00:00:00+00:00", "description": "Multimodal function calling allows tools to return images the model can process natively, similar to how you pass images in prompts. Instead of describing what's in a file, your tool returns the actual image and Gemini 3 processes it…", "external_url": "https://www.philschmid.de/interactions-multimodal-fc", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/philschmid-de/2026/02/2026-02-13-multimodal-function-calling-with-gemini-3-and-interactions.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/philschmid-de/2026/02/2026-02-13-multimodal-function-calling-with-gemini-3-and-interactions.md" }, "id": "urn:aggr:item:09cdde15981da94925849a65f6271f407ffc4cbd", "isBasedOn": { "@id": "https://www.philschmid.de/interactions-multimodal-fc" }, "mainEntity": { "@id": "https://www.philschmid.de/interactions-multimodal-fc", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-02-13-multimodal-function-calling-with-gemini-3-and-interactions/#webpage" }, "dateModified": "2026-02-13T00:00:00+00:00", "datePublished": "2026-02-13T00:00:00+00:00", "headline": "Multimodal Function Calling with Gemini 3 and Interactions API", "timeRequired": "PT4M", "url": "https://www.philschmid.de/interactions-multimodal-fc", "wordCount": 699 }, "name": "Archived snapshot: Multimodal Function Calling with Gemini 3 and Interactions API", "preview": null, "reading_minutes": 4, "replicated_at": null, "source": "philschmid-de", "source_name": "philschmid.de - RSS feed", "summary": "Multimodal function calling allows tools to return images the model can process natively, similar to how you pass images in prompts. Instead of describing what's in a file, your tool returns the actual image and Gemini 3 processes it natively.", "tags": [], "temporalCoverage": "2026-09-27T19:29:15.293681927+00:00", "title": "Multimodal Function Calling with Gemini 3 and Interactions API", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-02-13-multimodal-function-calling-with-gemini-3-and-interactions/", "word_count": 699 }