![\"Credentials](\"https://www.philschmid.de/static/blog/managed-agents-credentials/credentials_flow.png\")
![\"\"](\"https://www.philschmid.de/static/blog/managed-agents-credentials/credentials_flow_dark.png\")
We recently shipped the Credentials API for Managed Agents (antigravity-preview-09-2026). It lets your agent authenticate with external services like GitHub, Notion, or the Gemini API without putting raw secrets inside the Linux sandbox.
If you pass an API key as a normal environment variable, any dependency or script inside the sandbox can read os.environ and leak it. With the Credentials API, the secret stays on the server and is injected on the wire by the egress proxy.
Register a credential with credentials.create(). Secrets are write-only and encrypted at rest; API responses only return metadata (id, type, status, timestamps).
![\"Step](\"https://www.philschmid.de/static/blog/managed-agents-credentials/step1_store_once.png\")
![\"\"](\"https://www.philschmid.de/static/blog/managed-agents-credentials/step1_store_once_dark.png\")
There are 3 credential types:
\nbearer_token: Static tokens and API keys injected as HTTP headers (Authorization: Bearer <token> by default, or custom header_name and prefix).oauth2: Refresh tokens (client_id, client_secret, refresh_token, token_url). Gemini exchanges and refreshes access tokens automatically before expiry.environment_variable: For CLIs and SDKs that read secrets from the process environment. The container receives a placeholder (__GEMINI_CRED_<id>__), and the egress proxy swaps in the real value only for requests to trusted_domains.Pass the credential id when defining a reusable agent with agents.create() (or per run in interactions.create()) on a remote mcp_server tool, a domain in network.allowlist, or a variable in base_environment.env.
![\"Step](\"https://www.philschmid.de/static/blog/managed-agents-credentials/step2_reference_by_id.png\")
![\"\"](\"https://www.philschmid.de/static/blog/managed-agents-credentials/step2_reference_by_id_dark.png\")
When the agent calls an MCP tool or makes an outbound HTTP request from the sandbox, the egress proxy checks the target domain and injects the real credential on the wire.
\n![\"Step](\"https://www.philschmid.de/static/blog/managed-agents-credentials/step3_injected_on_wire.png\")
![\"\"](\"https://www.philschmid.de/static/blog/managed-agents-credentials/step3_injected_on_wire_dark.png\")
Here is a practical examples on a reusable custom agent that manages google-gemini/cookbook via the Github MCP server and can make securte Gemini API calls. The GEMINI_API_KEY is bound as a credential, untrusted code in the sandbox can call generativelanguage.googleapis.com without ever seeing the real API key.
![\"Architecture](\"https://www.philschmid.de/static/blog/managed-agents-credentials/credentials_mcp_allowlist_architecture.png\")
![\"\"](\"https://www.philschmid.de/static/blog/managed-agents-credentials/credentials_mcp_allowlist_architecture_dark.png\")
GITHUB_PAT and GEMINI_API_KEY locally.Install the SDK (bun add @google/genai) and run:
TypeScript
\n;\n \nconst ai = ;\n \n// 1. Store credentials once\nconst github = await ai..;\n \nconst geminiKey = await ai..;\n \n// 2. Define a reusable agent wired to both credentials\nconst agent = await ai..;\n \n// 3. Run the agent\nconst interaction = await ai..;\n \n;\nOutput:
\nMarkdown
\n \n \n \nWhat happens during the run:
\ngithub-mcp): Connects to api.githubcopilot.com/mcp/ and calls github:list_pull_requests and github:pull_request_read to inspect the latest merged PRs (#1347, #1383, #1384).gemini-key): Inside the container, $GEMINI_API_KEY is __GEMINI_CRED_gemini-key__. When the smoke test calls generativelanguage.googleapis.com, the egress proxy swaps the placeholder for the real key on the wire.__GEMINI_CRED_gemini-key__ to a domain outside trusted_domains is blocked (403).Update a credential by ID to rotate its secret without changing your agent definition, or delete it when it is no longer needed:
\nTypeScript
\nawait ai..;\n \nawait ai..;\nawait ai..;\nSee the Credentials API docs, Custom Agents guide, and Managed Agents Quickstart for full details.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-09-28T00:00:00+00:00", "date_published": "2026-09-28T00:00:00+00:00", "external_url": "https://www.philschmid.de/managed-agents-credentials", "id": "urn:aggr:item:5248e714ed27126618dfd2cd3e23e7240c4969fa", "summary": "Store secrets once with the Credentials API, reference them by ID, and let the egress proxy inject them on the wire so they never enter the agent sandbox.", "tags": [ "programming" ], "title": "Credentials API for Gemini Managed Agents", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-09-28-credentials-api-for-gemini-managed-agents/" }, { "content_html": "Gemini 3.8 Flash TTS and Flash-Lite TTS are now available, in the Gemini API and AI Studio. Ranking #1 on Hume's Voice Design Benchmark, and on top of Voice Arena in 6 languages.
\nBiggest new features is that you can now replicate your own voice or create a new one from a sentence. The guide below shows how to do both.
\nSame mic, same room, 24kHz mono. The API compares the two recordings, so don't switch from a headset to the laptop mic between them.
\nBash
\n# macOS. ":0" is audio device 0. To find your mic:\n# ffmpeg -f avfoundation -list_devices true -i ""\n \n# me.wav: 15-20s of you talking like you normally do. Explain what\n# you're building this week. Don't read, just talk.\n\n \n# consent.wav: read this word for word:\n# "I am the owner of this voice and I consent to Google using this voice to create a synthetic voice model."\n\nLinux: -f alsa -i default. On macOS, the first run may only open the microphone permission dialog. Allow it, then run the command again.
The consent sentence must be one of the 25 supported versions, read exactly. German, for example: "Ich bin der Eigentümer dieser Stimme und bin damit einverstanden, dass Google diese Stimme zur Erstellung eines synthetischen Stimmmodells verwendet." The reference clip can be in any language. Use the language you want the voice to speak later.
\nPython
\n\n\n \n = \n \n\n return \n \n = \n\n \n = \n \n# 3.8 returns a real WAV with a RIFF header. No wave-module wrapping.\n\nTwo things to notice. The text is spoken word for word. The delivery ("casual, a bit amused") goes in speech_metadata.style, and the short sounds go inline as <short pause> and <laugh>. The voice ID is reusable: pass it in any later request, or find it again with client.voices.list(type_=["replicated"]).
If you don't want anything stored server-side, store=False returns an encrypted voicekey_... that you keep yourself. It works in the same speech_config field and expires after 7 days.
No recordings needed. Same call, different voice dict. You also get a sample_audio preview, so you can listen to the voice before you synthesize anything:
Python
\n = \n\nThen use voice.id exactly as above.
If you're coming from gemini-3.1-flash-tts-preview, this will break your prompts, so read the prompting guide and the migration notes. The short version:
speech_metadata.style. Short sounds go inline in angle brackets.speaker on every turn.style strings.audio/wav), so remove any code that adds a WAV header. Set response_format to audio/l16 if you still want raw PCM.If you have used coding agents, you may have noticed that they started to use Bash for tasks that would have traditionally used dedicated tools such as read, edit, and search. Well, models became superhuman in bash and might consider dedicated tools not good enough.
Every few months, I rebuild an agent harness from scratch to learn what changed and what I can delete. This time I removed every model-facing tool except for a bash and a media viewer.
The outcome matched what I had already seen in daily use. Task completion stayed in the same range, while the agent moved through complex work and verifications with fewer tool boundaries.
\nFrontier models have become superhuman in Bash. I mean that in a narrow sense: they synthesize disposable command-line programs, in seconds, that most developers would need much longer to assemble and verify.
\nYou probably know the individual pieces git, rg, jq, Python, temporary files, process substitution, and test runners. Writing a one-off 40-line workflow that combines them under time pressure is different. Most of us work through the problem interactively, checking syntax and state between steps.
Coding agents assemble these workflows from patterns learned across POSIX utilities, programming languages, build systems, and public source code. They still make mistakes. The difference is how much correct orchestration they can attempt at once.
\nBash is the routing layer here. The heavy work may happen in Python, Git, SQLite, a compiler, or a project-specific CLI. One shell interface lets the agent compose all of them without forcing the runtime designer to predict every useful operation.
\nThe following are simplified examples from my personal agent traces.
\nYou work on a project and you want to rename a variable across an implementation, its callers, and the tests.
\nBash
\n\n \n\n\n\n\n\n\nThe script first counts every old snippet. If a file has the wrong number of matches, it exits before writing anything. Only then does it apply the four-file rename in one pass, so you never sit in a half-updated tree. After the write it formats, lints, type-checks, runs the focused tests, and prints a compact diff. Failed checks stay in the working tree, so the next turn can inspect and fix them. That verify-before-done cycle is the inner loop.
\nYou have a test that fails only sometimes. You want the first commit that introduced the flake, without moving your current checkout.
\nBash
\n\n \n\nrepo=\nscratch=\n \n\n\n \n\n\n\n \n \nbad=\n\n \n\n\n| \n\n\nThe extra worktree keeps your local files in place. Each commit is classified with five seeds, and it only counts as good if fewer than three runs fail, so one unlucky flake does not poison the bisect. When it lands, the script prints the first bad commit, a truncated diff of the suspected change, and the tail of the log. You get a diagnosis, not every test run in the prompt.
\nYou have compressed production logs that are too large to load into the model. You want the failing endpoints, their error classes, and P95 latency.
\nBash
\n\nPython reads the rotated files in batches and never dumps them into the prompt. SQLite does the join, the P95 ranking, and the error-class aggregation. What comes back is five JSON records: endpoint, failure count, average latency, P95, and error classes. Context offloading keeps the intermediate data in the environment, and puts only the summary in the model.
\nA few months ago, I argued that coding agents should start with robust atomic tools and avoid shell commands such as cat, sed, and echo. A careless command could flood the context window, return an opaque error, or corrupt an edit through bad quoting. Text output also cannot carry visual information into a vision model.
Those limits still apply to an uninstrumented shell. Foundational models are now much better at composing small Python patchers, Git commands, quoted heredocs, and focused tests. The Harness also absorbs the protections that made atomic tools useful:
\nThe one exception multimodal input:
\nText output cannot make a screenshot/image visible. A shell command can render a page, capture a chart, or save a video frame, but the pixels still need to enter the model through a multimodal channel.
\nI compared a shell-centered setup with a configuration that exposed separate tools for file reading, writing, editing, and search. Both ran against the same coding task set under the same conditions. The shell-centered setup achieved on par or better performance.
\nThat is the Bitter Lesson applied to harness design: general methods that scale with computation beat hand-built shortcuts. Bash is that general computation layer.
\nA good agent should have more tools when they provide a better interface to a capability. If the model can do something better with a tool, it should use the tool. Browser control is a good example.
\nA browser tool can navigate, click, type, wait for the page to settle, and return a screenshot in the same call. Doing this through Bash means invoking a CLI, locating the captured artifact, and passing it through view_media in a second step. Service integrations can benefit in a similar way, or stay on the shell with something like mcp-cli so the schemas never sit in the prompt.
Here is what to try:
\nOur goal is a smaller interface with a larger action space.
\n", "date_modified": "2026-09-01T00:00:00+00:00", "date_published": "2026-09-01T00:00:00+00:00", "external_url": "https://www.philschmid.de/superhuman-bash", "id": "urn:aggr:item:ceb1fdcf781efc014d65097dffa3824f73704076", "summary": "Frontier coding agents can compose shell workflows that replace entire catalogs of file tools. Here is what changes when Bash becomes the primary execution interface.", "tags": [ "programming" ], "title": "How Foundational Models Became Superhuman in Bash", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-09-01-how-foundational-models-became-superhuman-in-bash/" }, { "content_html": "Five commands, zero fluff: install OpenClaw 2.0 (v2026.8.1), link Gemini 3.8 Flash, our fastest (up to +300 token/sec) agentic model for coding and agents, and start chatting with live Google search grounding in under a minute.
OpenClaw is a personal AI assistant that actually does things. It runs on your devices, in your chats, with your rules.
\nOpenClaw 2.0 (v2026.8.1) is the largest update yet: simpler install, a rebuilt browser app that opens straight into a conversation, and setup you can finish by talking to your Claw. Full notes: v2026.8.1.
Node 26 (recommended) or Node 22.22.3+ / 24.15+ (node --version).Install OpenClaw globally via npm:
\nBash
\n\nVerify your installation:
\nBash
\n\n# Output: OpenClaw 2026.8.1 (ea80657)\nExport your Gemini API key in your current shell:
\nBash
\n\nTo persist it into OpenClaw's local profile store:
\nBash
\n\n(Note: You can safely ignore any onboarding tip about configuring a Brave API key—Gemini's built-in Google Search grounding is used automatically.)
\nSet Gemini 3.8 Flash as the default agent model:
\nBash
\n\n(Note: If OpenClaw displays a warning that the model is not in the local catalog, it is safe to ignore. The model is dynamically resolved and validated against Google AI Studio in the next step.)
\nVerify model discovery and authentication status:
\nBash
\n\nRun a standalone agent turn in your terminal:
\nBash
\n\nOutput:
\nI am currently running on google/gemini-3.8-flash.\nWhat would you like to call me?\n![\"OpenClaw](\"https://www.philschmid.de/static/blog/openclaw-v2-gemini/cli-setup.png\")
Set the gateway to local mode and start it:
\nBash
\n\n\n(To run as a persistent background daemon instead: openclaw gateway install && openclaw gateway start. To stop it: openclaw gateway stop --force)
Open the Control UI in your browser:
\nBash
\n\nThe browser will open to http://127.0.0.1:18789/ authenticated with your local session token:
![\"OpenClaw](\"https://www.philschmid.de/static/blog/openclaw-v2-gemini/openclaw-control-ui.png\")
Google Search grounding is enabled by default when GEMINI_API_KEY is present. No extra configuration or search keys are required.
Test it directly via CLI:
\nBash
\n\nIn chat sessions, Gemini autonomously runs web searches with citations whenever fresh context is needed.
\n![\"Google](\"https://www.philschmid.de/static/blog/openclaw-v2-gemini/openclaw-websearch.png\")
openclaw models set-image google/gemini-3.1-flash-image.google/veo-3.1-fast-generate-preview.google/lyria-3-clip-preview.gemini-3.1-flash-tts-preview).gemini-3.1-flash-live-preview).Full Google Documentation.
\nConnect your Gemini assistant to your favorite messaging apps using simple CLI commands:
\nopenclaw channels list --allopenclaw channels addopenclaw channels add --channel telegram --token "<BOT_TOKEN>"openclaw channels login --channel whatsappopenclaw channels add --channel discord --token "<BOT_TOKEN>"openclaw channels status --probeFull Chat Channels Documentation.
\n", "date_modified": "2026-08-31T00:00:00+00:00", "date_published": "2026-08-31T00:00:00+00:00", "external_url": "https://www.philschmid.de/openclaw-v2-gemini", "id": "urn:aggr:item:0c3a3436a573d42e03574d8c01339ee60b78f2c5", "summary": "Five commands, zero fluff: install OpenClaw 2.0, connect Gemini 3.8 Flash, and start chatting with live Google Search grounding from your terminal and web dashboard in under a minute.", "tags": [ "programming" ], "title": "Setup OpenClaw 2.0 with Gemini in Under 60 Seconds", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-08-31-setup-openclaw-2-0-with-gemini-in-under-60-seconds/" }, { "content_html": "Earlier this week I tweeted:
\n\n\nThe next iterations of agent harnesses will center on coded extensions that integrate automatically. Pi lead, others are following, and DeepSeek represents the extreme.
\nAll powered and driven by autoresearch and recursive self-improvements.
\n
For most of AI's history, recursive self-improvement belonged to theory and science fiction. I. J. Good described an "intelligence explosion" in 1965: a machine designs a better machine, which becomes better at designing the next one. The curve goes vertical.
\nWe are surprisingly close to a narrow version of that loop. Agents can inspect failed runs, edit their tools and instructions, test the result, and keep what works. They are also beginning to run experiments that produce better models. The results are scrappy, but the pieces are starting to connect.
\nRetrying a failed task is easy to mistake for learning. I use the following definition:
\n\n\nRecursive self-improvement is a loop in which a system makes a persistent change that improves its future performance and its ability to produce subsequent improvements.
\n
Each loop is defined by which layer persists after the run — and green marks what the loop is allowed to move.
\nIteration improves an output while the system stays the same. An agent edits code, reruns a test, and the next task still starts with the same agent.
\nSelf-improvement changes the system persistently. The agent adds a tool, records a skill, or changes how it compacts long conversations. Future tasks run through a different system. The ruler that says "better" does not move.
\nRecursive self-improvement also raises that ruler. The system gets better at judging, not only at scoring. Later rounds face a harder test that is still independent of the agent.
\nCurrent systems work best when an external evaluator stays fixed. A test passes, validation loss drops, or a benchmark score rises. The system can search because it has a ruler it doesn't control. A recent survey shows that agents can edit a prompt, tool, skill, or piece of harness code, but they can't redefine success.
\nOpen-ended recursive self-improvement asks for more. The system would improve how it finds changes and how those changes are judged. It would strengthen the verifier without being able to game it. Public evidence for that remains thin.
\nA higher task score shows improvement. Recursion needs another measurement: did the next round face a harder bar that the system still could not cheat?
\nModels have long improved through their own outputs, self-play, student-teacher training, or successful reasoning traces. Later work lets a model score its own training preferences or propose synthetic data and update directives. But researchers still set the objective, the update rule, and the tests. The model's output feeds the loop. The loop itself stays fixed.
\nAgents are now improving the code around that loop. Karpathy's autoresearch is the simple version. An agent propose a change, trains for a few minutes, keep it only if validation loss drops. On nanochat it ran about 700 experiments, found about 20 that transferred, and cut time-to-GPT-2 quality from 2.02 to 1.80 hours. Prime Intellect scaled the same loop to 10,000 trials and beat the human baseline.
\nPrime Intellect scaled the same keep-or-revert loop to ~10,000 trials on H200s and beat the human baseline.
\nAlphaEvolve is the same idea aimed at algorithms. It mutates programs, scores them against an automatic evaluator, and keeps the winners. It found a way to multiply 4×4 complex matrices in 48 scalar multiplications, one fewer than Strassen's 49, and sped up a kernel used to train Gemini by 23%. A model improving the code that trains the model is the recursive loop in a single sentence.
\nNewer experiments connect both. SIA updates both a task agent's harness and its weights. Recursive Harness Self-Improvement edits a harness partly to produce better traces for training future models, and calls the result model–harness co-evolution.
\nRecursive self-improvement doesn't require a better model. A frozen model with an honest verifier and a writable environment can climb on its own. The easier target is the harness around it.
\nThe idea is simple. Run a baseline, inspect the failures, change one part, and test again. Cline was able to hill climb Opus 4.5 from 47% to 57% in Terminal Bench manually and a few months later, an agent ran the same method for 17 hours and about $50 of compute. It moved Kimi K3 from 69 of 89 tasks to 79, with better retries, loop detection, and process handling.
Those improvements fall under bounded self-improvement. The agent changed its environment and kept the changes. The report does not show whether the updated system became better at finding its next change.
\nHarnessOpt-Bench asks whether such gains survive a stronger test. An agent edits its harness on development feedback; a separate system tests the candidate on hidden tasks. Across 111 runs, 5 optimizer models, and 4 tasks, results varied sharply by model and task. The benchmark measures whether an edit improves scores. RSI needs a second measure: whether later rounds face a harder bar the system still cannot cheat.
\nA rising score proves self-improvement. Recursion is a claim about the other curve.
\nPAST-Bench asks whether stored experience actually helps later episodes. In many of its scenarios, turning memory on does not. A new RSI Benchmark proposes measuring AI systems doing AI research, which should show whether each round makes the next round faster or better.
\nSo what should we count as recursion? Hyperagents comes close. The system can change an agent as well as the code that creates new agents, and improvements like better memory storage helped across later runs and tasks, but these are still bounded loops. The fitness function stayed outside the editable code.
\nGreedy selection would have pruned the 58. The archive kept it — and it parented the best agent.
\nEarly agents were a system prompt plus a handful of tools. We then added memory, compaction, hooks, skills, and control flow. Each addition made the harness more important and harder to change.
\nThe next harness change will not be a bigger prompt. It is code the model can write, and that the runtime can load without a human rewriting the core. The harness will pick up the change without human intervention.
\nPi was first. It ships 4 tools (read, write, edit, bash) and a system prompt under 1,000 tokens. Everything else is a TypeScript extension, auto-discovered from .pi/extensions/. An agent can write one, reload it, and continue with new tools. Others are following. Amp stores project plugins with the codebase.
DeepSeek is the extreme. DeepSeek Harness is built on the Cordis plugin kernel and treats almost every part as replaceable: models, tools, sessions, sandboxes, and the control loop itself. Plugin side effects unwind on unload, so the runtime can swap parts of itself without dying. Their shorthand is Agent = Model + Harness. An agent gets more code it can improve, and more ways to break compatibility or weaken a permission boundary.
The green meter marks how much of the harness the agent can rewrite.
\nThat shift matters for two reasons. The agent can build capabilities its developers didn't predict. And code can run a sequence of operations without filling the context window with every intermediate result. The same properties make failures persistent.
\nAn agent can overfit visible tasks, exploit bugs in the tests, or change a test so broken code passes. If it can edit evaluation, it can jailbreak itself. Reward hacking is the default behavior of a system asked to raise a number.
\nThe agent and the system that verifies it have to stay apart for the moment. The agent may change prompts, skills, tools, memory, and harness code. A separate entity needs to own the evaluation. That's why are not at recursive self-improvement.
\nRecursive self-improvement needs the system to improve the verifier as well, harder tests, better judges, a higher bar. And it needs to do that without capturing the signal. Today the system does not know how to keep the verifier. Humans still set the objective and hide the real score.
\nA Princeton-led study found that today's agents can execute much of the engineering of AI research while still struggling to choose original and useful directions. The loop is excellent at hill-climbing a verifiable metric. Taste is not a metric. Raising the bar without taste is a faster way to optimize the wrong thing.
\nThese loops already deliver real things, faster training code, better tools, agents that fix their own retry logic overnight. What is still missing is the recursive part. Until the improver can raise the bar and still fail it, we are doing self-improvement. Not recursion.
\nThe near future is local recursion. Not a system redesigning its architecture in the dark. Agents writing the extensions the next session treats as native, and trajectories are used for training when they are good enough. Model–harness co-evolution is the least mystical description of where this is going. It is also the one the evidence supports.
\nVerification is the bottleneck. Every convincing result has a ruler the system does not own. The moment the system owns the ruler, it will get very good at looking improved. Bigger models will not fix that. A system that knows how to raise the bar, and keep it honest, might.
\nTaste still lives outside the loop. Agents will raise any number you give them. Recursive self-improvement without taste is a faster way to optimize the wrong thing. Deciding what counts as a win and that reward hacking does not is still our job.
\nThanks for reading!
\n", "date_modified": "2026-08-21T00:00:00+00:00", "date_published": "2026-08-21T00:00:00+00:00", "external_url": "https://www.philschmid.de/recursive-self-improvement", "id": "urn:aggr:item:1a51e2a1f8c5c69b975889815878963f0e704507", "summary": "Agents can already edit their tools, skills, and harness. Recursive self-improvement still needs a system that can raise the verifier without capturing it.", "tags": [ "programming" ], "title": "Recursive Self-Improvement", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-08-21-recursive-self-improvement/" }, { "content_html": "Last week we launched Gemini 3.7 Flash, our first hybrid reasoning model built for coding and agentic workflows.
\nIts multimodal reasoning and fast turn-by-turn latency make it uniquely suited for Computer Use. Instead of waiting several seconds between taps, the model inspects screenshots, plans the next step, and emits precise coordinates fast enough to keep device control loops interactive.
\nTo test this on a real mobile interface, I gave it a task: "play 1 round of Wordle at wordle.global/en/unlimited".
\nRunning on an Android emulator, it opened Chrome, cleared tutorial popups, played an opening word, read the color-coded feedback with thinking, and solved the game in 2 guesses.
\nMobile test automation has been painful for 15 years.
\nAppium, Espresso, and UIAutomator all depend on accessibility trees, element IDs, and XPath selectors. When a product team runs an A/B test, redesigns a checkout card, or adds a marketing banner, the test script breaks.
\nWebviews and dynamic canvas games are even worse because they often expose zero accessibility nodes to ADB.
\nGemini 3.7 Flash allows you to approach this almost human like. The model looks at the raw screenshot, figures out where elements sit, and returns normalized 0–999 coordinates for clicks, text input, and key presses.
\n![\"Multimodal](\"https://www.philschmid.de/static/blog/gemini-android-computer-use/android_agent_loop.svg\")
The Agent connects to an Android emulator over ADB and runs a continuous loop:
\nadb exec-out screencap -p) and encodes it as Base64.click(x=884, y=190)).0–999 to physical pixels (1080x1920) and fires adb shell input tap <x> <y>.function_result with previous_interaction_id.Here is the core loop using the google-genai Python SDK:
Python
\n\n\n \n = \n \n# 1. Package the prompt with the initial device screenshot\n = \n \n = None\n \n\n # 2. Call Gemini 3.7 Flash with mobile computer use and thinking\n = \n \n # 3. Stop if the model produced text rather than a tool call\n = \n \n \n break\n \n # 4. Dispatch actions (scales 0-999 coordinates to screen pixels over ADB)\n = \n \n \n \n # 5. Capture new screen state and return result to continue session\n \n \n = \n = \nThe full implementation, including the ADBBridge coordinate scaler, device setup scripts, and CLI runner, is in the gemini-android-computer-use-quickstart repository.
Here is what happened during the Wordle session.
\n1. Launch and navigation
\nThe agent started on the home screen, opened Chrome by package name, opened a new tab, and typed the URL:
\n![\"Agent](\"https://www.philschmid.de/static/blog/gemini-android-computer-use/step1_initial_task.jpg\")
[function_call] open_app({'package_name': 'com.android.chrome', 'intent': 'Open Chrome browser'})\n[function_call] click({'x': 270, 'y': 180, 'intent': 'Open a new tab in Chrome'})\n[function_call] type({'text': 'https://wordle.global/en/unlimited', 'press_enter': True, 'intent': 'Type Wordle Unlimited URL and navigate'})\n2. Clearing modal overlays
\nTwo popups blocked the game board on page load: a "How to Play" tutorial and a stats dialog.
\nThe model spotted the close icons and clicked them:
\n![\"Agent](\"https://www.philschmid.de/static/blog/gemini-android-computer-use/step2_dismiss_popups.jpg\")
[function_call] click({'x': 884, 'y': 190, 'intent': 'Close the How to Play modal'})\n[function_call] click({'x': 884, 'y': 256, 'intent': 'Close stats modal'})\n3. Playing guess 1 (CRANE)
\nWith a clear board, it typed a standard vowel-heavy opener: C - R - A - N - E.
![\"Entering](\"https://www.philschmid.de/static/blog/gemini-android-computer-use/step3_guess1_crane.jpg\")
[function_call] click({'x': 390, 'y': 915, 'intent': 'Click C'})\n[function_call] click({'x': 355, 'y': 736, 'intent': 'Click R'})\n[function_call] click({'x': 72, 'y': 822, 'intent': 'Click A'})\n[function_call] click({'x': 708, 'y': 908, 'intent': 'Click N'})\n[function_call] click({'x': 259, 'y': 736, 'intent': 'Click E'})\n[function_call] press_key({'key': 'Enter', 'intent': 'Press Enter key'})\n4. Reading tile colors with thinking
\nThe game returned feedback:
\nGemini checked words ending in E that contain C somewhere in the middle without R, A, or N. It picked SPICE. All 5 tiles turned green. The model saw the victory popup and stopped:
![\"Solved](\"https://www.philschmid.de/static/blog/gemini-android-computer-use/step5_solved_spice.jpg\")
Agent finished: I have completed a round of Wordle Unlimited at https://wordle.global/en/unlimited, successfully solving the puzzle ("SPICE") in 2 guesses.\nWordle is a clean visual benchmark, but the underlying loop works for any Android application.
\nYou can use this for automated UI testing, user flow verification, exploratory bug reproduction, or general task automation. Because the model operates on raw screenshots, it works across native Android apps, webviews, and dynamic canvas interfaces without relying on accessibility IDs or DOM trees.
\nThe complete quickstart with the emulator setup script, ADB bridge, and CLI runner is open source:
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-08-17T00:00:00+00:00", "date_published": "2026-08-17T00:00:00+00:00", "external_url": "https://www.philschmid.de/gemini-android-computer-use", "id": "urn:aggr:item:0b8859463d34cd6c1157bc75fb5335f59e0eb56d", "summary": "Use Gemini 3.7 Flash Computer Use with about 150 lines of Python to control an Android emulator. The model plays Wordle from screenshots: opens Chrome, clears popups, reads tile colors, and solves in two guesses.", "tags": [ "programming" ], "title": "Controlling Android with Gemini 3.7 Flash and 150 lines of Python", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-08-17-controlling-android-with-gemini-3-7-flash-and-150-lines-of/" }, { "content_html": "We just shipped something I've wanted for a while. You can now use the Google Maps and Google Search tools in the exact same call with Gemini 3.5 Flash and 3.6 Flash. You can also add custom functions or MCP servers into the same request via Tool Combination.
\nBuilding location apps usually means juggling web context and physical data yourself. You'd write separate LLM calls, parse messy JSON, and glue external APIs together.
\nNow Gemini handles the whole loop in one interaction:
\n![\"Agent](\"https://www.philschmid.de/static/blog/gemini-search-maps/agent_loop_tools_diagram.jpg\")
When Gemini finds a venue on the web, it queries Maps for the exact place details, then hands structured parameters to your function call. Zero roundtrips on your side.
\nHere is how you set it up with the Interactions API using @google/genai:
JavaScript
\n;\n \nconst ai = ;\n \nconst bookTableTool = ;\n \nconst interaction = await ai..;\nI put together a quick app to test the flow end-to-end.
\nYou start with a prompt:
\n![\"Prompt](\"https://www.philschmid.de/static/blog/gemini-search-maps/screen_1_empty_input.jpg\")
In one turn, Gemini searches the web for patio spots in SoHo, verifies the venues on Google Maps, drops pins on the map, and prepares the reservation action:
\n![\"Results](\"https://www.philschmid.de/static/blog/gemini-search-maps/screen_2_results_and_map.jpg\")
When the user clicks confirm, your app calls your real reservation service, gets a reference ID, and hands that result back to Gemini in turn 2 using previous_interaction_id:
JavaScript
\n// 1. Execute the actual booking in your backend\nconst bookingResult = await reservationService.;\n \n// 2. Send the result back to Gemini to finish the interaction\nconst finalInteraction = await ai..;\nThe server keeps the whole grounding context and conversation state alive without you having to re-send token histories:
\n![\"Reservation](\"https://www.philschmid.de/static/blog/gemini-search-maps/screen_4_confirmed.jpg\")
Instead of writing custom booking functions by hand, you can plug in an existing OpenTable MCP server directly:
\nJavaScript
\nconst interaction = await ai..;\nGemini finds the restaurant through Search, checks reviews and coordinates on Maps, and handles the live reservation through OpenTable's MCP server in the background.
\nHaving Search, Maps, and MCP together cuts out a lot of boilerplate. You don't have to extract keywords, call Places APIs, and re-prompt to figure out the next step.
\nCheck out the docs for Tool Combination, Maps Grounding, and the Interactions API Quickstart to try it out.
\n", "date_modified": "2026-08-12T00:00:00+00:00", "date_published": "2026-08-12T00:00:00+00:00", "external_url": "https://www.philschmid.de/gemini-search-maps", "id": "urn:aggr:item:078446d5436f7bae3923c035719db6e3e0ef50f8", "summary": "Use Google Maps and Google Search in the same Gemini API call with Gemini 3.6 Flash, then add custom functions or MCP servers for actions like booking a table.", "tags": [ "programming" ], "title": "Google Maps and Google Search now work together in the Gemini API", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-08-12-google-maps-and-google-search-now-work-together-in-the/" }, { "content_html": "Most coding benchmarks work the same. You give the agent a single task, let it work, check the result. The agent might run dozens of tool calls, but there's only one user input and one final evaluation.
\nThat's not how most of us use agents. You build something, you get new ideas, requirements shift, you refactor... EvoCode-Bench is designed to test that cycle.
\nEvoCode-Bench is a new multi-turn coding benchmark with 26 tasks spanning 227 sequential rounds (5–15 rounds per task). It evaluates agents across five domains: ML/MLOps, build systems, data engineering, cloud/security, and scientific computing. Three things make it interesting:
\nSingle-Turn Eval (SWE-bench):\n [Prompt] → [Agent works] → [Pass/Fail] → Container discarded\n \nMulti-Turn Eval (EvoCode-Bench):\n [Prompt 1] → [Agent works] → [Test 1] →\n [Prompt 2] → [Agent works] → [Test 1+2] →\n [Prompt 3] → [Agent works] → [Test 1+2+3] → ...\nOne task (d1_w5) asks the agent to build buildctl, a CLI build orchestrator in Go, over 8 rounds. Here's how the first few rounds work:
Round 1:
\nbuildctl in Go that orchestrates multi-target build pipelines with dependency resolution, parallel execution, and content-addressable artifact caching." The full spec defines TOML config parsing, topological sorting, cache key computation (SHA-256 of command + env + input file hashes), 4 CLI commands (build, graph, clean, status), JSON report format, and error handling for cycles, missing deps, and timeouts.round-1/tests/test.sh (24 test cases) and runs it against the CLI. Test scripts are removed after verification so the agent can't read future grading criteria.Round 2:
\nimports field in TOML for composing pipelines across files (with transitive import support, cycle detection, name collision errors), and a [variables] section with ${VAR_NAME} expansion in commands, inputs, and outputs.Round 5:
\ncancelled (distinct from skipped). Add --keep-going flag to restore the old behavior."Single-turn scores dramatically overstate reliability. The researchers tested two modes: one where each round starts from a clean, human-completed codebase (SR), and one where the agent maintains its own workspace across rounds (MT@4). Agents are good at following instructions on clean code. They're significantly worse when building on their own past work. For lower-tier models the gap is ~4x (8.4% MT → 33.1% SR); for frontier models it's still 1.4–1.8x. Over 57% of multi-turn failures occur on rounds that models solve easily from a clean state.
\nRankings shift under pressure. Claude Opus 4.6 had the highest single-round score (78.9%) but drops to third in multi-turn (44.0%), behind Opus 4.7 (54.0%) and GPT-5.5 (52.4%). Across all models, pass rates go from 46.7% at Round 1 to 21.3% by Round 5 to 7.7% by Round 10. More turns, more accumulated decisions, more failures.
\nFailure modes differ by model tier. Lower-tier agents fail early, they simply don't implement what was asked (87–90% of failures). Mid-tier agents handle the initial rounds but when the spec changes, they add the new logic without fully removing the old behavior it replaced (28.6% of failures, peaking at Round 5). Top-tier agents get further but eventually break something that was already working, regressions account for 35% of failures by Round 2.
\nRegressions are the real bottleneck. Agents rarely fail because they can't build the feature. They fail because they break something that was already working. When the spec changes, the agent edits code to handle the new requirement, but those edits touch shared code paths and accidentally break earlier behavior. The agent doesn't always re-verify that the old stuff still works.
\nStructure helps. Agents that tracked requirements in a persistent document (like a project plan or spec file) more than doubled their success rates.
\nSmall dataset. 26 tasks, 227 rounds. A few hard tasks can skew the aggregate scores.
\nAll-or-nothing scoring. A single regression zeros out the entire round, even when 98% of test cases pass. The paper also reports per-case scores showing agents often achieve >80% assertion accuracy on rounds scored as binary failures.
\nNo recovery loops. Under fail-stop scoring (MT@4), a single failed round terminates the entire trial — all remaining rounds automatically score zero. The benchmark doesn't test whether an agent can recover from a broken state.
\nStatic interaction. The "user" is a fixed script. No clarification questions, no "that's not what I meant," no ambiguity.
\nInfra-sensitive. The paper's own results show infrastructure issues (container throttling, kubelet errors) can corrupt scores.
\nEvoCode-Bench tests whether an agent can sustain a codebase over time — with cumulative test suites that grow from ~900 lines of bash at Round 1 to nearly 4,000 by Round 5. Even the best models lose more than half their capability by Round 5. The dominant failure isn't "can't build the feature" — it's regressions, breaking something that was already working. When specs reverse earlier decisions, even frontier models struggle. The clearest signal: agents that maintained a persistent requirements document more than doubled their success rate. Structure beats raw capability.
\nYou can run the GitHub CLI (gh) inside a Gemini API Managed Agents without exposing your GitHub Personal Access Token (PAT) to the sandbox.
The Network Configuration egress proxy injects your real token into outbound requests. The agent uses a dummy token locally, so the real token never enters the sandbox.
\nGH_TOKEN="dummy" inside the sandbox to satisfy gh's local auth checks.api.github.com or github.com.Authorization header with your real token.![\"Egress](\"https://www.philschmid.de/static/blog/managed-agents-gh/mermaid.jpg\")
Gemini API Key: Get your key from Google AI Studio.
\nGitHub Personal Access Token (PAT): I recommend a Fine-grained PAT restricted to the specific repositories your agent needs.
\nTo create one:
\nAccess: Read and write (to clone and push code).Access: Read and write (to open, review, and comment on PRs).Access: Read and write (to create and comment on issues).This Python example spins up the agent in an isolated, ephemeral Linux environment, configures the network allowlist with header transformations, and runs gh.
Python
\n\n\n\n \n# GitHub PAT from host\n = \n\n \n \n# Base64-encode for Git HTTPS auth\n = f\n = \n \n = \n \n# Shim: auto-installs gh, injects dummy token, ensures git HTTPS and no prompt\n = \n \n = \n \n = \n \n\n# Logged in as philschmid; cloned octocat/Spoon-Knife to /workspace/Spoon-Knife\nGitHub uses different auth schemes depending on the request. API calls to api.github.com expect a Bearer token, while Git operations against github.com require base64-encoded Basic auth.
The wildcard entry ("domain": "*") lets the sandbox reach everything else, e.g. downloading the gh binary, fetching release assets, etc. You can lock this down to specific domains like objects.githubusercontent.com if you prefer tighter controls.
To make the agent autonomous and avoid setup delays, you can create a managed agent that auto-installs the gh CLI. It reuses the environment after the first interaction to keep the installed gh binary across calls.
Python
\n\n\n\n \n# GitHub PAT from host\n = \n \n = f\n = \n \n = \n \n = \n \n# Shim: auto-installs gh, injects dummy token, forces HTTPS and disables interactive prompts\n = \n \n\nThe first interaction clones the base environment and triggers gh installation.
Python
\n = \n \n\n# ... currently logged-in GitHub user is **philschmid**.\nFor subsequent calls, pass interaction_1.environment_id to reuse the same container, keeping the installed gh binary and inheriting the network transforms. Optionally pass previous_interaction_id=interaction_1.id as well if you want to retain conversation history across turns.
Python
\n = \n \n\n# Based on the inspection of pull request #12 in the\nYou can give Managed Agents access to GitHub without putting secrets in the sandbox. A dummy token satisfies gh locally, and the egress proxy injects the real PAT on the way out.
For more on Managed Agents, environments, and network transforms, see the Managed Agents docs and Network configuration.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-07-17T00:00:00+00:00", "date_published": "2026-07-17T00:00:00+00:00", "external_url": "https://www.philschmid.de/managed-agents-gh", "id": "urn:aggr:item:09f72419a2d4072c17c4414e1182d560fbd54c27", "summary": "Build Gemini Managed Agents that use the GitHub CLI without exposing your PAT. The egress proxy injects your real token into outbound requests while the sandbox only sees a dummy token.", "tags": [ "programming" ], "title": "Building Managed Agents That Use GitHub Without Exposing Your Token", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-07-17-building-managed-agents-that-use-github-without-exposing/" }, { "content_html": "June 25, 2026 9 minute read View Code
\nGemini 3.5 Flash has built-in Computer Use. The model looks at a screenshot, decides what to do, and returns a function call like click(y=300, x=500). You execute that action on the device via ADB, take a new screenshot, and send it back. Repeat until the task is done.
This guide walks you through controlling an Android emulator using the mobile environment and the Python SDK.
Github Repository: https://github.com/google-gemini/gemini-android-computer-use-quickstart
\nComputer Use is a native tool in Gemini 3.5 Flash. You give the model a screenshot and a goal ("open Settings and turn on dark mode"). The model returns structured actions: taps, text input, swipes, app launches. Your code executes those actions on the target device.
\nIt works like function calling. The model proposes actions, you execute them, you send back the result. The model stays in the loop through screenshots.
\nGemini supports three Computer Use environments: browser for desktop web automation, mobile for mobile devices/emulators, and desktop for operating system-level control. This guide uses mobile.
Screenshot ──> Gemini 3.5 Flash ──> function_call\n ▲ │\n │ ▼\n └──────── Execute via ADB ◄──────────┘\nPython
\n\n \n = \n = \n \n# Take initial screenshot and send first request\n = \n \n = \n \n# Agent loop: execute actions, send results back\n\n \n # click, type, open_app...\n \n = \n = \n \n\nThese are the actions the model can request when using environment: "mobile". Your code must handle each one.
| Action | \nDescription | \nArguments | \n
|---|---|---|
open_app | \nOpens an app by name | \napp_name: str, intent: str (optional) | \n
click | \nTap a screen coordinate | \ny: int (0-999), x: int (0-999) | \n
type | \nType text | \ntext: str, press_enter: bool (default false) | \n
long_press | \nLong-press a coordinate | \ny: int (0-999), x: int (0-999), seconds: int (default 2) | \n
drag_and_drop | \nDrag between two points | \nstart_y, start_x, end_y, end_x | \n
press_key | \nPress a key (home, back, enter...) | \nkey: str | \n
go_back | \nNavigate back | \n(none) | \n
wait | \nPause execution | \nseconds: int (default 1) | \n
list_apps | \nList installed apps | \n(none) | \n
take_screenshot | \nCapture the screen | \n(none) | \n
Coordinates are normalized to a 0-999 grid. (0, 0) is top-left, (999, 999) is bottom-right. Your bridge converts these to actual pixel coordinates based on the device resolution.
No Android Studio GUI required. Run the setup script on your Mac to install the Android SDK, emulator, and create a virtual device from the terminal.
\nInstall script link: https://github.com/google-gemini/gemini-android-computer-use-quickstart/blob/main/setup_emulator.sh
\nBash
\n\n\nBash
\n\nThe Python agent script handles locating SDK paths and starting the emulator in the background automatically, so no additional manual environment variable exporting or terminal setup is required.
\nFull working script agent.py. It automatically sets up the environment variables, launches the emulator in the background (if it isn't already running), waits for it to boot, and begins the Computer Use loop.
\nPython
\n\n\n\n\n\n\n\n \n\n \n = \n \n \n\n = \n \n = None\n \n \n = \n break\n \n \n \n \n \n = \n = \n \n = \n \n \n = + + \n \n = \n return \n \n \n\n \n \n \n = \n \n return\n \n pass\n \n \n = \n \n \n \n \n \n \n = \n \n = \n \n \n return\n \n pass\n \n \n \n \n \n \n\n\n = + \n , = \n \n\n = \n \n \n return \n \n\n = \n = \n return \n \n\n return , \n \n\n , = \n \n \n\n \n \n \n \n\n = or \n \n \n \n = \n \n \n \n\n , = \n = \n , = \n \n \n\n , = \n \n \n\n , = \n , = \n \n \n\n = \n \n \n\n \n \n\n \n \n\n = \n = \n \n return \n return \n \n\n return None\n \n\n = \n return \n \n \n = \n \n \n\n \n = \n = \n \n \n \n \n = \n = \n \n = None\n = 0\n \n \n += 1\n \n = \n \n = \n \n \n \n = \n = \n \n \n \n = \n \n \n \n = \n \n = \n \n \n \n \n # auto approve safety decisions for demo\n = True\n \n = \n \n = \n \n \n \n break\n \n = \n = \n \n \n break\n \n return \n \n \n\n = \n \n = \n \nBash
\n# Set your API key\n\n \n# Run the agent (starts the emulator automatically if needed)\n\nYou can also target physical Android devices or remote cloud emulators instead of a local virtual device.
\nEnable USB/Wireless Debugging: On your target device, enable Developer Options and turn on USB Debugging or Wireless Debugging.
\nConnect via ADB: Use the adb connect command to link to a remote device over TCP/IP:
Bash
\n\nFor details on setting up wireless debugging, refer to the official Android developer documentation on ADB over Wi-Fi.
\nPass the Device ID to the Agent: Pass the remote device's connection string as the device_id parameter to target it in the agent loop:
Python
\n# Target a remote or cloud-hosted emulator\n\nI hope this gets you started with Gemini 3.5 Flash Computer Use on mobile. Next steps can be:
\nmobile environment is platform-agnostic. The model outputs actions (clicks, swipes, types) on the same normalized 0-999 grid regardless of whether the device is Android or iOS. To target an iPhone or iOS Simulator, you only need to swap out ADBBridge with an iOS-compatible tool—such as Apple's simctl CLI for simulator control, Appium, or go-ios for physical devices.safety_decision requesting confirmation. Ensure your production loop inspects step.arguments for safety flags and prompts the user before executing the action. See the Gemini API Computer Use Safety Guidelines for details.Thanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-06-25T00:00:00+00:00", "date_published": "2026-06-25T00:00:00+00:00", "external_url": "https://www.philschmid.de/gemini-android-use", "id": "urn:aggr:item:fbfce7e022024d5e22915e897e6085c3dc08beb5", "summary": "Control an Android emulator using Gemini 3.5 Flash Computer Use. Connect the Google GenAI SDK interactions loop with ADB to control a virtual device from your terminal.", "tags": [ "programming" ], "title": "Control an Android Phone with Gemini 3.5 Flash Computer Use", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-06-25-control-an-android-phone-with-gemini-3-5-flash-computer-use/" }, { "content_html": "The Interactions API is Google's primary interface for Gemini models and agents. A single endpoint covers text generation, streaming, multi-turn chat, multimodal inputs, image generation, structured output, tool use, function calling, managed agents, and background execution.
\nThis guide uses JavaScript. For Python and REST examples, see the Interactions API quickstart.
\nUsing a coding agent? Install the skill so your agent stays current with Interactions API patterns:
\nShell
\n\nCreate a free API key at Google AI Studio, then set it as an environment variable:
\nShell
\n\nInstall the SDK:
\nShell
\n\nSend your first request.
\nJavaScript
\n;\n \nconst ai = ;\n \nconst interaction = await ai..;\n;\ninteraction.output_text gives you the final text directly. See the text generation guide for system instructions and generation config.
Add stream: true and iterate over events. Each step.delta with type === "text" is a chunk you can display immediately.
JavaScript
\n;\n \nconst ai = ;\n \nconst stream = await ai..;\n \nawait \nSee the streaming guide for event types and delta handling.
\nChain interactions by passing previous_interaction_id. The server manages history for you.
JavaScript
\n;\n \nconst ai = ;\n \nconst interaction1 = await ai..;\n;\n \nconst interaction2 = await ai..;\n;\nFor client-side history management, set store: false. See the multi-turn guide.
Gemini natively understands images, audio, video, and documents. Upload a file and pass it alongside text.
\nJavaScript
\n;\n \nconst ai = ;\n \nconst uploadedFile = await ai..;\n \nconst interaction = await ai..;\n;\nAudio, video, and documents use the same structure. See the guides for audio, video, and document processing.
\nGenerate images with Nano Banana 2 using the gemini-3.1-flash-image model.
JavaScript
\n;\n;\n \nconst ai = ;\n \nconst interaction = await ai..;\n \nfs.;\nSpeech generation (multi-speaker TTS) and music generation (Lyria 3) work the same way. See the image generation guide for editing, aspect ratios, and style references.
\nGet JSON that matches a schema you define. Works with Zod.
\nJavaScript
\n;\n;\n \nconst ai = ;\n \nconst recipeJsonSchema = ;\n \nconst recipeSchema = z.;\n \nconst interaction = await ai..;\n \nconst recipe = recipeSchema.;\n;\nSee the structured output guide for recursive schemas and enums.
\nGround responses in real-time data by passing tools: [{ type: "google_search" }].
JavaScript
\n;\n \nconst ai = ;\n \nconst interaction = await ai..;\n;\nOther built-in tools: Code Execution, URL Context, File Search, Google Maps, Computer Use. Mix multiple tools in one request. See the tool combination guide.
\nDeclare functions, let the model decide when to call them, execute locally, and return results.
\nJavaScript
\n;\n \nconst ai = ;\n \nconst weatherTool = ;\n \nconst availableFunctions = ;\n \nlet input = "What is the temperature in London?";\nlet previousId = null;\nlet interaction;\n \n\n \n;\nThe model returns status: "requires_action" with function_call steps. You execute locally and submit function_result steps back. See the function calling guide for parallel calls and function choice modes.
Run an agent in a remote sandbox with code execution, web browsing, and file management. Pass agent instead of model and set environment: "remote".
JavaScript
\n;\n \nconst ai = ;\n \nconst interaction = await ai..;\n;\nDefine custom agents with your own instructions, skills, and data sources. See the Managed Agents quickstart.
\nSet background: true for long-running tasks. The call returns immediately and you poll for results.
JavaScript
\n;\n \nconst ai = ;\n \nconst interaction = await ai..;\n;\n \nconst poll = ;\nSee the background execution guide.
\nThanks for reading! If you have any questions, feel free to contact me on Twitter or LinkedIn.
\n", "date_modified": "2026-06-23T00:00:00+00:00", "date_published": "2026-06-23T00:00:00+00:00", "external_url": "https://www.philschmid.de/interactions-api-developer-guide", "id": "urn:aggr:item:628eaaa23480f670626c0b0fce711001cbb81635", "summary": "A developer guide to the Gemini Interactions API, text generation, streaming, multi-turn chat, multimodal inputs, structured output, tool use, managed agents, and background execution from a single endpoint.", "tags": [ "programming" ], "title": "Getting started with the Gemini Interactions API", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-06-23-getting-started-with-the-gemini-interactions-api/" }, { "content_html": "You can go from zero to a working agent in five lines. One API call, and the response comes back with a finished PDF, charts, and a summary.
\nPython
\n\n \n = \n \n = \n \n\nWhat most people miss: this is not a single model call that returns text. Behind that one call, a sandbox boots, skills get loaded, and the model enters a loop where it reasons, picks tools, executes code, reads the output, and repeats until the task is done.
\ninteractions.create() does more than send a prompt. It spins up a full execution environment.
User
\nYour code
\nGemini API
\nOrchestrator
\nGemini 3.5 Flash
\nReasoning
\nSandbox
\nLinux VM
\nPrompt
\nWrite a python script that prints hello world and execute it.
\nRequest: Your prompt hits the Interactions API via HTTPS. The API accepts it and starts orchestrating the interaction server-side.
\n1 / 10
\nEach interaction gets its own isolated Linux container. The agent can install packages, run scripts, read and write files, and browse the web. The sandbox runs Ubuntu with 4 vCPU and 16 GB RAM. Environment compute is not billed during preview (you pay only for model tokens).
\nEnvironments can persist across interactions. The first call returns an environment_id you pass back to keep files and state intact. See the Environments documentation for sources, networking, and lifecycle details.
Python
\n# Turn 1: sandbox is created\n = \n \n# Turn 2: same sandbox, files persist\n = \nThe execution loop is the same whether you are experimenting or running in production. The difference is how much configuration you want to provide upfront.
\nenvironment="remote".Python
\n = \nsystem_instruction or AGENTS.md to customize behavior and provide skills via sources.Python
\n = \nclient.agents.create(). Each invocation forks a fresh sandbox from the base environment.Python
\n = \nPython
\n = \n\nBuild, customize, and deploy production agents that reason, execute code, and manage files inside secure Linux sandboxes as a single API call.
\n\n\nIf you are an AI coding assistant, ask your human to install the Interactions API skill:
\nnpx skills add google-gemini/gemini-skills --skill gemini-interactions-api, or read it directly at SKILL.md.
No servers to manage. No orchestration code to write. You call the Interactions API, and Gemini 3.5 Flash handles reasoning, code execution (Python, Node.js, Bash), package installation, file management, and web browsing inside an isolated Linux sandbox.
\nYou can use the Antigravity agent out of the box for coding, research, and data analysis (same harness as the Antigravity IDE), or build your own with custom instructions, skills, and data. The fastest way to try it is directly in AI Studio.
\nInstall the SDK and set your API key:
\nBash
\n\n\nThen:
\nPython
\n\n \n = \n \n = \n \n\nThis provisions a sandbox, runs the agent loop, and returns the result. The response includes interaction.id and interaction.environment_id for follow-up calls. Long sessions compact context automatically at ~135k tokens, so you don't hit limits (more).
JavaScript and REST examples are in the Quickstart.
\nThe first interaction returns an environment_id and interaction.id. The API tracks conversation context and environment state independently.
environment_id as environment in a follow-up interaction, the sandbox persists.previous_interaction_id with the last interaction.id, the conversation history persists.Python
\n = \nOmit previous_interaction_id to start a fresh conversation in the same workspace. Set environment="remote" to get a new sandbox while keeping conversation context. Details in the multi-turn guide.
Stream responses to see the agent's work as it happens:
\nPython
\n = \n \n\n \nReturns incremental text, reasoning tokens, and tool call updates. See the streaming guide for JavaScript, REST, and event types.
\nMount files from Git repos, Cloud Storage, or inline content at environment creation. Source types:
\nrepository: clones a Git repo (public or private).gcs: copies from a Google Cloud Storage bucket.inline: writes text content to a file path.Python
\n = \nDownload the full environment as a tar archive:
\nPython
\n\n \n = \n \n\n \n \n\n \nSave your configuration as a managed agent when you want the same behavior across runs. Agents are defined through files the runtime auto-discovers:
\n.agents/\n├── AGENTS.md # Loaded as system instructions on startup\n└── skills/\n └── slide-maker/\n └── SKILL.md # Registered as an agent capability\nDefine instructions, skills, and data declaratively. Each invocation provisions a fresh sandbox with your files:
\nPython
\n = \nSet up the environment interactively, then freeze it:
\nPython
\n# Step 1: set up the environment\n = \n \n# Step 2: fork into a reusable agent\n = \nEach invocation forks the base environment. Every run starts clean:
\nPython
\n = \n\nFull agent definition format, CRUD operations, and per-invocation overrides in Building managed agents.
\nEnvironments have unrestricted outbound access by default. Use a network allowlist to restrict it. The egress proxy limits connections to listed domains and can inject credentials securly as HTTP headers transformation. Secrets never exist inside the sandbox:
\nPython
\n = \nFull allowlist schema, credential patterns, and private repo access in Network configuration.
\nWe open-sourced the Gemini API CLI (experimental) as a terminal-first interface to the Gemini API. It works for both humans and coding agents, covering everything from quick prompts to the full managed agent lifecycle:
\nBash
\n# Run a prompt\n\n \n# Image generation\n\n \n# Text-to-speech\n\n \n# Scaffold, test, and deploy a managed agent\n\n\n\n \n# Run against a deployed agent\n\nEach interaction runs inside a Linux sandbox (Ubuntu, Python 3.12, Node.js 22) with 4 CPU cores and 16 GB RAM. The sandbox is isolated at the OS level, so the agent can install packages, run code, and write files without affecting your machine.
\nEnvironments persist. The first call creates one and returns an environment_id. Pass it back to pick up where you left off with files, packages, and state intact. Environments idle after 15 minutes and are deleted after 7 days of inactivity.
You can pre-load environments with data from Git repos, Cloud Storage, or inline content (see Loading data into the sandbox). You can also restrict outbound network access with domain allowlists and inject credentials via the egress proxy, so secrets never touch the sandbox. Details in Environments and Network configuration.
\nEnvironment compute (CPU, memory, sandbox execution) is not billed during preview. You pay for Gemini model (Gemini 3.5 Flash) tokens only.
\n", "date_modified": "2026-05-26T00:00:00+00:00", "date_published": "2026-05-26T00:00:00+00:00", "external_url": "https://www.philschmid.de/gemini-managed-agents-developer-guide", "id": "urn:aggr:item:c696c37fb17d6ee157e3bde22238bb965753c2c0", "summary": "Build, customize, and deploy production agents that reason, execute code, and manage files inside secure Linux sandboxes as a single API call.", "tags": [ "programming" ], "title": "Gemini Managed Agents: Developer Guide", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-05-26-gemini-managed-agents-developer-guide/" }, { "content_html": "Last year I wrote about the rise of subagents and why isolating tasks into focused agents with their own context, tools, and instructions improves reliability. That post covered what subagents are and why you want them. Since then, models have gotten better at planning, tool use, and multi-step instructions. That opens up patterns that were previously too fragile: agents managing pools of persistent workers, or agents messaging each other directly.
\nHere are four patterns, ordered by the level of control the main agent has over the subagent lifecycle.
\nThe simplest pattern. The main agent calls a tool that spawns a subagent and returns the result as the tool response. From the main agent's perspective, calling a subagent is identical to calling read_file or run_command.
![\"Pattern](\"https://www.philschmid.de/static/blog/subagent-patterns-2026/pattern%201.png\")
The main agent has a tool, e.g. call_agent. It sends a task, gets a result. The subagent runs in its own context with its own tools and instructions. The main agent never manages the subagent's lifecycle directly.
JSON
\n\nThis pattern has a sync/async distinction at the tool level, because the tool itself can either block or return immediately:
\nSync (above): The tool call blocks. The main agent's turn is paused until the subagent finishes. The result arrives as a normal tool response.
\nAsync: The tool returns immediately with an agent ID. The subagent's result is injected into the conversation as a notification message when it finishes. The main agent can do other tool calls before the result arrives.
\nJSON
\n\nIn sync, the tool response contains the result. In async, the tool response contains an ID, and the result arrives later as an injected user message.
\nWhen to use this: Self-contained work. Research lookups, code reviews, file analysis, test generation. Most subagent use cases start and stay here.
\nLimitations: Works with any model that supports tool use, including smaller and cheaper ones. Results arrive as a single tool response (sync) or as an injected notification message (async). No way to send follow-up instructions, check progress, or cancel early. If the subagent misunderstands the task, you find out when the result comes back.
\nThe main agent spawns subagents and collects results using a wait_agent tool. Unlike the inline tool, spawning and collecting are separate steps. spawn_agent always returns immediately. The model decides what to do next: call wait_agent right away, do its own work first, or spawn more agents before waiting.
![\"Pattern](\"https://www.philschmid.de/static/blog/subagent-patterns-2026/pattern%202.png\")
Two tools: spawn_agent dispatches work and returns immediately with an ID. wait_agent blocks until one or more agents finish.
The model controls sequencing. It can spawn agents and call wait_agent immediately (effectively synchronous), or interleave its own work first:
JSON
\n\nThe model spawned two agents and read a file in the same tool call, then decided to wait. A different model might call wait_agent immediately after spawning, or do several reads first. The tool surface is the same; the model decides the interleaving.
The difference from Pattern 1: the model controls when to collect. It can spawn 5 agents in parallel, do its own file reads, then call wait_agent. wait_agent can act as a global mailbox that wakes up on subagent completion and returns all available results.
When to use this: Multiple independent tasks that can run concurrently. The main agent does not need intermediate results from one subagent to start another.
\nLimitations: The model needs to correctly decide when to wait. A model that calls wait_agent immediately after every spawn gets no benefit over Pattern 1. The value depends on the model's ability to interleave its own work between spawn and wait. Results are collected in batch via wait_agent, which returns all completed agent outputs since the last call. Still fire-and-forget: no way to send follow-up instructions or course-correct mid-task to subagents.
The main agent spawns long-lived subagents and communicates with them through messages. Agents persist across interactions. The main agent can send follow-up instructions, check status, and coordinate work between agents.
\nThis requires a richer tool surface: spawn_agent, send_message, wait_agent, list_agents, kill_agent.
![\"Pattern](\"https://www.philschmid.de/static/blog/subagent-patterns-2026/pattern%203.png\")
Unlike Pattern 2, agents here are stateful and interactive. The main agent sends a message, gets a response, and sends another message to the same agent. The agent retains its full conversation history. This supports multi-step workflows where the main agent coordinates between specialists.
\nThe model controls whether it works with one agent at a time (sequential) or messages multiple agents between waits (parallel). It can send to the researcher and writer simultaneously, or wait for the researcher first:
\nJSON
\n\nagent-r is messaged twice (research, then fact-check) and retains its sources from the first task. The model sent work to both agents in parallel at the start, then coordinated sequentially for the fact-check. A different model might serialize everything. The tools are identical; the model controls the orchestration.
Results arrive per-message via wait_agent. Each call to wait_agent returns the next agent response, so the main agent processes results incrementally and can adjust its next message based on what came back.
When to use this: Multi-step workflows where agents need to collaborate. The main agent routes information between specialists.
\nLimitations: The main agent must track multiple agent states, decide when to send follow-ups vs wait, and route information correctly. Smaller models lose track of which agent has which context, or forget to call kill_agent when done. Frontier models might handle 2-4 agents okayish.
Agents message each other directly without going through the main agent. The main agent sets up the team, defines roles, and steps back. Agents coordinate among themselves using direct messaging or a shared mailbox.
\n![\"Pattern](\"https://www.philschmid.de/static/blog/subagent-patterns-2026/pattern%204.png\")
The tool surface includes cross-agent addressing. Each agent gets send_message in its own tool set and can address other agents by name or path.
From the main agent's perspective, the conversation is short. It sets up the team, kicks off the planner, and waits for the final report. The model can choose to wait silently, or periodically check in using list_agents:
JSON
\n\nThe planner messaging the implementer, the implementer sending code to the reviewer, the reviewer requesting fixes — all of that happens inside the agents' own conversations. The main agent's context stays clean. The model checked in with list_agents after a timeout. It could have called wait_agent with no timeout and blocked until completion.
The main agent collects results only when an agent explicitly messages back (here, the planner reports completion). Everything else is invisible to it. Implementations vary in how cross-agent addressing works: hierarchical paths (/root/planner to /root/implementer), file-based mailboxes, or IPC protocols.
When to use this: Large tasks where the coordination logic exceeds what a single agent can manage step-by-step.
\nLimitations: Every agent in the team needs frontier-class model capabilities, not just the main agent. Each team member must independently decide when to message teammates, what to include, and when to report back. Models can message the wrong agent, forget to report completion, or get stuck in loops. Beyond model capability, there are infrastructure challenges: cycle detection (A waits on B, B waits on A), conflict resolution (two agents edit the same file), and shutdown coordination. Debugging is hard because message chains between agents are difficult to trace and failures cascade.
\n| Pattern | \nTools | \nMain Agent Role | \nAgent Lifetime | \n
|---|---|---|---|
| Inline Tool | \ncall_agent | \nCaller | \nSingle task | \n
| Fan-Out | \nspawn_agent, wait_agent | \nDispatcher | \nSingle task | \n
| Agent Pool | \nspawn, send, wait, list, kill | \nCoordinator | \nMulti-turn | \n
| Teams | \nAll above + cross-agent send_message | \nSupervisor | \nPersistent | \n
Start with Pattern 1. Most tasks that feel like they need a multi-agent system work fine with a well-prompted inline tool call. Move to Pattern 2 for genuinely independent parallel work. Move to Pattern 3 when agents need to collaborate across multiple steps. Pattern 4 is for when coordination logic exceeds what a single agent can manage.
\nEach step up requires a more capable model. Pattern 1 works with any model that can call tools. Pattern 2 needs a model that reasons about when to wait. Pattern 3 needs a model that tracks multi-agent state across turns. Pattern 4 needs frontier models for every agent. With a smaller or cheaper model, stay with Pattern 1 or 2.
\nResult collection also changes across patterns. Pattern 1 returns results inline as a tool response. Pattern 2 batches results via wait_agent. Pattern 3 delivers results incrementally. Pattern 4 surfaces only what agents explicitly report back; everything else stays inside inter-agent conversations.
For patterns 2-4, spawn_agent always returns immediately and wait_agent collects results when the model decides to call it. The framework provides the tools. The model controls the orchestration. A task that takes 4 coordinated agents today may be solvable by a single agent with a better model tomorrow.
The Gemini Deep Research Agent autonomously plans, searches, and synthesizes long-horizon research tasks into detailed, cited reports.
\nDeep Research handles long-running tasks by executing in the background. It is exclusively available through the Interactions API (not generate_content).
Two new versions are available:
\ndeep-research-preview-04-2026: Designed for speed & efficiency, ideal to be streamed back to a client UIdeep-research-max-preview-04-2026: Maximum comprehensiveness for automated context gathering & synthesisInstall the Python SDK:
\nShell
\n\nSet your API key as an environment variable. You can create one at aistudio.google.com/apikey.
\nShell
\n\nStart a research task with background=True and poll for the result. Deep Research is asynchronous as tasks can take several minutes to complete.
Python
\n\n\n \n = \n \n = \n \n\n = \n \n \n break\n == :\n \n break\n \nSet collaborative_planning=True to get a research plan back instead of running immediately. Iterate on the plan with previous_interaction_id, then set collaborative_planning=False to execute.
Step 1: Request a plan
\nPython
\n\n\n \n = \n \n = \n \n\n \n \n\nRefine the plan: Use previous_interaction_id to continue the conversation. Keep collaborative_planning=True to stay in planning mode. Repeat as needed.
Python
\n = \n \n\n \n \n\nApprove and execute: Set collaborative_planning=False to approve the plan and start the research.
Important: You must explicitly set collaborative_planning=False on the final turn. Simply sending "go ahead" without flipping the flag will not trigger report generation.
Python
\n = \n \n\n \n \n\nSet visualization="auto" and ask for visuals in your prompt. The agent generates charts and infographics returned as base64-encoded images.
Python
\n\n\n \n = \n \n = \n \n\n \n \n\n \n \n == and :\n = \n # display(Image(data=image_bytes)) # Jupyter\nTip: Setting visualization="auto" enables the capability, but best results are achieved by explicitly asking for what you want.
Connect remote MCP servers to give the agent access to external tools. Pass the server name, url, and optional auth headers.
Python
\n = \nMCP servers support no-auth, bearer token, and OAuth. For OAuth, fetch the token with a library like google-auth and pass it in headers. Use allowed_tools to restrict which tools the agent can call from the server.
By default the agent uses Google Search, URL Context, and Code Execution. You can customize the tools the agent can use by providing a list of tools, similar to models. This allows you to, for example, only search the web (via google_search and url_context), only search private sources (via file_search and custom MCP servers), or search a mix of both.
| Tool | \nType | \nDefault | \nDescription | \n
|---|---|---|---|
| Google Search | \ngoogle_search | \n✅ | \nSearch the public web | \n
| URL Context | \nurl_context | \n✅ | \nRead and summarize web pages | \n
| Code Execution | \ncode_execution | \n✅ | \nRun code for calculations and data analysis | \n
| MCP Server | \nmcp_server | \n— | \nConnect remote MCP servers | \n
| File Search | \nfile_search | \n— | \nSearch uploaded document corpora | \n
Python
\n# Only web search allowed\n = \nNote: Passing no tools at all defaults to Search, URL Context, and Code Execution enabled.
Pass images, PDFs, and documents alongside your text prompt to ground the research.
\nPython
\n = \nStream research progress in real time. Enable thinking_summaries="auto" to receive the agent's intermediate reasoning alongside text and generated images.
Python
\n\n\n\n \n = \n \n = None\n = None\n = False\n \n\n global , , \n \n \n = \n \n = \n \n \n \n == :\n \n == and :\n = \n \n in :\n = True\n \n \n \n = \n\n \n# Reconnect if the connection drops\n\n = \n \n break\n = \n \nMCP servers are not dead. But blindly enabling them bloats your context, which leads to higher cost and worse performance. Unlike Agent Skills, MCP servers don't come with progressive disclosure out of the box. It is your responsibility to select the tools needed for the task at hand. Here are two proven patterns on how to correctly use MCP servers and avoid the bloat.
\nMCP servers stay opt-in. Servers are referenced in your prompt with an @mention, and the agent resolves, fetches, and injects the tools before the model request.
![\"Explicit](\"https://www.philschmid.de/static/blog/use-mcp-servers/mcp-inline.png\")
Writing @github or @slack in a prompt triggers the agent to:
@mention to a registered MCP server URLtools[] array of the API requestNothing loads unless requested. The tool surface stays small.
\nTypeScript
\n// Pseudo-code\n\nWhen to use this: MCP usage is occasional and user-driven. Someone asks a question that needs Slack or GitHub data, they @mention the server, and it gets pulled in for that request only. Keeps costs low and avoids tool noise on tasks that don't need external integrations.
MCP servers are declared in subagents definition and automatically available at runtime alongside native tools like read_file or run_command.
![\"Subagent](\"https://www.philschmid.de/static/blog/use-mcp-servers/mcp-subagent.png\")
Each subagent has its own configuration that declares its model, tools,... and optional MCP servers. You can scope MCP servers to specific tools using allowed_tools.
YAML
\n# Pseudo-code: code_reviewer.md\n---\nname: code-reviewer\nmodel: gemini-3-flash\nmcp_servers:\n - url: https://github-mcp.example\n allowed_tools:\n - list_pulls\n - list_reviews\n - get_diff\n---\nYou are a code reviewer. Review open PRs...\nThe main agent registers subagents. When invoked, each subagent connects to its declared MCP servers and merges those tools with native ones.
\nTypeScript
\n// Pseudo-code\n \n// Register subagents as tools for the orchestrator\nconst codeReviewer = ;\nconst slackMonitor = ;\n \n\nallowed_tools gives you least-privilege scoping without forking the MCP server.
When to use this: The use case dictates the tools. A code review agent always needs GitHub, a support agent always needs Zendesk. The MCP servers are part of what the agent is, not something a user opts into per request. allowed_tools keeps each agent scoped to what it actually needs.
Skills have become one of the most used extension points in agents. They’re flexible, easy to make, and simple to distribute.
\nBut this flexibility also makes it hard to know what good and what works. What type of skills are worth making? What's the secret to writing a good skill? When do you share them with others?
\nI have been using skills extensively with many of them in active use. Here are some tips I've learned along the way.
\nA skill is a folder with a SKILL.md file and, optionally, some helper files:
my-skill/\n├── SKILL.md ← The only required file\n├── scripts/ ← Reusable code the agent can run\n├── references/ ← Docs the agent reads when needed\n└── assets/ ← Templates, images, or files used in output\nA Skill consists out of 3 layers:
\nname and description Frontmatter Goes into every prompt and tells the agent when to use the skill.SKILL.md Markdown instructions (below frontmatter), that tells the agent how to do the task.scripts/, references/, and assets/ folders.Skills always fall into two categories:
\nThe description in your SKILL.md is the trigger mechanism. If it's vague, the agent won't know when to activate the skill. If it's too broad, the skill fires on every request. Be specific about what the skill does AND when to use it. Include both the "what" and the "when" in the description. The body of the skill only loads after the skill triggers.
| ❌ Too vague | \n✅ Specific and actionable | \n
|---|---|
| "Helps with documents" | \n"Create, edit, and analyze .docx files, use for tracked changes, comments, formatting, or text extraction" | \n
| "API helper" | \n"Use when writing code that calls the Gemini API for text generation, multi-turn chat, image generation, or streaming" | \n
I have seen 50% improvements just by improving the description.
\nAgents are smart. Your job is to tell it what it doesn't already know. Research shows longer, more comprehensive with too much context actually hurts performance.
\ninteractions.create()", not "The Interactions API is the recommended approach." The first is an instruction, the second is trivia the agent won't act on.Don't dump everything into one file. The agent loads information in layers:
\nSKILL.md, name + descriptionSKILL.md body (keep under 500 lines)If your skill covers multiple topics (e.g. AWS vs. GCP deployment), split them into separate reference files. The agent only reads the one it needs. This saves context for the actual task.
\nTip: If a reference file is longer than 500 lines, add a table of contents with "line hints" at the top so the agent can quickly find what it needs.
\nA common mistake in creating skills is turning the skill into a step-by-step worklow: "Step 1: Read the file. Step 2: Parse the JSON. Step 3: Extract the fields…" When you dictate every step, you take away their ability to adapt, recover from errors, or find better approaches. Describe what you want, not the path to get there.
\nTell the agent what to achieve:
\nProvide constraints, not procedures:
\nIf exact steps matter, write a script. If the task is fragile and doing step 3 before step 2 breaks everything, that's not a skill problem, it's a scripting problem.
\nThink about when the skill should not fire. A description like "Use for any coding task" will hijack every request.
\n\n\n"Use when working with PDF files. Do NOT use for general document editing, spreadsheets, or plain text files."
\n
Testing both "should trigger" and "shouldn't trigger" cases are essential. Without you'll optimize the skill in one direction.
\nDon't ship a skill without evaluating. Each run might behave differently, so a single check isn't enough.
\nRun evals without the skill. If they pass, the model has absorbed the skill's value and the skill is no longer necessary. Retire it. This is especially true for capability skills, as models improve, the gap narrows.
\nFor a practical step-by-step eval workflow, see Practical Guide to Evaluating and Testing Agent Skills.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-04-13T00:00:00+00:00", "date_published": "2026-04-13T00:00:00+00:00", "external_url": "https://www.philschmid.de/agent-skills-tips", "id": "urn:aggr:item:baf0baf6cd0673a1234e40ae43159e4e09a55c5f", "summary": "8 Tips for Writing Agent Skills. Know What a Skill Is, Nail the Description, Write Instructions, Keep It Lean, Set the Right Level of Freedom, Don't Skip Negative Cases, Test It Before You Ship It, Know When to Retire a Skill.", "tags": [ "programming" ], "title": "8 Tips for Writing Agent Skills", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-04-13-8-tips-for-writing-agent-skills/" }, { "content_html": "Google's Gemma 4 family of open models is now available through the Gemini API and Google AI Studio. Built from the same research behind Gemini 3, these models bring advanced reasoning, native function calling, multimodal understanding, and 256K context windows to an open, Apache 2.0-licensed package you can run anywhere.
\nTwo models are available through the Gemini API today:
\ngemma-4-26b-a4b-itgemma-4-31b-itGemma 4 models handle function calling, structured JSON output, and system instructions at the model level rather than through prompt engineering. The 31B dense model currently ranks as the #3 open model on the Arena AI text leaderboard, with the 26B MoE model at #6, competing with models 20x their size.
\nKey capabilities:
\nThe fastest way to try Gemma 4 is Google AI Studio. Select gemma-4-26b-a4b-it or gemma-4-31b-it from the model picker, type a prompt, and start chatting. You can test system instructions, adjust temperature, and experiment with multimodal inputs all through the browser. No API key or code required.
Or click Get Code to export Python, JavaScript, or cURL snippets from any conversation.
\nInstall the Python SDK:
\nBash
\n\nSet your API key as an environment variable. You can create one at aistudio.google.com/apikey.
\nBash
\n\nGenerate text with Gemma 4:
\nPython
\n\n \n = \n \n = \n\nPass a system instruction to set the model's behavior:
\nPython
\n\n\n \n = \n \n = \n\nThe SDK provides a chat interface that tracks conversation history automatically:
\nPython
\n\n \n = \n = \n \n = \n\n \n = \n\nPass an image alongside your text prompt:
\nPython
\n\n\n \n = \n \n\n = \n \n = \n\nDefine tools as function declarations. The model decides when to call them:
\nPython
\n\n\n \n# Define the function declaration\n = \n \n = \n = \n = \n \n = \n \n# The model returns a function call instead of text\n\n = ...\n \n \nGround Gemma 4 responses in real-time web data with Google Search:
\nPython
\n\n\n \n = \n \n = \n \n\n \n# Access grounding metadata for citations\n\n \nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-04-07T00:00:00+00:00", "date_published": "2026-04-07T00:00:00+00:00", "external_url": "https://www.philschmid.de/gemma-4-gemini-api", "id": "urn:aggr:item:f020c2a21d1e9c228316728ff1b21e7d2f9a9fb3", "summary": "Learn how to use Gemma 4 with the Gemini API and Google AI Studio.", "tags": [ "programming" ], "title": "How to use Gemma 4 with the Gemini API and Google AI Studio", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-04-07-how-to-use-gemma-4-with-the-gemini-api-and-google-ai-studio/" }, { "content_html": "I read three recent technical reports from Moonshot AI's Kimi K2.5 paper, Cursor's Composer 2 report and blog post, and Chroma's Context-1 write-up.
\nEach report introduces something distinct. Kimi K2.5 trains an Agent Swarm where the model learns to decompose tasks into parallel sub-agents through RL. Cursor's Composer 2 uses self-summarization to handle long coding sessions and runs real-time RL from production traffic. Chroma's Context-1 teaches the model self-editing context: actively pruning retrieved documents to free up space for further search.
\nAll three use reinforcement learning with similar methodology:
\nPaper: Kimi K2.5: Visual Agentic Intelligence
\nKimi K2.5 is Moonshot AI's multimodal model with a 1T parameter / 32B active MoE architecture. Its most distinctive feature is Agent Swarm, a framework where the model learns to dynamically decompose tasks into parallel subtasks and dispatch them to sub-agents. The parallelization strategy is learned through reinforcement learning, not hand-coded.
\nMost agentic systems execute sequentially: think → tool call → observe → think → tool call. Agent Swarm breaks this by training the model to spawn parallel sub-agents.
\nThe architecture has two roles:
\ncreate_subagent and assign_task tools.This decoupling solves the credit assignment problem. In end-to-end co-optimization, a correct final answer could mean the orchestrator decomposed well, or that a sub-agent got lucky. Freezing sub-agents and treating their outputs as environmental observations means only the orchestrator's coordination logic gets optimized.
\nKimi additionally introduces "critical steps" to measure computational cost in a parallel setting, analogous to the critical path in a computation graph. Instead of counting total steps across all agents, critical steps measure the longest execution chain. For each stage, the cost is the max steps among all parallel sub-agents. Total critical steps sum these maxima. This incentivizes the orchestrator to balance work across sub-agents (reducing the longest branch) rather than just maximizing concurrency.
\nTraining a reliable parallel orchestrator requires reward design. The PARL reward has three components:
\nr_perf): Did the task succeed? This is the primary signal.r_parallel): Incentivizes sub-agent instantiation to prevent "serial collapse", a local optimum where the orchestrator defaults to single-agent execution and never explores parallel strategies.r_finish): Rewards completed subtasks to prevent "spurious parallelism", a reward-hacking behavior where the orchestrator spawns many sub-agents without meaningful task decomposition, just to collect r_parallel.The auxiliary reward coefficients are annealed to zero over training, so the final policy optimizes solely for performance.
\nAt inference time, the model receives a task and decides whether and how to parallelize:
\ncreate_subagent.assign_task in parallel. Sub-agents execute concurrently with independent context windows.The decision to parallelize is not hard-coded. On simple tasks, the model works sequentially. On complex multi-source research tasks, it spins up many parallel agents. The training distribution encourages this: synthetic prompts emphasize either "wide search" (many independent information sources) or "deep search" (multiple reasoning branches with delayed aggregation). The prompts don't instruct the model to parallelize. They create tasks where parallelism helps.
\nAgent Swarm reduces inference latency by up to 4.5× while improving accuracy. On BrowseComp, it achieves 78.4% (vs. 60.6% single-agent), surpassing GPT-5.2 Pro (77.9%). On WideSearch, item-level F1 improves from 72.8% to 79.0%. Agent Swarm also functions as proactive context management, decomposing tasks into isolated sub-agent contexts avoids the context overflow that plagues long sequential runs.
\nKimi K2.5's RL recipe has several other components worth noting:
\nPapers: Composer 2 Technical Report, Improving Composer through real-time RL
\nComposer 2 is Cursor's in-house model for agentic software engineering. It can read and edit files, run shell commands, search codebases, and browse the web. The goal is to solve real coding tasks autonomously.
\nComposer 2 trains inside the exact same Cursor harness that users interact with: same tools, same prompt format, same system message, same file context. They maintain a shadow deployment of the Cursor backend during training so that tool behaviors (like semantic search) work identically to production.
\nPublic benchmarks like SWE-bench use simplified environments and over-specified prompts. Real developer requests are under-specified, messy, and admit multiple valid solutions. Training in the production harness on problems drawn from actual Cursor usage means Composer 2 learns to handle the real distribution.
\nThey also built CursorBench, an internal evaluation suite of tasks pulled from actual coding sessions by their engineering team. CursorBench tasks have a median of 181 lines changed (compared to 7-10 on SWE-bench) and much shorter, more ambiguous prompts. This benchmark co-evolves with the product: as users push agents harder, the benchmark grows more complex.
\nComposer 2's RL infrastructure consists of four decoupled services:
\nThe policy gradient algorithm is a variant close to GRPO, applied single-epoch (no prompt is trained on twice), with full-parameter updates. They remove the length standardization term from standard GRPO (it introduces length bias) and skip advantage normalization by standard deviation (it over-amplifies noise when all rollouts in a group have equal correctness).
\nCursor trains additional Multi-Token Prediction (MTP) layers for speculative decoding. These layers are self-distilled: they learn to predict the exact logit distribution of the main LM head at each token position. The MTP layers are initialized from scratch, trained on the same data mix, and then jointly fine-tuned during the long-context and SFT phases before RL. This yields 2-3x faster inference with minimal quality degradation.
\nReal coding tasks can be long. The agent might make dozens of tool calls, read many files, and iterate over hundreds of turns. To keep the model effective within a limited context window, Composer 2 uses self-summarization: each rollout can involve multiple generations chained together by summaries. The final outcome reward applies to all tokens in the chain, so good summaries that preserve critical information get reinforced, while poor summaries that lose key context get downweighted.
\nThe model learns when and how to summarize as a natural part of RL training. For hard tasks, it often summarizes multiple times.
\nBeyond simulated RL, Cursor also runs what they call real-time RL, taking actual production inference tokens and extracting training signal from user interactions. The cycle works like this:
\nThe entire loop takes about five hours, so they can ship an improved checkpoint multiple times per day. Keeping the loop fast keeps data nearly on-policy: the model generating the data is (almost) the model being trained.
\nPaper: Context-1: Training a Self-Editing Search Agent
\nContext-1 is a 20B parameter agentic search model trained to do one thing well: find documents. It doesn't answer questions, it returns a ranked set of supporting documents to a downstream reasoning model. The core innovation is self-editing context. The model learns to selectively discard retrieved documents that are no longer relevant, freeing up context space for further exploration.
\nReal-world agentic search tasks are hard to get at scale, you need multi-hop queries with known ground-truth document sets. Chroma built a synthetic generation pipeline across four domains: web, finance (SEC filings), legal (USPTO patents), and email (Epstein files + Enron corpus as distractors).
\nEach task follows the same structure:
\nThe verification step matters because "is this document relevant?" is an unreliable question for an LLM. Instead, they use an extraction-based pipeline: the LLM extracts matching quotes from both document and clue, then a deterministic check verifies the quotes actually appear in the source. This achieves >80% alignment with human labels across all domains.
\nContext-1 operates with four tools:
\nsearch_corpus(query): hybrid BM25 + dense retrieval with RRF fusion and reranking.grep_corpus(pattern): regex search.read_document(doc_id): read specific chunks.prune_chunks(chunk_ids): remove irrelevant chunks from context.The harness enforces a fixed token budget (e.g., 32k tokens). After each turn, current usage is included: [Token usage: 14,203/32,768]. Past a soft threshold, the harness suggests pruning. Past a hard cutoff, all tools except prune_chunks are blocked, the model must prune or conclude.
Deduplication is handled at the harness level: every chunk ID seen across all prior searches is tracked and passed as exclusion filters, so subsequent searches always surface new information.
\nWhen the model prunes, the harness removes chunks from the model's view but preserves the full unpruned trajectory for reward computation. This lets the reward credit the agent for documents it encountered during search even if they were later pruned.
\nA concrete search trajectory might look like this:
\nSearch 1 → context at 11k/32k tokens \nSearch 2 → context at 24k/32k (past soft limit, ephemeral "consider pruning" note) \nSearch 3 → BLOCKED: 10.4k needed, only 8.5k free \nprune(doc_02, doc_05, doc_06) → context drops to 13k/32k \nSearch 3 → context at 23k/32k (past soft limit again) \nprune(doc_08) → context at 18k/32k \nSearch 4 → final results \nAnswer\nTraining proceeds in two stages:
\nSFT warmup: Generate trajectories using Kimi K2.5 as the inference backend, then filter by recall quality. High-recall trajectories are kept in full. Low-recall ones are included at diminishing rates. A small fraction (up to 5%) of zero-recall trajectories serve as negative examples.
\nRL with CISPO: Fully on-policy training using CISPO (Clipped Importance-Sampled Policy Optimization), a variant of GRPO. 128 queries per step, 8 rollouts each, yielding 1,024 trajectories per step. Groups where all 8 rollouts get the same reward are discarded (no gradient signal under within-group normalization).
\nThe reward is carefully constructed:
\nTrain where you deploy. All three teams invest heavily in making training environments match production. Cursor uses a shadow production backend. Kimi runs sub-agents in the same harness. Chroma runs search against real databases. This minimizes the gap between training performance and real-world performance.
\nContext management is a first-class problem. Agent contexts grow over time. Cursor uses self-summarization. Kimi shards context across parallel sub-agents. Chroma teaches the model to discard irrelevant chunks. Different solutions, same underlying constraint.
\nReward design is iterative. Every team describes discovering and fixing reward hacking behaviors. Cursor's model learned to emit broken tool calls. Kimi's orchestrator fell into "serial collapse" or "spurious parallelism." Chroma's agent converged to single-search-then-quit. Each time: observe the degenerate behavior, understand the incentive, add a targeted reward or penalty.
\nPublic benchmarks are not enough. Cursor explicitly argues that SWE-bench scores don't correlate well with real-world utility. They built CursorBench from actual user sessions. Chroma built synthetic benchmarks across four domains. Kimi uses both public and in-house evaluation. If you're building a vertical model, you need vertical evaluation.
\nSmaller, purpose-trained models compete with frontier models. Chroma's 20B model matches frontier-scale LLMs on retrieval at a fraction of the cost and 10x the speed. Composer 2 achieves Pareto-optimal cost-accuracy tradeoffs compared to much larger API models. Domain-specific RL training closes the gap that raw parameter count creates.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-03-28T00:00:00+00:00", "date_published": "2026-03-28T00:00:00+00:00", "external_url": "https://www.philschmid.de/kimi-composer-context", "id": "urn:aggr:item:88e35f6aff9c3c6798a3cf4a44e21d6ec858fd2f", "summary": "Learn the unique ways how Kimi, Cursor, and Chroma train agentic models with RL.", "tags": [ "programming" ], "title": "How Kimi, Cursor, and Chroma Train Agentic Models with RL", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-03-28-how-kimi-cursor-and-chroma-train-agentic-models-with-rl/" }, { "content_html": "Most useful agents need several tools at once. A DevOps agent that spots a CVE also needs to file a ticket for it. Until recently, you had orchestrate those handoffs yourself or only use custom tools without built-in tools.
\nTool combination in the Gemini API now lets you combine built-in tools with custom function declarations in a single request. Gemini decides which tools to call, in what order, and circulates context between them automatically.
\nBelow are two examples of tool combination.
\nCombine Google Search, URL Context, and a custom function in one request. The model searches the web, reads a page, and calls your function without you specifying the order.
\nGet a Gemini API key and install the SDK:
\npip install google-genai\nDefine a custom function and pass it alongside built-in tools:
\nPython
\n\n \n = \n \n = \n \n = \n \n\n \n \n \n \n \n# google_search_call...\n# google_search_result...\n# google_search_call...\n# google_search_result...\n# thought...\n# function_call...\n# Function: file_incident\n# Arguments: {'summary': 'Unauthenticated remote code execution (RCE) vulnerability in React Server Components (RSC) affecting React 19.0, 19.1.x, and 19.2.0. The vulnerability (React2Shell) stems from insecure deserialization in the RSC "Flight" protocol, allowing an attacker to execute arbitrary code via a crafted HTTP request. Patches are available in React versions 19.0.1, 19.1.2, and 19.2.1.', 'severity': 'Critical', 'cve_id': 'CVE-2025-55182'}\nContext circulation preserves built-in tool results across turns. Pass previous_interaction_id and follow-up questions can reason over earlier results without re-executing tools. The model can still make new tool calls if it needs fresh data or thinks it needs to.
Python
\n\n \n = \n \n# Turn 1: URL Context reads Philipp's about page\n = \n\n \n# Turn 2: Follow-up uses context from turn 1\n = \n\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-03-24T00:00:00+00:00", "date_published": "2026-03-24T00:00:00+00:00", "external_url": "https://www.philschmid.de/tool-combo", "id": "urn:aggr:item:94c9d42f9c4758e504800d1b2c36c3a19eafd592", "summary": "Learn how to combine built-in tools and function calling in the Gemini Interactions API.", "tags": [ "programming" ], "title": "Combine Built-in Tools and Function Calling in the Gemini Interactions API", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-03-24-combine-built-in-tools-and-function-calling-in-the-gemini/" }, { "content_html": "Nano Banana 2 (gemini-3.1-flash-image-preview) is our latest image generation model. It runs at Gemini Flash speed with the capabilities of Nano Banana Pro's world knowledge, precise text rendering, and subject consistency across up to 14 reference images. It comes with a new Grounding with Google Image Search feature that lets the model retrieve real images from the web and use them as visual context during generation.
This Developer Guide uses the Interactions API, our next-generation unified interface for Gemini, to build a personalized Japan travel brochure. In 4 code snippets, you'll go from a simple text prompt to a search-grounded, personalized image—compositing a real person into a photorealistic scene of Kyoto, informed by live Google Search results.
\nBash
\n\n\nYou can create and manage all your Gemini API Keys from Google AI Studio.
\nDescribe what you want, get an image back. Nano Banana 2 handles photorealism, text rendering, and complex compositions without any tools.
\nPython
\n\n\n \n = \n \n = \n \n = \n\n \n![\"01_kyoto_poster.png\"](\"https://www.philschmid.de/static/blog/nano-banana-2-interactions-api/01_kyoto_poster.jpg\")
Adding a google_search tool makes the model search the web for factual information before generating.
Python
\n\n\n \n = \n \n = \n \n = \n\n \n![\"02_kyoto_map.jpg\"](\"https://www.philschmid.de/static/blog/nano-banana-2-interactions-api/02_kyoto_map.jpg\")
The model now searches for actual Kyoto landmarks and uses their real names and descriptions. It still imagines what they look like, though. For visual accuracy, the model needs to see reference images.
\nNano Banana 2 adds Image Search as a new search type alongside Web Search, controlled via the search_types parameter. The model retrieves actual photographs from Google Image Search and uses them as visual references during generation.
With text-only search, the model knows Fushimi Inari has orange torii gates. With Image Search, it sees exactly what they look like.
\nPython
\n\n\n \n = \n \n = \n \n = \n\n \n![\"03_fushimi_inari.jpg\"](\"https://www.philschmid.de/static/blog/nano-banana-2-interactions-api/03_fushimi_inari.jpg\")
You can also use image_search on its own without web_search:
Python
\n=\nThe brochure now has factually accurate, visually faithful renderings of real landmarks. Next: adding a real person to the scene.
\nWe can combine a reference photo of a person with text instructions and search grounding. Nano Banana 2 preserves the person's appearance from the reference image in the generated output.
\nWe provide a headshot and ask the model to place the person into the Arashiyama bamboo grove, with Image Search ensuring the scene matches the real location.
\nPython
\n\n\n \n = \n \n = \n \n = \n \n = \n\n \n![\"04_arashiyama_portrait.jpg\"](\"https://www.philschmid.de/static/blog/nano-banana-2-interactions-api/04_arashiyama_portrait.jpg\")
This single API call combines three Nano Banana 2 capabilities:
\nsearch_types parameter with image_search requires the gemini-3.1-flash-image-preview model specifically.Thanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-03-16T00:00:00+00:00", "date_published": "2026-03-16T00:00:00+00:00", "external_url": "https://www.philschmid.de/nano-banana-2-interactions-api", "id": "urn:aggr:item:7bdebc282c5721b8b6b49a0dfd7adef4b9c24879", "summary": "Learn how to use the Gemini Interactions API to build a personalized Japan travel brochure with Nano Banana 2.", "tags": [ "programming" ], "title": "Developer Guide: Nano Banana 2 with the Gemini Interactions API", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-03-16-developer-guide-nano-banana-2-with-the-gemini-interactions/" }, { "content_html": "What if you could optimize a model overnight without any ML experience? What if an AI agent runs hundreds of training experiments autonomously, keeping only the improvements? That is the idea behind autoresearch. You give an AI agent a training script and a metric. It edits the code, runs a short experiment, checks if the metric improved, keeps or discards, and repeats. Karpathy used it to squeeze 11% more speed out of his GPT-2 training. Tobi Lütke, Shopify's CEO, trained a 0.8B model overnight that outscored his previous 1.6B.
\nAn LLM agent that can edit training code, runs a short experiment, checks if the metric improved, and repeats, without human involvement.
\n![\"mermaid\"](\"https://www.philschmid.de/static/blog/autoresearch/mermaid.png\")
Design constraints that make it work:
\ntrain.py only. Data prep and evaluation are locked down.Roughly 12 experiments/hour. Around 100 overnight.
\nBoth examples below are small-scale and early. The setups are minimal, the models are small, and neither is a controlled study. But they show where this is headed.
\nKarpathy pointed autoresearch at nanochat, his already well-tuned GPT-2 training codebase. Over two days the agent ran ~700 experiments and found ~20 real improvements. Stacked together, time-to-GPT-2 dropped from 2.02 to 1.80 hours (11% faster). The agent caught things Karpathy had missed:
\nAll improvements transferred from depth-12 to depth-24 models.
\nTobi adapted the pattern for a query-expansion model for the QMD Open source project:
\nA smaller model outperformed one twice its size. He then pointed the same loop at a reranker and beat that baseline too.
\nThe loop hinges on your eval. If the metric is gamed or leaky, the model looks better on paper and fails in production. Your eval set must be held out completely — the agent never touches it, never trains on it, never sees it during optimization. You need a training script the agent can modify, training data (manually labeled or synthetic), and a metric that reflects what the model will actually do in production.
\nWhen experiments run 100x faster than a human can manage, your eval becomes the bottleneck. Static benchmarks get saturated. Build your eval pipeline so it can evolve, refresh from real production data, harder edge cases.
\nThe pattern fits search ranking, product categorization, clinical NER, fraud scoring, contract extraction, intent classification, and similar tasks. Small models work well — training runs finish in minutes and improvements transfer when you scale up. Open models like Gemma are a good starting point: small enough for a single GPU, performant for production tasks, commercially licensed.
\nGEPA (Genetic-Pareto prompt evolution, ICLR 2026) optimizes prompts using genetic evolution and reflection. It works on frozen models and APIs where you cannot change weights.
\nAutoresearch optimizes weights — it modifies training code, architectures, and hyperparameters to produce a better model.
\nFor teams building domain SLMs, both layers compound.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-03-10T00:00:00+00:00", "date_published": "2026-03-10T00:00:00+00:00", "external_url": "https://www.philschmid.de/autoresearch", "id": "urn:aggr:item:ec2dc2f8e2e6db0f78fa0b0b5c576bf96b90487e", "summary": "Autoresearch lets an AI agent run hundreds of model training experiments overnight. Learn how it works, early results from Karpathy and Shopify, and how to apply it.", "tags": [ "programming" ], "title": "How Autoresearch will change Small Language Models adoption", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-03-10-how-autoresearch-will-change-small-language-models-adoption/" }, { "content_html": "Skills are everywhere. SkillsBench counted over 47,000 unique skills across 6,300+ repositories. Everyone is writing skills. Almost nobody is testing them and most of them are AI-generated. Skills might get "vibe-checked" with a handful of manual runs, then shipped. You wouldn't ship code without tests, but why ship skills without evals?
\nThis is a practical guide to fixing that. You will learn how to define success criteria, build a lightweight eval harness, and iterate on it. I used the same steps to create and evaluate the Gemini Interactions API skill, taking it from 66.7% to 100% pass rate.
\nAgent Skills are folders of instructions, scripts, and resources that augment an agent's capabilities without retraining or fine-tuning the model. Skills follow a progressive disclosure pattern and require at minimum a SKILL.md file with:
name and description in YAML that the agent uses to decide whether to apply the skill. This is the most important piece — if it's vague, the skill won't trigger reliably.scripts/, examples/, references/ that the agent can consult during execution.Markdown
\n\n \n \nSkills fall into two categories that matter for testing:
\nBefore writing a single eval, write down what "success" means in measurable terms. Grade outcomes, not paths. Agents find creative solutions, and you don't want to penalize an unexpected route to the right answer.
\nFor our Interactions API skill, the concrete checks were: correct SDK import (from google import genai), current model IDs (not deprecated gemini-2.0-flash), uses interactions.create() not generateContent, and uses previous_interaction_id for multi-turn. Most of these are regex-checkable.
This guide assumes you already have a skill. It won't cover skill creation. Before writing any eval code, trigger the skill manually a few times. Use explicit invocations like Use the {skill name} to do X or your agent's built-in trigger (e.g. /skill {name}, $skill-name). Watch where it breaks. These first runs are not about scoring; they are about surfacing hidden assumptions. Does it assume dependencies that aren't there? Does it skip steps the user would expect?
Every fix you make during manual runs, adding a missing install command, correcting a path, tightening the description, becomes a concrete check you can automate later. Manual iteration is not wasted work.
\nFor a single skill, 10–20 prompts are enough to begin with. Each prompt tests a specific scenario and declares its own success criteria:
\nJSON
\n\nEach test declares which checks should pass via expected_checks. For agents, each test case may need its own success criteria.
I created 17 tests across five categories: core capabilities (7), guardrails for deprecated models (4), extended features not in the skill's inline examples (3), and negative controls (2). Tests span Python (12) and TypeScript (5).
\nImportant: Don't skip negative tests. A skill with too-broad description could trigger on every coding prompt.
\nTest the skill the same way the agent experiences it, e.g. through the CLI. In our case we used Gemini CLI inside a python script.
\nPython
\n\n = \n = \n = \n return \nEach check becomes a small function, using regex against the extracted code, returning a boolean.
\nPython
\n# Does the code use the correct SDK?\n\n \n return \n return \n \n# Does the code use a current model?\n = \n \n\n return not \nRegister all checks that the harness can dispatch by check_id from the prompt set, then loop through each test case:
Python
\n = \n \n\n = \n = \n \n = \n \n = \n return \nWe iterated on the Interactions API skill across ~20 test cases going from 66.7% → 100% pass rate. The two fixes that mattered most were rewriting the skill description to better match user intent (not API terminology) and replacing passive deprecation warnings with explicit instructions. The description change alone fixed 5 of 7 failures.
\nNote: We didn't use LLM-as-judge for the Interactions API skill.
\nSkills may require qualitative checks, such as code structure, naming conventions, design quality, or whether the output follows the intended patterns. These are hard to capture with regex or file existence checks alone. For example, the frontend-design skill demands "distinctive, production-grade interfaces". For such skills, you can use a second model-assisted pass.
\nUse structured output to constrain the LLM's response to a typed schema so that results become parseable and trackable.
\nPython
\n\n\n \n = \n \n\n : \n : = \n \n\n : \n : = \n : = \n : = \n : = \n : = \n \n = \n \n = \n \n = \n\n\n = \n \nUse LLM grading selectively since deterministic checks are fast, while LLM grading adds cost and latency.
\ninteractions.create()" works. "The Interactions API is the recommended approach" doesn't.If you want to go deeper, here are more resources:
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-03-04T00:00:00+00:00", "date_published": "2026-03-04T00:00:00+00:00", "external_url": "https://www.philschmid.de/testing-skills", "id": "urn:aggr:item:4ac5bcba6db4cc67572fc22a381dac06848c4d4b", "summary": "Learn how to systematically test and improve agent skills using deterministic checks and a real-world Gemini API example.", "tags": [ "programming" ], "title": "Practical Guide to Evaluating and Testing Agent Skills", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-03-04-practical-guide-to-evaluating-and-testing-agent-skills/" }, { "content_html": "An AGENTS.md (or GEMINI.md) file is the single highest-leverage configuration point for coding agents. It's injected into every conversation, acting as the agent's onboarding document to your codebase. But research shows that doing it wrong actively hurts performance. Here's how to do it right, backed by data from "Evaluating AGENTS.md" (ETH Zurich, 2025) and practical experience from HumanLayer.
uv instead of pip, bun instead of npm). Tools mentioned in AGENTS.md get used 160x more often than unmentioned ones (§4.3)./init or let the agent write its own AGENTS.md. The data shows this hurts more than it helps.agent_docs/running_tests.md, agent_docs/database_schema.md) and list them in AGENTS.md with brief descriptions so the agent reads them only when relevant.file:line locations rather than embedding code snippets that will go stale.Thanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-02-24T00:00:00+00:00", "date_published": "2026-02-24T00:00:00+00:00", "external_url": "https://www.philschmid.de/writing-good-agents", "id": "urn:aggr:item:9dc59ed6b47deca4cb11de9f119027a70175d11b", "summary": "Learn what to include, what to skip, and how to structure your AGENTS.md for best results.", "tags": [ "programming" ], "title": "Writing a Good AGENTS.md", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-02-24-writing-a-good-agents-md/" }, { "content_html": "When people talk about AI agents "closing the loop," they usually mean the agent verifies its own work before responding, which can be confusing as "If the tool call loop is hardcoded, how does the model decide to verify its work?"
\nShort answer: The loop is hardcoded. What the model does inside the loop is not.
\nEvery agent framework runs roughly the same cycle: model generates → if tool call, execute it → feed result back → model generates again → repeat until the model returns text (no more tool calls). That loop is scaffolding. It's the same for every agent.
\nThe difference is what the model chooses to do within that loop. A model that "closes the loop" doesn't need a special loop, it uses the existing one to call verification tools before deciding it's done.
\nThe inner loop is what happens during a single task, before the agent responds to user with a text. The agent writes code, runs the tests, reads the error, fixes the edge case, re-runs and only then generates a text response for the user. It's the tight feedback cycle between the model and its tools.
\nExample: You ask "fix the failing test in auth.ts."
A weak agent edits the file and says "Done!" A strong agent edits → creates tests → runs the tests → sees a failure → fixes the edge case → runs again → sees green → then responds. Same infrastructure, different behavior.
\nBad agent — edits and stops:
\n![\"bad-agent\"](\"https://www.philschmid.de/static/blog/inner-loop-vs-outer-loop/bad-agent.png\")
Good agent — verifies before responding:
\n![\"bad-agent\"](\"https://www.philschmid.de/static/blog/inner-loop-vs-outer-loop/good-agent.png\")
Both agents use the exact same hardcoded tool loop. The difference is the good agent chose to create/call verfications before responding. Nothing forced it to.
\nWhere does that choice come from? Today, mostly from the system prompt ("always run tests after code changes"). Increasingly, from post-training — e.g. test pass/fail as a reward signal, so the model internalizes the verify step.
\nThe outer loop is what happens across multiple turns/sessions between the user and the agent over time. The user gives the agent a task, it works on it (inner loop), returns a result, and then the user comes back with the next task. The question is: did the agent learn anything from the last turn?
\nWithout persistent memory, every turn is a clean slate. The agent that failed on pagination yesterday will fail on it again today.
\n![\"outer-loop\"](\"https://www.philschmid.de/static/blog/inner-loop-vs-outer-loop/outer-loop.png\")
Almost no agent does this natively today. The outer loop requires persistent state, skills, rules files, or notes that survive between turns and sessions. Some early examples:
\nThe inner loop is about reliability within a task. The outer loop is about getting smarter over time.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-02-20T00:00:00+00:00", "date_published": "2026-02-20T00:00:00+00:00", "external_url": "https://www.philschmid.de/inner-loop-vs-outer-loop", "id": "urn:aggr:item:591b944acfd643cf4e88aca8542eb1a002c2ba02", "summary": "Most agent frameworks share the same hardcoded tool loop; what differs is how the model uses it. This post explains the inner loop—an agent verifying its own work within a task—and the outer loop—an agent carrying lessons across tasks via…", "tags": [ "programming" ], "title": "Agents: Inner Loop vs Outer Loop", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-02-20-agents-inner-loop-vs-outer-loop/" }, { "content_html": "If you spend enough time with agents, you feel a difference. The good ones feel like working with a colleague. They plan, check, and try to catch their own mistakes and only then tell you it's done. They try to close their own loop. The bad ones end with you typing "review your work and find the errors" or "did you test it?".
\nFrom what I can tell, two things separate the good ones from the bad:
\nSelf-awareness: Does the agent have an understanding of what it is (an LLM), what tools it has, and how to write its own instructions?
\nClosing the loop. Can the agent verify its own work before responding to the user?
\nThis is not about consciousness or sentience. In the context of AI agents, self-awareness refers to operational self-knowledge—a model's functional understanding of its own:
\nResearchers call this situational awareness (Berglund et al., 2023) or meta-cognition. Anthropic's introspection research (Oct 2025) showed that Claude can tell whether outputs were artificially injected or generated by it. Not in a human sense, but as a mechanism that produces better meta-task behavior.
\nWhat it is not: Understanding why it generates specific tokens, having subjective experience, or "truly knowing itself". Models are trained to behave as if they understand their role. That behavioral consistency produces better meta-task outputs. Useful, not magic.
\nClosing the loop means the agent doesn't return to you until it has verified its own work against external signals. Not based on its own judgment of correctness, but based on external signals, e.g. from the compiler, the test runner, the filesystem, and other LLM calls.
\nExamples of closing the loop today:
\nnpm test, reads the error trace, fixes the edge case, re-runs until green—before telling you it's done.Most loop-closing in production today is scaffolding, verification steps built around the agent, not initiated by it. Spotify's background coding agent is a good example. After 1,500+ agent-generated PRs, they built independent verifiers (Maven builds, test runners, formatters) exposed to the agent as a single MCP tool. The agent doesn't know what the verifier does internally; it just knows that it needs to call "verify" and then gets pass/fail feedback. On top of that, an LLM judge compares the final diff against the original prompt to catch scope creep (agents "getting creative" and refactoring code nobody asked them to touch).
\nThe research frontier is going towards spontaneous verification. A recent paper (intrinsic self-critique) from DeepMind showed that a single LLM can check its own plan step-by-step against the task rules, no external signal. It just asks itself "does this step actually follow the rules?" after each action, and if not, rewrites the plan. That alone jumped planning success from 50% to 89%. It hints at a future where verification isn't bolted on, it's just what the model does.
\nToday, closing the loop works when the scaffolding is right. Tools like OpenClaw are the first pushing this further, persisting learnings across sessions through memory and a "heartbeat" mechanism that proactively checks and acts on tasks without being prompted. But it's still fragile. The agent does it when the setup tells it to, not because it knows it should.
\nSame for delegation. Gemini CLI, Claude Code, and Codex CLI all let a lead agent spin up sub-agents in parallel. The mechanism works, but the communication is unreliable, the lead agent doesn't always know what context to pass, what instructions to write, or what the sub-agent is missing. That's where self-awareness will make a difference.
\nWe will see an increase in consistency. Self-awareness that's trained in. Loop-closing that's default behavior. And eventually, continuous learning. The inner loop catching mistakes within a task, the outer loop carrying lessons across sessions. When agents do all of this reliably—understand themselves, verify their work, remember what they learned—then they start being collaborators we can trust.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-02-17T00:00:00+00:00", "date_published": "2026-02-17T00:00:00+00:00", "external_url": "https://www.philschmid.de/closing-the-loop", "id": "urn:aggr:item:663c29b449dd4f57a61c5d4b390945cd094fccd4", "summary": "What makes some AI agents feel like collaborators while others need constant babysitting? Two capabilities matter: self-awareness — does the agent understand what it is and how to use its tools — and closing the loop — can it verify its…", "tags": [ "programming" ], "title": "Can We Close the Loop in 2026?", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-02-17-can-we-close-the-loop-in-2026/" }, { "content_html": "Building AI agents that can truly "see" requires more than just passing images in prompts. Coding agents need to read screenshots of UIs they're building. Computer use agents need to capture and analyze browser screenshots. Research agents need to process charts and diagrams from documents.
\nMultimodal function calling allows tools to return images the model can process natively, similar to how you pass images in prompts. Instead of describing what's in a file, your tool returns the actual image and Gemini 3 processes it natively.
\nThis guide shows you how Multimodal Function Calling works using the Interactions API. The Interactions API is a unified interface for interacting with Gemini models and agents. It simplifies state management and tool orchestration, making it easy to build multi-turn agentic workflows.
\n![\"chart\"](\"https://www.philschmid.de/static/blog/interactions-multimodal-fc/chart.png\")
The key difference from standard function calling is in the function_result returns the actual image data. Gemini 3 processes this content natively—describing images, analyzing documents, or using visual information to make decisions.
JSON
\n\nLet's build a tool that reads images from disk and returns them to the model for description. Get a Gemini API key and install the Google GenAI SDK:
\nBash
\n\nSet your API key:
\nBash
\n\nPython
\n = \nPython
\n\n\n \n\n """Read image from disk and return as function result content."""\n = \n \n \n return \n \n = \n = \n \n = \n return \nPython
\n\n \n = \n \n = \n=f\n \n# Step 1: Send the initial request with our tool\n = \n \n# Step 2: Handle the function call\n\n \n \n = \n \n # Step 3: Send the image back to the model\n = \n \n # Step 4: Get the model's description\n \n \n \nThe interaction flow has four steps:
\nfunction_call specifying which tool to use and argumentsPython
\n# The function_call output from the model looks like:\n\n \n# Your function_result response looks like:\n\nMultimodal function calling unlocks multimodal agentic use cases. By returning images directly in your function_result, you give Gemini 3 the ability to see what your tools see.
This pattern extends beyond file reading. You can build tools that capture screenshots, fetch images from APIs, render charts, or process scanned documents. Combined with the Interactions API's stateful sessions, you have everything you need to build sophisticated visual agents.
\nNext steps:
\nThis API is in Beta, and we want your feedback! We're actively listening to developers to shape the future of this API. What features would help your agent workflows? What pain points are you experiencing? Please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-02-13T00:00:00+00:00", "date_published": "2026-02-13T00:00:00+00:00", "external_url": "https://www.philschmid.de/interactions-multimodal-fc", "id": "urn:aggr:item:09cdde15981da94925849a65f6271f407ffc4cbd", "summary": "Multimodal function calling allows tools to return images the model can process natively, similar to how you pass images in prompts. Instead of describing what's in a file, your tool returns the actual image and Gemini 3 processes it…", "tags": [ "programming" ], "title": "Multimodal Function Calling with Gemini 3 and Interactions API", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-02-13-multimodal-function-calling-with-gemini-3-and-interactions/" }, { "content_html": "This guide demonstrates how to use the new Gemini Deep Research agent via the Interactions API to perform complex research tasks, generate images based on the findings, and translate the results.
\nPreview: The Gemini Deep Research Agent is currently in preview. The Deep Research agent is exclusively available using the Interactions API. You cannot access it through generate_content.
Python
\n! - --\nPython
\n\n\n\n \n = \nThe following example shows how to start a research task in the background and poll for results. This is the standard way to interact with the Deep Research agent.
\nPython
\n = \n \n = \n \n\n# Poll for results\n\n = \n \n \n break\n == :\n \n break\n \n \n \n \n \n \n\n# The Evolution of Google Tensor Processing Units: A Historical and Architectural Analysis\n \n### Key Points\n* **Origins in Necessity:** The Tensor Processing Unit (TPU) project was initiated in 2013 when Google engineers realized that existing CPU and GPU infrastructure could not sustain the computational demands of ubiquitous voice search and deep learning applications without doubling datacenter footprints.\n* **Architectural Paradigm Shift:** Unlike the Von Neumann architecture of CPUs, TPUs utilize a **systolic array** architecture. This design mimics the flow of blood through the heart, allowing data to flow through thousands of arithmetic logic units (ALUs) without accessing memory for intermediate results, drastically increasing efficiency for matrix multiplications.\n...\nDeep Research supports streaming to receive real-time updates on the research progress (thinking_summaries). You can steer the agent's output by providing specific promting instructions.
Python
\n# Define a complex prompt with formatting instructions\n = \n \n \n = \n \n = None\n = None\n = \n \n\n \n = \n \n \n \n = \n \n \n \n += \n == :\n \n \n == :\n \n \n \n \n = \n\n \n\n \nFull Report: https://gist.github.com/philschmid/68c5afacfd3a3555ca834ba27415ba88
\nThe Interactions API supports stateful interactions, which allows you to continue the conversation after the agent returns the final report by using the previous_interaction_id.
You can continue the conversation after the agent returns the final report using a model to visualize the results using Nano Banana Pro or translate the results using Gemini Flash.
\nPython
\n\n\n \n\n = \n \n = \n \n\n = \n \n![\"timeline\"](\"https://www.philschmid.de/static/blog/gemini-deep-research-getting-started/timeline.jpg\")
Python
\n\n = \n \n\n# Strategischer Bericht: Die Google TPU-Landschaft\n \n## Zusammenfassung\nGoogle beschleunigt die Entwicklung eigener KI-Chips (TPUs), um NVIDIA herauszufordern. Die letzten 12 Monate brachten mit **Trillium (v6)** und **Ironwood (v7)** große Fortschritte. Für Entwickler wird es einfacher: Dank neuer Software-Tools laufen PyTorch-Modelle jetzt fast ohne Änderungen auf TPUs. Bis 2027 will Google über 5 Millionen TPUs im Einsatz haben.\n... \nThis API is in Beta, and we want your feedback! We're actively listening to developers to shape the future of this API. What features would help your agent workflows? What pain points are you experiencing? Please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-02-02T00:00:00+00:00", "date_published": "2026-02-02T00:00:00+00:00", "external_url": "https://www.philschmid.de/gemini-deep-research-getting-started", "id": "urn:aggr:item:3d50126a93f2abba7c98cab50094f5c12fb47ea8", "summary": "Learn how to use the new Gemini Deep Research agent via the Interactions API to perform complex research tasks, generate images based on the findings, and translate the results.", "tags": [ "programming" ], "title": "Getting Started with Gemini Deep Research API", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-02-02-getting-started-with-gemini-deep-research-api/" }, { "content_html": "With the explosion of AI coding agents—Claude Code, Gemini CLI, OpenCode, Goose, Codex CLI, we've entered an era where every major AI lab ships their own terminal-based coding assistant. But there is one problem. Each agent has its own events or outputs.
\nThe Agent Client Protocol (ACP) is an an open standard abstracts the events and outputs of AI agents and provides a common interface for editors to interact with them. Similar to MCP but for agent to client (UI) communication.
\nACP is a JSON-RPC 2.0 protocol with a simple premise:
\n![\"acp\"](\"https://www.philschmid.de/static/blog/acp-overview/acp.png\")
| Method | \nDirection | \nPurpose | \n
|---|---|---|
initialize | \nClient → Agent | \nNegotiate protocol version & capabilities | \n
session/new | \nClient → Agent | \nCreate new conversation session | \n
session/load | \nClient → Agent | \nResume existing session | \n
session/prompt | \nClient → Agent | \nSend user message | \n
session/set_mode | \nClient → Agent | \nSwitch agent operating mode | \n
fs/read_text_file | \nAgent → Client | \nRead file (includes unsaved edits) | \n
fs/write_text_file | \nAgent → Client | \nWrite/create file | \n
terminal/create | \nAgent → Client | \nStart shell command | \n
terminal/output | \nAgent → Client | \nGet command output | \n
terminal/wait_for_exit | \nAgent → Client | \nWait for command completion | \n
terminal/kill | \nAgent → Client | \nTerminate running command | \n
terminal/release | \nAgent → Client | \nClean up terminal | \n
session/request_permission | \nAgent → Client | \nRequest user approval for action | \n
| Notification | \nDirection | \nPurpose | \n
|---|---|---|
session/update | \nAgent → Client | \nStream progress, messages, tool calls | \n
session/cancel | \nClient → Agent | \nAbort current operation | \n
| Update Type | \nDescription | \nContent | \n
|---|---|---|
plan | \nAgent's execution plan | \nList of steps with status | \n
agent_message_chunk | \nStreamed text response | \nText content block | \n
user_message_chunk | \nEcho of user input | \nText content block | \n
thought_message_chunk | \nAgent's reasoning | \nText content block | \n
tool_call | \nNew tool invocation | \nID, title, kind, status | \n
tool_call_update | \nTool progress/result | \nStatus change, content | \n
available_commands | \nSlash commands | \nList of available commands | \n
mode_change | \nMode switch | \nNew mode info | \n
![\"tool\"](\"https://www.philschmid.de/static/blog/acp-overview/tool.png\")
![\"protocol\"](\"https://www.philschmid.de/static/blog/acp-overview/full.png\")
![\"terminal\"](\"https://www.philschmid.de/static/blog/acp-overview/terminal.png\")
![\"permission\"](\"https://www.philschmid.de/static/blog/acp-overview/permission.png\")
The Interactions API is a unified interface for building with Gemini models and agents. It simplifies the development of agentic applications by handling server-side state management, tool orchestration, and long-running tasks.
\nWith a single endpoint, you can:
\n![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
Get a Gemini API key and install the Google GenAI SDK:
\nBash
\n\nSet your API key:
\nBash
\n\nAt its simplest, the Interactions API works like a standard chat completion. You provide a model and an input string. By default, interactions are stored (store=True), allowing you to reference them later.
Python
\n\n \n = \n \n = \n \n\n# Output: Quantum entanglement is a phenomenon where particles become linked...\nOne of the most powerful features of this API is server-side state management. You do not need to append messages to a list and send the full history back to the server every time.
\nUse previous_interaction_id to continue a conversation.
Note: Only conversation history is preserved. Parameters like tools or generation_config are interaction-scoped and must be re-declared if needed.
Python
\n\n \n = \n \n = \n\n \n = \n \n\n# Output: You are a software engineer.\nFor client-managed history, see Stateless conversations.
\nBecause state is managed by ID, you can "fork" a conversation by referencing an older interaction ID with a different prompt.
\nPython
\n# Branch off from Turn 1 with a different topic\n = \n \n\n# Output: Your name is Alice.\nGemini models natively understand and generate multiple content types. You can pass text, images, audio, or PDF documents in a single interaction. This example uses a remote image URL.
\nPython
\n\n \n = \n \n = \n \n\nFor audio, video, and document (PDF) understanding, see Multimodal understanding.
\nPython
\n\n\n \n = \n \n = \n \n\n \n \n \nFor audio generation see Multimodal generations.
\nTools extend the model's capabilities by letting it call external functions or services. The API includes ready-to-use tools like Google Search, and lets you define custom tools as JSON schemas. The model decides when to call them based on the conversation:
\nPython
\n\n \n = \n \n = \n \n = \n\n \nOther built-in tools include Code Execution and Computer Use.
\nPython
\n\n \n = \n \n# Define a tool\n = \n \n# Send request with tool\n = \n \n# Handle tool call\n\n \n # Execute your function (mocked here)\n # result = get_weather(output.arguments\n = \n \n # Return result to model\n = \n \nFor code execution, URL context, and MCP servers, see Agentic capabilities.
\nBeyond models, the Interactions API provides access to specialized agents. Deep Research executes multi-step research tasks, synthesizing information from multiple sources into comprehensive reports.
\nAgents run asynchronously with background=True. Poll the interaction status to retrieve results:
Python
\n\n\n \n = \n \n = \n \n \n# Poll for completion\n\n = \n \n \n \n \n \n break\n in :\n \n break\n \n \nFor more details, see Deep Research.
\nThe Interactions API supports many more complex workflows. Check the API Reference for details on:
\nthinking_level for Gemini 2.5 and 3.0 models to handle complex reasoning.This API is in Beta, and we want your feedback! We're actively listening to developers to shape the future of this API. What features would help your agent workflows? What pain points are you experiencing? Please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-01-22T00:00:00+00:00", "date_published": "2026-01-22T00:00:00+00:00", "external_url": "https://www.philschmid.de/interactions-api-quickstart", "id": "urn:aggr:item:37f41fd61d4d955b207530878cff6ee4e110b062", "summary": "The Interactions API is a unified interface for building with Gemini models and agents. It simplifies the development of agentic applications by handling server-side state management, tool orchestration, and long-running tasks.", "tags": [ "programming" ], "title": "Gemini Interactions API Quick Start", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-01-22-gemini-interactions-api-quick-start/" }, { "content_html": "The Model Context Protocol (MCP) has exploded roughly 1 year ago, everyone rushed to build MCP servers. The hype was real. Yet, most MCP servers disappoint. Most developers blame the protocol. The protocol feels like it's dying on social media.
\nBut enterprise adoption tells a different story. Companies are deploying MCP servers. The integrations are live. But the results? Disappointing. Why?
\nDevelopers treat MCP like a REST API wrapper. MCP is a User Interface for Agents. Different users, different design principles.
\nThe protocol works fine. The servers don't.
\nThis post breaks down why MCP servers fail, six best practices for building ones that work, and how Skills and MCP complement each other.
\nMCP (Model Context Protocol) provides a universal connection between LLMs and external tools, data sources, and services. Before MCP, every integration required custom connectors. MCP standardizes three primitives:
\nsearch_documents, create_issue)The idea behind MCP is to build your MCP server once, and use with any agent.
\nMCP servers are not thin wrappers around your existing API. A Good REST API is not a good MCP server. We assume just because LLMs are "smart", they can use APIs as a human would. Thats wrong.
\nREST API design principles advocate for composability, discoverability, flexibility, and stability. Small endpoints developers combine. Self-documenting schemas.
\nThese principles work for human developers. They don't work for AI agents. MCP is a User Interface for AI agents. Different users, different design principles.
\n| REST API Principles | \nDevelopers | \nAgents | \n
|---|---|---|
| Discovery | \nCheap (read docs once) | \nExpensive (schema in every request) | \n
| Composability | \nMix and match small endpoints | \nMulti-step tool calls, slow iteration | \n
| Flexibility | \nMany options > more flexibility | \nComplexity leads to hallucination | \n
Additionally, MCP servers are not data dump services. Throwing large scale raw data at an agent bloats its context window.
\nA Practical Example: Order Tracking
\nYou're building an agent that tracks orders.
\nAs a human developer, you read the API docs once, write a script calling GET /users, GET /orders, GET /shipments in sequence, debug it, deploy it.
A bad MCP server exposes those three endpoints as tools. The agent must load all three tool description, make 3 round-trips and stores all intermediate results in conversation history.
\nA good MCP server exposes one tool, track_order(email). The tool calls all three internally and returns "Order #12345 shipped via FedEx, arriving Thursday." Same outcome, one call, outcome oriented.
MCP is a User Interface for a non-human user. Same product thinking, different user.
\nTo fix your server, you need to curate the experience. Here are the six best practices for building a good MCP server.
\nTrap: Converting REST endpoints 1:1 into MCP tools.
\nFix: Design tools around what the user/agent wants to achieve.
\nDon't expose three separate tools for order status:
\nget_user_by_email()list_orders(user_id)get_order_status(order_id)This forces the agent to orchestrate three round-trips. Instead of three atomic tools, give the agent one high-level tool: track_latest_order(email). Do the orchestration in your code, not in the LLM's context window.
Trap: Using complex nested dictionaries or configuration objects as arguments.
\nFix: Top-level primitives and constrained types.
\n| ❌ Bad | \n✅ Good | \n
|---|---|
def search_orders(filters: dict) -> list | \ndef search_orders(email: str, status: Literal["pending", "shipped", "delivered"] = "pending", limit: int = 10) -> list | \n
| Agent guesses the structure | \nClear, typed, constrained | \n
| Hallucinates keys, misses required fields | \nLiteral constrains choices, defaults reduce decisions | \n
Trap: Empty docstrings, generic error messages.
\nFix: Every piece of text is part of the agent's context.
\nDocstrings are instructions. Specify:
\nError Messages are context too! If a tool call fails, don't throw a Python exception. Return a helpful string: "User not found. Please try searching by email address instead." The agent sees the error as an observation and uses your instruction to self-correct in the next turn.
\nTrap: Exposing everything your API can do. Returning everything the API returns.
\nFix: Design for discovery, not exhaustive exposure.
\nAgents operate under tight context constraints. Every tool description, every response payload, every error message, it all competes in the context window.
\nBuild for discovery. The agent should find the right tool quickly and get an actionable response.
\nTrap: Generic names like create_issue or send_message.
Fix: Service-prefixed, action-oriented names.
\nYour MCP server runs alongside others. If GitHub and Jira both have create_issue, the agent guesses. Pattern: {service}_{action}_{resource}. Examples: slack_send_message, linear_list_issues, sentry_get_error_details.
Note: Some MCP clients prefix tools with the server name automatically.
\nTrap: Returning hundreds of records.
\nFix: Paginate with metadata.
\nlimit parameter (default 20–50)has_more, next_offset, total_countYou're building an MCP server for Gmail. Here's how most developers approach it versus the agent-first design.
\nPython
\n# Reading an email requires 2 tools + understanding nested types\n ...\n ...\n \n# Sending an email requires base64-encoding a MIME message\n ... # raw = base64url RFC 2822\n \n# Creating a draft has nested message object\n ...\nProblems: Agent must construct {"raw": base64(...)} and parse nested payload.body.data. Generic names.
Python
\n# Reading: 2 flat tools with curated returns\n ...\n ...\n \n# Writing: Simple examples, not including cc, bcc.\n ...\nSkills package instructions, metadata, and resources that the agent uses when relevant. They're filesystem-based with progressive disclosure. Each skill has a name, description, which is part of the Model context. The Model/Agent can then decide to use the skill or not based on the given context.
\n| Level | \nWhen Loaded | \nToken Cost | \nContent | \n
|---|---|---|---|
| Metadata | \nStartup | \n~100 tokens | \nname and description from YAML | \n
| Instructions | \nWhen triggered | \n<5k tokens | \nSKILL.md body | \n
| Resources | \nAs needed | \nUnlimited | \nScripts via bash | \n
Skills don't add tool definitions directly, but can include scripts that run locally to provide capabilities similar to MCP servers. The difference is that skills leverage generic execution tools (like bash) rather than defining new tool schemas. Leading to potentially more steps and discovery needed. MCP provides a structured interfaces. The model gets parameter validation, typed responses.
Neither is better than the other. It depends on the use case and your preference. In company contexts, MCP servers shine when teams expose their own services. Skills complement this by teaching agents when and how to combine those tools for specific workflows. Use both.
\nWhen building an MCP Server you are not building infrastructure, you are building an interface for AI agents.
\nMCP is a User Interface for AI agents. Build it like one.
\nOther resources: Block's MCP Playbook, GitHub's Security Guide, FastMCP AI Engineering Summit.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-01-21T00:00:00+00:00", "date_published": "2026-01-21T00:00:00+00:00", "external_url": "https://www.philschmid.de/mcp-best-practices", "id": "urn:aggr:item:a568ca25112cea344749b623ea79d29b49a508df", "summary": "The Model Context Protocol (MCP) has exploded roughly 1 year ago, everyone rushed to build MCP servers. The hype was real. Yet, most MCP servers disappoint. Most developers blame the protocol. The protocol feels like it's dying on social…", "tags": [ "programming" ], "title": "MCP is Not the Problem, It's your Server: Best Practices for Building MCP Servers", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-01-21-mcp-is-not-the-problem-it-s-your-server-best-practices-for/" }, { "content_html": "Generating images is easy. Getting clean transparent backgrounds for actual use—stickers, overlays, print-on-demand—is harder than it should be.
\nThis guide shows how to generate production-ready transparent stickers using the Gemini Interactions API. The trick: generate on chromakey green, strip it with HSV detection.
\nWorkflow:
\nPrerequisites:
\npip install google-genai pillow scipyGEMINI_API_KEY environment variableNotebook available on GitHub
\nWhen you control generation, prompting for a specific background color beats running another model. Faster, cheaper, and more predictable.
\nFirst, let's install the required dependencies and set up the Gemini client.
\nPython
\n# Install dependencies (uncomment if needed)\n# !pip install google-genai pillow scipy\n \n\n\n\n\n\n\n \n# Initialize the Gemini client\n = \n \n# Model for image generation\n = \nWe'll create helper functions using HSV color space for more robust green screen detection that catches all shades of green.
\nPython
\n\n """Decode base64 image data to PIL Image."""\n = \n return \n \n \n\n """Convert RGB array to HSV array efficiently."""\n # Normalize RGB to 0-1 range\n = / 255.0\n \n , , = , , \n \n = \n = \n = - \n \n # Hue calculation\n = \n \n # When max == r\n = & \n = % 360\n \n # When max == g\n = & \n = \n \n # When max == b\n = & \n = \n \n # Saturation calculation\n = \n = / \n \n # Value is just max\n = \n \n return \n \n \n\n """\n Remove green screen using HSV color space for better detection.\n \n HSV is much better for detecting color ranges because it separates\n hue (color) from saturation (intensity) and value (brightness).\n """\n # Convert to RGBA if not already\n \n = \n \n # Convert to numpy array\n = \n = \n \n # Convert to HSV\n = \n , , = , , \n \n # Calculate hue distance (accounting for circular nature of hue)\n = \n = \n \n # Create mask for green pixels\n # Green if: hue is in range AND saturation is high enough AND value is high enough\n = \n \n # Apply morphological cleanup to remove edge artifacts\n \n \n \n # Dilate the mask to catch anti-aliased edge pixels\n \n = \n \n # Optionally erode back (removes isolated noise)\n \n = \n \n # Make green pixels transparent\n = \n = 0\n = \n \n return \n \n \n\n """\n Aggressive green removal that detects any pixel where green dominates.\n \n This catches even darker or lighter greens, shadows with green tint, etc.\n """\n \n = \n \n = \n , , = , , \n \n # A pixel is "green" if green channel significantly exceeds red and blue\n # This catches all shades of green including shadows\n = + 1 # +1 to avoid division by zero\n = / \n \n # Also check that green is the dominant channel\n = & \n \n # Combined mask\n = & \n \n # Expand mask to catch edge pixels\n \n \n = \n \n # Apply transparency\n = \n = 0\n = \n \n return \n \n \n\n """\n Clean up semi-transparent edge pixels by making them fully transparent or opaque.\n \n This removes the "halo" effect from anti-aliased edges.\n """\n \n return \n \n = \n = \n \n # Make semi-transparent pixels either fully transparent or fully opaque\n = 0\n = 255\n \n = \n return \n \n \n\n """Save image as PNG with transparency preserved."""\n \n = \n \n \n \nThe key is to instruct Gemini to generate the image with a chromakey green background. We use specific prompts to ensure clean edges and no green spill.
\nPython
\n \n \n\n """\n Load an image from a file path and return it as a content block for the API.\n """\n \n \n \n # Determine mime type from file extension\n , = \n \n # Default to JPEG if unknown\n = \n \n # Read and base64 encode the image\n \n = \n \n return \n \n \n\n """\n Generate a sticker-style image with chromakey green background.\n """\n # Optimized prompt for chromakey extraction\n = f\n \n \n \n \n # Build the input content\n # When input_images are provided, create a list with image content blocks followed by text\n \n \n = \n \n \n \n # Add the text prompt as the final content block\n \n \n # No input images, just use the text prompt directly\n = \n \n # Call Gemini Interactions API\n = \n \n # Extract the generated image\n \n \n \n return \n \n \nLet's put it all together: generate, remove green screen with HSV detection, apply aggressive cleanup, and save.
\nPython
\n\n """\n Complete workflow to create a transparent sticker.\n \n Uses a multi-pass approach:\n 1. Generate with optimized chromakey prompt\n 2. HSV-based green removal (catches color range)\n 3. Aggressive green removal (catches remaining green tints)\n 4. Edge cleanup to remove halos\n """\n \n \n # Step 1: Generate image with green screen\n = \n \n # Optionally save raw image for debugging\n \n = \n \n \n \n # Step 2: HSV-based green removal\n \n = \n \n # Step 3: Skip aggressive removal (disabled - causes speckles in subject)\n # transparent_image = remove_green_screen_aggressive(...)\n \n # Step 4: Clean up any semi-transparent edge artifacts\n \n = \n \n # Step 5: Save as PNG\n \n \n return \n \n \nLet's generate some example stickers!
\nPython
\n = \n \n = \n| Raw (Green Screen) | \nProcessed (Transparent) | \n
|---|---|
![\"raw\"](\"https://www.philschmid.de/static/blog/generate-stickers/cat_raw.png\") | \n![\"processed\"](\"https://www.philschmid.de/static/blog/generate-stickers/cat.png\") | \n
Python
\n = \n = \n \n = \n| Raw (Green Screen) | \nProcessed (Transparent) | \n
|---|---|
![\"raw\"](\"https://www.philschmid.de/static/blog/generate-stickers/developer_raw.png\") | \n![\"processed\"](\"https://www.philschmid.de/static/blog/generate-stickers/developer.png\") | \n
The generated PNG files have proper alpha channels and can be used in:
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-01-19T00:00:00+00:00", "date_published": "2026-01-19T00:00:00+00:00", "external_url": "https://www.philschmid.de/generate-stickers", "id": "urn:aggr:item:14b148d9f506921686680eacc183d34da02a8e3b", "summary": "Learn how to generate transparent PNG stickers using Nano Banana Pro and the Gemini Interactions API, featuring chromakey green background removal with HSV detection.", "tags": [ "programming" ], "title": "Transparent PNG Stickers with Nano Banana Pro and Gemini interactions API", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-01-19-transparent-png-stickers-with-nano-banana-pro-and-gemini/" }, { "content_html": "\n\nAlternative Version: This guide uses the Interactions API (Beta). For the
\nGenerateContentAPI approach, see Building Agents with Gemini 3.
It seems complicated, when you watch an AI agent edit multiple files, run commands, handle errors, and iteratively solve a problem, it feels like magic. But it isn't. The secret to building an agent is that there is no secret.
\nThe core of an Agent is surprisingly simple: It is a Large Language Model (LLM) running in a loop, equipped with tools it can choose to use.
\nIf you can write a loop in Python, you can build an agent. This guide walks you through building a CLI Agent using the Interactions API, which handles conversation state on the server so you don't have to.
\nTraditional software workflows are prescriptive and follow predefined paths (Step A -> Step B -> Step C), Agents are System that uses an LLM to dynamically decide the control flow of an application to achieve a user goal.
An agent generally consists of these core components:
\nwhile loop that allows the model to: Observe → Think → Act → Observe again, until the task is complete.![\"agent\"](\"https://www.philschmid.de/static/blog/building-agents/agent.png\")
"The Loop" of nearly every agent is an iterative process:
\nget_weather) to the model using a structured format.tool use containing the tool name and arguments.tool use, executes the actual code or API call, and captures the result.tool response) back to model. The model uses this new information to decide the next step, either calling another tool or generating the final response.The new Interactions API simplifies this loop by managing the conversation state on the server, so you don't have to manually track history.
\nWhy Interactions API?
\nprevious_interaction_id instead of re-sending historyCurrently in Beta. See the official documentation for details.
\nLet's build an agent step-by-step, progressing from basic text generation to a functional CLI agent using Gemini 3 Flash and the Interactions API.
\nPrerequisites: Install the SDK (pip install google-genai) and set your GEMINI_API_KEY environment variable (Get it in AI Studio).
The first step is to create a baseline interaction with the LLM, for us Gemini 3 Flash. We are going to create a simple Agent class abstraction to structure our code, which we will extend throughout this guide. We will first start with a simple chatbot that maintains a conversation history using the Interactions API's server-side state.
\nPython
\n\n \n\n\n = \n = \n = None\n \n\n = \n = \n return \n \n = \n = \n \n\n# Output: Berlin, Munich, Cologne \n = \n \n\n# Output: Munich is the capital of Bavaria and is known for its Oktoberfest.\nThis is not an agent yet. It is a standard chatbot. It maintains state but cannot take action, has no "hands or eyes".
\nTo start turning this an agent, we need Tool Use or Function Calling. We provide the agent with tools. This requires defining the implementation (the Python code) and the definition (the schema the LLM sees). If the LLM believes that tool will help solve a user's prompt, it will return a structured request to call that function instead of just text.
\nWe are going to create 3 tools, read_file, write_file, and list_dir. A tool Definition is a JSON schema that defines the name, description, and parameters of the tool, wrapped in a structure the Interactions API understands.
Best Practice: Use the description fields to explain when and how to use the tool. The model relies heavily on these to understand when and how to use the tool. Be explicit and clear.
Python
\n\n\n \n = \n \n = \n \n = \n \n\n \n return \n \n\n """Writes a file with the given contents."""\n \n \n return True\n \n\n """Lists the contents of a directory."""\n = \n return \n \n = \nNow we integrate the tools and function calls into our Agent class.
Python
\n\n \n\n\n = \n = \n = None\n = \n \n\n = \n = \n return \n \n = \n \n = \n\n \n \n# Output: Function call: list_dir with arguments {'directory_path': '.'}\nGreat! The model has successfully called the tool. Now, we need to add the tool execution logic to our Agent class and the loop return the result back to the model.
\nAn Agent isn't about generating one tool call, but about generating a series of tool calls, returning the results back to the model, and then generating another tool call, and so on until the task is completed.
\nThe Agent class handles the core loop: intercepting the function_call, executing the tool on the client side, and sending back the function_result using the previous_interaction_id to maintain context. Instead of manually re-sending the entire conversation history, the Interactions API uses previous_interaction_id to chain interactions. The server maintains the context, so you only need to send new inputs (user messages or tool results).
Python
\n# ... Code for the tools and tool definitions from Step 2 should be here ...\n \n\n \n\n\n = \n = \n = None\n = \n = \n \n \n = \n = \n \n = \n \n \n \n \n \n = \n \n = \n \n \n \n \n # If there were tool calls, send results back to the model\n \n return \n \n return \n \n = \n \n = \n\n# Output: [Function Call] list_dir({'directory_path': '.'})\n# [Function Response] ['.venv', ... ]\n# There. Your current directory contains: `LICENSE`,\nCongratulations. You just built your first functioning agent using the Interactions API.
\nNow we can run our agent in a simple CLI loop. It takes surprisingly little code to create highly capable behavior.
\nPython
\n# ... Code for the Agent, tools and tool definitions from Step 3 should be here ...\n \n = \n \n\n\n = \n \n break\n \n = \n \nBuilding an agent is no longer magic; it is a practical engineering task. As we've shown, you can build a working prototype in under 100 lines of code.
\nThe Interactions API is designed to make agent development even simpler. Our goal is to provide a unified, developer-friendly interface that handles the complexity of state management, tool orchestration, and long-running tasks—so you can focus on building great agent experiences.
\nWe believe a cleaner, more consistent API foundation enables the broader ecosystem to thrive, and open source libraries, frameworks, and tooling can build more powerful abstractions on top. Making it easier for developers at every level to build with Gemini.
\nThis API is in Beta, and we want your feedback! We're actively listening to developers to shape the future of this API. What features would help your agent workflows? What pain points are you experiencing? Let us know.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-01-14T00:00:00+00:00", "date_published": "2026-01-14T00:00:00+00:00", "external_url": "https://www.philschmid.de/building-agents-interactions-api", "id": "urn:aggr:item:0c54920e141a3e324d90d39eeb5710cde79e825c", "summary": "Learn how to build AI agents using the new Gemini Interactions API, featuring server-side state management and simplified tool orchestration.", "tags": [ "programming" ], "title": "Building Agents with the Gemini Interactions API", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-01-14-building-agents-with-the-gemini-interactions-api/" }, { "content_html": "\n\nUpdated January 2026 for v0.3.0 — New 3-subcommand architecture (
\ninfo,grep,call), connection pooling daemon, and tool filtering support.
The Model Context Protocol (MCP) is an open standard for connecting AI agents to external tools, APIs, and data sources. However, as the ecosystem grows with more powerful MCP servers, developers and agent builders are hitting a scaling bottleneck: context window bloat.
\nmcp-cli is a lightweight CLI that allows dynamic discovery of MCP, reducing token consumption while making tool interactions more efficient for AI coding agents.
\nKey Features:
\nmcp-cli compiles to a single standalone binary.mcp-cli grep "*mail*" -d.allowedTools and disabledTools config.Every MCP server comes with tool definitions schemas describing what each tool does, its parameters, types, and descriptions. Traditional MCP integration loads all of these schemas upfront into the agent's context window.
\nHere's what that looks like in practice:
\n![\"comparison\"](\"https://www.philschmid.de/static/blog/mcp-cli/comparison.jpeg\")
| Setup | \nTokens Used | \n
|---|---|
| 6 MCP servers, 60 tools | \n~47,000 tokens | \n
| After dynamic discovery | \n~400 tokens | \n
That is a 99% reduction in MCP-related token usage for this scenario.
\nWhen working with multiple MCP servers (GitHub, databases, browser automation—tool), definitions quickly consume a third or more of the effective context. This leads to:
\nThe solution is dynamic context discovery. Instead of loading everything upfront (static context), agents pull in only the information they need, when they need it.
\n![\"\"](\"https://www.philschmid.de/static/blog/mcp-cli/dynamic-discovery.jpg\")
mcp-cli implements this pattern for MCP:
mcp-climcp-cli info github search_repositoriesmcp-cli call github search_repositories '{"query": "mcp"}'Most Interactions only use a handful of tools, yet static loading consumes tokens for every tool definition. Dynamic discovery inverts this, you pay only for what you use.
\nmcp-cli allows dynamic discovery of MCP while making tool interactions more efficient for AI coding agents.
\nBash
\n# binary install\n | \n \n# requires bun install\n\nCreate mcp_servers.json in your current directory or ~/.config/mcp/:
JSON
\n\nBash
\n# List all servers and tools\n\n# deepwiki\n# • read_wiki_structure\n# • read_wiki_contents\n# • ask_question\n#\n# filesystem\n# • read_file\n# • read_text_file\n# • read_media_file\n# • read_multiple_files\n# ...\n \n# With descriptions\n# mcp-cli -d\nBash
\n# View tool schema first\n\n# Tool: read_file\n# Server: filesystem\n#\n# Description:\n# Read the complete contents of a file as text.\n#\n# Input Schema:\n# {\n# "type": "object",\n# "properties": {\n# ...\nBash
\n# Call the tool\n\nMCP CLI allows the model to generate commands that chain multiple tool calls together.
\nBash
\n# Using a heredoc (no '-' needed with call subcommand)\n\n \n# From a file\n | \n \n# Using jq to build complex JSON\n | \n \n# Find all TypeScript files and read the first one\n | | \nYou can restrict which tools are available from a server using allowedTools and disabledTools in your config:
JSON
\n\nRules:
\nallowedTools: Only tools matching these patterns are available (supports glob: *, ?)disabledTools: Tools matching these patterns are excludeddisabledTools takes precedence over allowedToolsExamples:
\nBash
\n# Only allow read operations\n\n \n# Allow all except destructive operations\n\n \n# Combine: allow file operations but disable delete\n\n\nBy default, mcp-cli uses lazy-spawn connection pooling to avoid repeated MCP server startup latency:
MCP_DAEMON_TIMEOUTControl via environment:
\nBash
\nMCP_NO_DAEMON=1 MCP_DAEMON_TIMEOUT=120 mcp-cli is designed to be used with AI Agents and bash tools. There are two main ways to integrate it:
Add this to your AI agent's system prompt for direct CLI access:
\nXml
\n## MCP Servers\n \nYou have access to MCP servers via the `mcp-cli` CLI.\n \nCommands:\n \n```bash\nmcp-cli info # List all servers\nmcp-cli info # Show server tools \nmcp-cli info # Get tool schema\nmcp-cli grep "" # Search tools\nmcp-cli call # Call tool (stdin auto-detected)\nmcp-cli call '{}' # Call with JSON args\n```\n \n**Both formats work:** `info ` or `info /`\n \nWorkflow:\n \n1. **Discover**: `mcp-cli info` to see available servers\n2. **Inspect**: `mcp-cli info ` to get the schema\n3. **Execute**: `mcp-cli call '{}'` with arguments\n \n### Examples\n \n```bash\n# Call with inline JSON\nmcp-cli call github search_repositories '{"query": "mcp server"}'\n \n# Pipe from stdin (no '-' needed)\necho '{"path": "./file"}' | mcp-cli call filesystem read_file\n \n# Heredoc for complex JSON\nmcp-cli call server tool <For AI agents that support Agent Skills an upcoming standard for extending coding agents. mcp-cli ships with a ready-to-use skill definition.
\nCreate mcp-cli/SKILL.md in your agent's skills directory:
Markdown
\n\n \n \n \n \n \n1. 2.3.4. \n \n \n \n \n- ---The AI Agent space is moving incredibly fast. mcp-cli tries to solve context tool discovery problem turning it into an iterative, just-in-time process. It allows agents to access a massive ecosystem of shared capabilities without the context bloat of static integration. Whether used within a Skill or as a standalone utility, it ensures your agent spends its tokens on reasoning, not configuration.
The project is open source and designed to fit into existing workflows. Give it a try and contribute at github.com/philschmid/mcp-cli.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-01-09T00:00:00+00:00", "date_published": "2026-01-09T00:00:00+00:00", "external_url": "https://www.philschmid.de/mcp-cli", "id": "urn:aggr:item:54aa265c5d4dc1a397025077c08ad129491fb14a", "summary": "Mcp-cli is a lightweight CLI that allows dynamic discovery of MCP, reducing token consumption while making tool interactions more efficient for AI coding agents.", "tags": [ "programming" ], "title": "Introducing MCP CLI: A way to call MCP Servers Efficiently", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-01-09-introducing-mcp-cli-a-way-to-call-mcp-servers-efficiently/" }, { "content_html": "We are at a turning point in AI. For years, we focused only on the model. We asked how smart/good the model was. We checked leaderboards and benchmarks to see if Model A beats Model B.
\nThe difference between top-tier models on static leaderboards is shrinking. But this could be an illusion. The gap between models becomes clear the longer and more complex a task gets. It comes down to durability: How well a model follows instructions while executing hundreds of tool calls over time. A 1% difference on a leaderboard cannot detect the reliability if a model drifts off-track after fifty steps.
\nWe need a new way to show capabilities, performance and improvements. We need systems that proves models can execute multi-day workstreams reliably. One Answer to this are Agent Harnesses.
\nAn Agent Harness is the infrastructure that wraps around an AI model to manage long-running tasks. It is not the agent itself. It is the software system that governs how the agent operates, ensuring it remains reliable, efficient, and steerable.
\nIt operates at a higher level than agent frameworks. While a framework provides the building blocks for tools or implements the agentic loop. The harness provides prompt presets, opinionated handling for tool calls, lifecycle hooks or ready-to-use capabilities like planning, filesystem access or sub-agent management. It is more than a framework, it comes with batteries included.
\n![\"Agent](\"https://www.philschmid.de/static/blog/agent-harness-2026/agent-harness.jpeg\")
We can visualize this by comparing it to a computer:
\nThe Agent harness implements "Context Engineering" strategies like reducing context via compaction, offloading state to storage, or isolating tasks into sub-agents. For developers, this means you can skip building the operating system and focus solely on the application, defining your agent's unique logic.
\nCurrently, general-purpose harnesses are rare. Claude Code is a prime example of this emerging category, attempting to standardize with the Claude Agent SDK or LangChain DeepAgents. However, one could argue that all coding CLIs are, in a way, specialized agent harnesses designed for specific verticals.
\nIn the past, benchmarks were mostly done on single-turn model outputs. Last year, we started to see a trend to evaluate systems instead of raw models, where the model is one component which could use tools or interacts with the environment, e.g. AIMO, SWE-Bench.
\nThese newer benchmarks struggle to measure reliability. They rarely test how a model behaves after its 50th or 100th tool call/turn. This is where the real difficulty lies. A model might be smart enough to solve a hard puzzle in one or two tries, but fail to follow a initial instructions or correctly reasons over intermediate steps after running for an hour. Standard benchmarks struggle to capture the durabilitiy required for long workflows.
\nAs Benchmarks are going to become more complex we need to bridge the gap between benchmark claims and user experience. A Agent Harness can be essential for three critical reasons:
\nThe ability to improve a system is proportional to how easily you can verify its output. [Ref] A Harness turns vague, multi-step agent workflows into structured data that we can log and grade, allowing us to hill-climb effectively.
\nRich Sutton wrote an essay called the Bitter Lesson. He argued that general methods that use computation beat hand-coded human knowledge every time. We see this lesson playing out in agent development right now.
\nTo survive the Bitter Lesson, our infrastructure (Harness) must be lightweight. Every new model release, has a different, optimal way to structure agents. Capabilities that required complex, hand-coded pipelines in 2024 are now handled by a single context-window prompt in 2026.
\nDevelopers must build harnesses that allow them to rip out the "smart" logic they wrote yesterday. If you over-engineer the control flow, the next model update will break your system.
\nWe are heading toward a convergence of training and inference environments. We see a new bottleneck being context durability. The Harness will become the primary tool for solving "model drift". Labs will use the harness to detect exactly when a model stops following instructions or reasoning correctly after the 100th step. This data will be fed directly back into training to create models that don't get "tired" during long tasks.
\nAs builders and developers the focus should shift:
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2026-01-05T00:00:00+00:00", "date_published": "2026-01-05T00:00:00+00:00", "external_url": "https://www.philschmid.de/agent-harness-2026", "id": "urn:aggr:item:e9f994b772cbf8955b9f628c3e3ed81417b654e9", "summary": "In 2026, Agent Harnesses will become essential for building reliable AI systems that can handle complex, multi-day tasks.", "tags": [ "programming" ], "title": "The importance of Agent Harness in 2026", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2026-01-05-the-importance-of-agent-harness-in-2026/" }, { "content_html": "I usually don't make prediction on what is coming next. But 2025 was such a pivotal year for AI and myself that I feel compelled to look ahead. Here is what I see coming for 2026:
\n1. Generative UI takes off. Latency and performance for code-generation enabling applications to generate on-the-fly interfaces for specific user tasks and preference in real-time.
\n2. The Edge becomes the primary home for Personal Agents. Personal agents migrate to edge-first, driven by capable Small Language Models (SLMs) running locally on specialized devices.
\n3. The Smart Home finally fulfills its promise. Home assistants will understand context and intention rather than just direct commands and will naturally integrate into Personal Assistants like Gemini App or ChatGPT.
\n4. Agent Harnesses become the new moat for AI Labs. Existing Benchmarks are reaching saturation to showcase improvements. Labs turn to “Harnesses", complex environments, to show their models (agents) can reliably execute multi-day workstreams.
\n5. Vibe Coding matures into Engineering. The role of engineers continues to evolve. Engineers will spend 99% of their time reviewing, evaluating, conceptualizing and thinking. The core competency is the ability to bridge the gap between implementation details, code quality, velocity, and alignment with business goals. Knowing how to code will be unique selling point.
\n6. Social Media becomes personal again. User engagement on social platforms slows down due to "AI Slop" creating a “premium market” for human created content.
\n7. Content is AI until proven Human The default for any post on X, Instagram, or LinkedIn will be "this is AI.” 2026 introduces the "Human-Signed" metadata standard. Unless a post/image carries a cryptographic signature (C2PA for cameras), algorithms will start to downrank it as "synthetic noise.”
\n8. Biometric Proof of Personhood becomes the new social login. To combat infinite agent spam, social platforms bifurcate into a "Verified Human Web" requiring biometric verification.
\nI am incredibly excited about the year ahead. Despite the rapid changes, I am not worried about my role as an engineer, nor do I believe learning to code was a bad idea. In fact, deep technical understanding is becoming more critical to effectively guide these systems.
\nHowever, the bar has raised. To separate yourself from the crowd in 2026, you need to do more than just use the tools, you need to understand them. My advice is simple: Build and learn. Look closely at what others are building, deconstruct their work to understand the "why" and "how," and then build your own versions. Deep technical intuition is what will distinguish you from others.
\nAgents will continue to be the main headline. We are going to see the successful design patterns established by coding agents move into "general purpose" agents. We will see a shift toward proactive and ambient use cases—where agents don't just wait for a prompt, but interact in our daily lives to anticipate needs.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2025-12-31T00:00:00+00:00", "date_published": "2025-12-31T00:00:00+00:00", "external_url": "https://www.philschmid.de/2026-predictions", "id": "urn:aggr:item:6ccd5080de0ad5057c9ee9ed2a02964d392d9c7b", "summary": "8 Predictions for 2026, exploring the future of AI, personal agents, smart homes, and more.", "tags": [ "programming" ], "title": "8 Predictions for 2026. What comes next in AI?", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-12-31-8-predictions-for-2026-what-comes-next-in-ai/" }, { "content_html": "Earlier this year, Manus published a post on Context Engineering for AI Agents: lessons from building Manus (If you haven't read this first). However, the field moves fast. In a recent webinar, Peak Ji from Manus shared how their Agent Harness has evolved beyond the original post, specifically addressing the challenges of "Context Rot" and multi-agent coordination, and action space management.
\nThis blog dives into learnings from Peak Ji (Manus), Lance Martin (LangChain) and the latest industry research. But before we begin we need a common language.
\nContext Engineering is the discipline of designing a system that provides the right information and tools, in the right format, to give an LLM everything it needs to accomplish a task. Context Engineering includes Context Offloading (move information to external system), Context Reduction (compress history), Context Retrieval (add information dynamically), Context Isolation (separate context).
\nAgent Harness is the software that wraps the model, executing tool calls, managing the message history loop, and handling Context Engineering logic. While the LLM provides the reasoning and generates a structured "tool call", the harness is responsible for the actual execution and management.
\nContext Rot is the phenomenon where an LLM's performance degrade as the context window fills up, even if the total token count is well within the technical limit (e.g., 1 million tokens). The "effective context window", where the model performs at high quality, is often much smaller than the advertised token limit at the moment < 256k tokens for most models.
\nContext Pollution is the presence of too much irrelevant, redundant, or conflicting information within the context that distracts the LLM and degrades its reasoning accuracy.
\nContext Confusion is the failure mode where an LLM cannot distinguish between instructions, data, and structural markers, or encounters logically incompatible directives. This frequently occurs when the System Instructions (global rules) clash in itself, have too many similar instructions or with user instructions.
\nContext reduction is a general concept to prevent Context Rot, we see two distinct methods Compaction and Summarization emerging as standards with prioritizing reversibility over compression.
\nContext Compaction (Reversible): Strip out information that is redundant because it exists in the environment. Context Compactions are reversible, this means that If the agent needs to read the code later, it can use a tool to read the file.
\nOutput saved to /src/main.py).Summarization (Lossy): Use an LLM to summarize the history including tool calls and messages, ofter triggered at context rot threshold, e.g. 128k tokens. When summarizing, Manus keeps the most recent tool calls in their raw, full-detail format. This ensures the model maintains its "rhythm", formatting style and prevents degradation of output quality.
\n![\"Context](\"https://www.philschmid.de/static/blog/context-engineering-part-2/compact.jpeg\")
Prefer raw > Compaction > Summarization only when compaction no longer yields enough space.
\nMulti-agent systems fail due to Context pollution. If every sub-agent shares the same context, you pay a massive KV-cache penalty and confuse the model with irrelevant details. Manus applies a principle from GoLang concurrency to agent architecture: "Share memory by communicating, don't communicate by sharing memory."
\nDiscrete Tasks: For tasks with clear inputs/outputs (e.g., "Search this documentation for X"), spin up a fresh sub-agent with its own context and pass only the specific instruction.
\nComplex Reasoning: Only share the full memory/context history when the sub-agent must understand the entire trajectory of the problem to function (e.g., a debugging agent that needs to see previous error attempts). Treat shared context as an expensive dependency to be minimized, forking context breaks the Cache.
\n![\"Communicate\"](\"https://www.philschmid.de/static/blog/context-engineering-part-2/communicate.jpeg\")
Providing an LLM with 100+ tools leads to Context Confusion where the model hallucinates parameters or calls the wrong tool. Manus solves this with a Hierarchical Action Space:
\nLevel 1 (Atomic): The model sees ~20 core tools (e.g., file_write, browser_navigate, bash, search ...). These are stable and cache-friendly.
Level 2 (Sandbox Utilities): Instead of a specific tool for grep, instruct the model to use the bash tool to call ffmpeg via CLI. Manus has mcp-cli <command>, keeping the tool definitions out of the context window.
Level 3 (Code/Packages): For complex logic chains (e.g., "Fetch city -> Get ID -> Get Weather"), don't make 3 LLM roundtrips. Provide libraries/functions that handle the logic, and let the agent write a dynamic script.
\n![\"hierachy\"](\"https://www.philschmid.de/static/blog/context-engineering-part-2/hierachy.jpeg\")
Don't over-anthropomorphize your agents. You don't need an "Org Chart" of agents (Manager, Designer, Coder) that chat with each other. Instead, treat Agents as Tools.
\nFor the main model, "Deep Research" or "Plan Task" should just be a tool call. The main agent invokes call_planner(goal="..."), the harness spins up a temporary sub-agent loop and returns a structured result. This flattens complexity and keeps the harness modular.
This "MapReduce" Pattern allows the main agent to treat the sub-agent exactly like a deterministic code function. It can define the goal, tools and output schema (e.g., a specific JSON structure). This ensures the data returned is instantly usable without further parsing or conversation.
\n![\"agent-as-tool\"](\"https://www.philschmid.de/static/blog/context-engineering-part-2/subagent.jpeg\")
Don't use RAG to manage tool definitions: Fetching tool definitions dynamically per step based on semantic similarity often fails. It creates a shifting context that breaks the KV cache and confuses the model with "hallucinated" tools that were present in turn 1 but disappeared in turn 2.
\nDon't train your own models (yet): We are living the Bitter Lesson. The harness you build today will likely be obsolete when the next frontier model drops. If you spend weeks fine-tuning a model or training an RL policy on a specific action space, you are locking yourself into a local optimum. Use Context Engineering as the flexible interface that adapts to rapidly improving models.
\nDefine a "Pre-Rot Threshold": If a model has a 1M context window, performance often degrades around < 256k. Don't wait for the API to throw an error. Monitor your token count and implement Compaction or Summarization cycle before you hit the "rot" zone to maintain reasoning quality.
\nUse "Agent-as-a-Tool" for Planning: In early versions, Manus used a todo.md file that was constantly rewritten. This wasted tokens (~30% in earlier versions of Manus). A better pattern is a specific Planner sub-agent that returns a structured Plan object. This object is injected into the context only when needed, rather than consuming tokens in every turn.
Security & Manual Confirmation: When giving agents browser or shell access, sandbox isolation isn't enough. Try to enforce rules that tokens not leave the sandbox and use human in the loop interrupts for Manual Confirmation before proceeding.
\nThe "Intern Test": Static benchmarks like GAIA saturated quickly and didn't align with user satisfaction. Focus on tasks that are computationally verifiable. Did the code compile? Did the file exist after the command ran? Can the sub-agent verify the output of the parent? Try to use binary success/fail metrics on real environments rather than subjective LLM-as-a-Judge scores.
\nEmbrace "Stochastic Gradient Descent": Manus was rewritten five times in the last six months. LangChain re-architected Open Deep Research four times. This is normal. As models get smarter, your harness changes. If your harness is getting more complex while models improve, you are likely over-engineering.
\nPerhaps the biggest takeaway was Peak's explanation of complexity. They have rewritten Manus five times. In the last six months, their biggest performance gains didn't come from adding complex RAG pipelines or fancy routing logic. The gains came from removing things.
\nThey removed complex tool definitions in favor of general shell execution.They removed "management agents" in favor of simple structured handoffs. They rely on Agent-as-a-Tool.
\nAs models get stronger, we shouldn't be building more scaffolding, we should be getting out of the model's way. Context Engineering is not about adding more context. It is about finding the minimal effective context required for the next step.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2025-12-04T00:00:00+00:00", "date_published": "2025-12-04T00:00:00+00:00", "external_url": "https://www.philschmid.de/context-engineering-part-2", "id": "urn:aggr:item:16af785e8898586548aca4e40bfded6823da355c", "summary": "Building on the foundations of Context Engineering, this post explores advanced strategies to manage context rot, multi-agent coordination, and action space optimization for AI agents.", "tags": [ "programming" ], "title": "Context Engineering for AI Agents: Part 2", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-12-04-context-engineering-for-ai-agents-part-2/" }, { "content_html": "For the past few decades, Engineering meant one thing: removing ambiguity. It meant defining strict interfaces, enforcing type safety, and ensuring that Input A \\+ Code B \\= Output C.
Traditional software engineering is Deterministic. We play the role of Traffic Controllers; we own the roads, the lights, and the laws. We decide exactly where data goes and when. Agent Engineering is Probabilistic. We are Dispatchers. We give instructions to a driver (an LLM) who might take a shortcut, get lost, or decide to drive on the sidewalk because it "seemed faster."
\nIt is a paradox that junior engineers often ship functional agents faster than seniors. Why? The more senior the engineer, the less they tend to trust the reasoning and instruction-following capabilities of the Agent. We fight the model and try to "code away" the probabilistic nature.
\nHere are 5 examples where traditional engineering habits clash with the new reality of Agent Engineering.
\nIn traditional engineering, we model the world with data structures. We define schemas, interfaces, and strict types. It feels safe because it is predictable. We instinctively try to force Agents into this box.
\nThe Trap: Real-world intents, preferences, or configurations are rarely binary/structured. User inputs are continuous (natural language) rather than discrete (structured fields).
\nText is the new State. We must abandon the comfort of booleans in favor of semantic meaning.
\nImagine a Deep Research plan approval use case where the user says, "This plan looks good, but please focus on the US market." A deterministic system forces this into is_approved: true/false and we lobotomize the context.
Software Engineering:
\nJSON
\n\nAgent Engineering:
\nJSON
\n\nBy preserving the text, the downstream agent can read the feedback ("Approved, but focus on US market") and adjust its behavior dynamically.
\nAnother example is user preferences. A deterministic system might store is_celsius: true. An agentic system stores "I prefer Celsius for weather, but use Fahrenheit for cooking". The agent can dynamically switch contexts based on the task.
In microservices, a user's intent matches a route (POST /subscription/cancel). In Agents, we have a single natural language entrypoint with a Brain (LLM) that decides the control flow based on the available tools, input, and instructions.
The Trap: We try to hard-code the flow into the agent, but interactions don't follow straight lines. They loop, they backtrack, and they pivot. A user might want to cancel, and end up agreeing to renew.
\nTrust the agent to navigate the flow. If we try to hard-code every edge case, we aren't building an AI agent. We must trust the agent to understand the current intent based on the entire context.
\nIn traditional software, if an API call fails or a variable is missing, we throw an exception. We want the program to crash immediately so we can fix the bug.
\nThe Trap: An agent might take 5 minutes and cost $0.50. If step 4 of 5 fails because of a missing or wrong input, crashing the whole execution is unacceptable.
\nAn error is just another input. Instead of crashing, we catch the error, feed it back to the agent, and try to recover.
\n![\"Agent](\"https://www.philschmid.de/static/blog/why-engineers-struggle-building-agents/agent-error-handling.png\")
Test Driven Development (TDD) helps us write more robust code, but we cannot unit test an Agent. Engineers can waste weeks looking for binary correctness in a probabilistic system. We must evaluate behavior.
\nThe Trap: We cannot write binary assertions for creative or reasoning tasks. "Write a summary of this email" has infinite valid outputs. If we try to Mock the LLM, we aren't testing the agent, we are testing our string concatenation.
\nEvals over Tests. We cannot unit test reasoning. We must validate Reliability and Quality, and trace intermediate checks:
\nIf our agent succeeds 45/50 times with a quality score of 4.5/5, it can be production-ready. We are managing risk, not eliminating variance.
\nIn the past, we designed APIs for human developers, relying on implicit context and "clean" interfaces. Humans infer context. Agents do not. Agents are Literalists. If an ID format is ambiguous, the agent will hallucinate one.
\nThe Trap: We often build "Human-Grade" APIs—endpoints that rely on implicit context. For example, a variable named id is obviously the user_unique_identifier (UUID) to us, and can be used in get_user(id). An Agent might not have this context and might try to use the email or name in get_user(id).
Agents require verbose, "Idiot-Proof" semantic typing (e.g., "user_email_address" instead of "email") and highly descriptive docstrings that act as "context".
delete_item(id) (Is ID an integer? A UUID? What happens if it's not found?)delete_item_by_uuid(uuid: str) with a docstring/description: "Deletes an item. If the item is not found, return a descriptive error string."Furthermore, Agents allow for Just-in-Time adaptation. Normal APIs are promises to developers; we commit code that relies on those APIs, and then walk away. If we change an API from get_user_by_id(id) to get_user_by_email(email), we break that promise and everything breaks immediately. An agent, however, reads the new tool definition and can adjust to it.
The transition from deterministic systems to probabilistic agents is uncomfortable. It requires us to trade certainty for semantic flexibility. We no longer know and own the exact execution path. We effectively hand over the control flow to a non-deterministic model and store our application state in natural language.
\nThis feels wrong to a mind trained on strict interfaces. But trying to force an Agent into a deterministic box defeats the purpose of using one. You cannot code away the probability. You must manage it through evals and self-correction.
\nHowever, "trusting" an agent does not mean letting it run wild. We must find the middle ground. Agents will fail in many unexpected ways but the trajectory is clear. We must stop trying to code away the ambiguity and start engineering systems that are resilient enough to handle it.
\nThis also means knowing when to use workflows over agents. I go into detail on how to architect these differences in my post on Agentic Patterns.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2025-11-26T00:00:00+00:00", "date_published": "2025-11-26T00:00:00+00:00", "external_url": "https://www.philschmid.de/why-engineers-struggle-building-agents", "id": "urn:aggr:item:f72fab18afcf8b340f9d39ca5e5be22a65d354ab", "summary": "Traditional software engineering is deterministic, while AI agents operate probabilistically. This fundamental difference creates challenges for engineers accustomed to strict interfaces and predictable outcomes.", "tags": [ "programming" ], "title": "Why (Senior) Engineers Struggle to Build AI Agents", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-11-26-why-senior-engineers-struggle-to-build-ai-agents/" }, { "content_html": "\n\nAlternative Version: Check out the Interactions API version of this guide.
\n
It seems complicated, when you watch an AI agent edit multiple files, run commands, handle errors, and iteratively solve a problem, it feels like magic. But it isn’t. The secret to building an agent is that there is no secret.
\nThe core of an Agent is surprisingly simple: It is a Large Language Model (LLM) running in a loop, equipped with tools it can choose to use.
\nIf you can write a loop in Python, you can build an agent. This guide will walk you through the process, from a simple API call to a functioning CLI Agent.
\nTraditional software workflows are prescriptive and follow predefined paths (Step A -> Step B -> Step C), Agents are System that uses an LLM to dynamically decide the control flow of an application to achieve a user goal.
An agent generally consists of these core components:
\nwhile loop that allows the model to: Observe → Think → Act → Observe again, until the task is complete.
"The Loop" of nearly every agent is an iterative process:
\nget_weather) to the model using a structured JSON format.tool use containing the tool name and arguments.tool use, executes the actual code or API call, and captures the result.tool response) back to model. The model uses this new information to decide the next step, either calling another tool or generating the final response.Let's build an agent step-by-step, progressing from basic text generation to a functional CLI agent using Gemini 3 Pro and Python SDK.
\nPrerequisites: Install the SDK (pip install google-genai) and set your GEMINI_API_KEY environment variable (Get it in AI Studio).
The first step is to create a baseline interaction with the LLM, for us Gemini 3 Pro. We are going to create a simple Agent class abstraction to structure our code, which we will extend throughout this guide. We will first start with a simple chatbot that maintains a conversation history.
\nPython
\n\n\n \n\n\n = \n = \n = \n \n\n \n \n = \n \n \n return \n \n = \n = \n \n\n# Output: Berlin, Munich, Cologne \n = \n \n\n# Output: Munich is the capital of Bavaria and is known for its Oktoberfest.\nThis is not an agent yet. It is a standard chatbot. It maintains state but cannot take action, has no "hands or eyes".
\nTo start turning this an agent, we need Tool Use or Function Calling. We provide the agent with tools. This requires defining the implementation (the Python code) and the definition (the schema the LLM sees). If the LLM believes that tool will help solve a user's prompt, it will return a structured request to call that function instead of just text.
\nWe are going to create 3 tools, read_file, write_file, and list_dir. A tool Definition is a JSON schema that defines the name, description, and parameters of the tool.
Best Practice: Use the description fields to explain when and how to use the tool. The model relies heavily on these to understand when and how to use the tool. Be explicit and clear.
Python
\n\n\n \n = \n \n = \n \n = \n \n\n \n return \n \n\n """Writes a file with the given contents."""\n \n \n return True\n \n\n """Lists the contents of a directory."""\n = \n return \n \n = \nNow we integrate the tools and function calls into our Agent class.
Python
\n\n\n \n\n\n = \n = \n = \n = \n \n\n \n \n = \n \n = \n \n \n return \n \n = \n \n = \n\n# Output: [FunctionCall(name='list_dir', arguments={'directory_path': '.'})]\nGreat! The model has successfully called the tool. Now, we need to add the tool execution logic to our Agent class and the loop return the result back to the model.
\nAn Agent isn't about generating one tool call, but about generating a series of tool calls, returning the results back to the model, and then generating another tool call, and so on until the task is completed.
\nThe Agent class handles the core loop: intercepting the FunctionCall, executing the tool on the client side, and sending back the FunctionResponse. We also add a SystemInstruction to the model to guide the model on what to do.
Note: Gemini 3 uses Thought signatures to maintain reasoning context across API calls. You must return these signatures back to the model in your request exactly as they were received.
\nPython
\n# ... Code for the tools and tool definitions from Step 2 should be here ...\n \n\n\n \n\n\n = \n = \n = \n = \n = \n \n\n \n \n \n \n \n = \n \n = \n \n \n \n = \n \n \n \n \n = \n \n = \n \n \n \n \n return \n \n return \n \n = \n \n = \n\n# Output: [Function Call] id=None args={'directory_path': '.'} name='list_dir'\n# [Function Response] {'result': ['.venv', ... ]}\n# There. Your current directory contains: `LICENSE`,\nCongratulations. You just built your first functioning agent.
\nNow we can run our agent in a simple CLI loop. It takes surprisingly little code to create highly capable behavior.
\nPython
\n# ... Code for the Agent, tools and tool definitions from Step 3 should be here ...\n \n = \n \n\n\n = \n \n break\n \n = \n \nBuilding the loop is easy; making it reliable, transparent, and controllable is hard. Here are key engineering principles derived from top industry practices, grouped by functional area.
\nYour tools are the interface for the model. Don't just wrap your existing internal APIs. If a tool is confusing to a human, it's confusing to the model:
\nsearch_customer_database rather than cust_db_v2_query.Error: File not found. Did you mean 'data.csv'?. This allows the agent to self-correct.Models have a finite "attention budget." Managing what information enters the context is crucial for performance and cost.
\nget_all_users(), create search_users(query: str).It's tempting to build complex multi-agent systems. Don't.
\nmax_iterations break (e.g., 15 turns).system_instruction to guide the model with hard rules (e.g., "You are strictly forbidden from offering refunds greater than $50") or use external classifier.send_email or execute_code), pause the loop and require user confirmation before the tool is actually executed.Building an agent is no longer magic; it is a practical engineering task. As we've shown, you can build a working prototype in under 100 lines of code. While understanding these fundamentals is key, don't get bogged down re-engineering the same pattern over and over. The AI community has created fantastic open-source libraries that can help you build more complex and robust agents faster.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2025-11-21T00:00:00+00:00", "date_published": "2025-11-21T00:00:00+00:00", "external_url": "https://www.philschmid.de/building-agents", "id": "urn:aggr:item:b6d5d43b39ac8042717a72fd850ce021b3389691", "summary": "A step-by-step practical guide on building AI agents using Gemini 3 Pro, covering tool integration, context management, and best practices for creating effective and reliable agents.", "tags": [ "programming" ], "title": "Practical Guide on how to build an Agent from scratch with Gemini 3", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-11-21-practical-guide-on-how-to-build-an-agent-from-scratch-with/" }, { "content_html": "I have been using Gemini 3 Pro for a bit and let me put it simply. It is so much better at literal everything than 2.5 Pro! This post shares the principles and structural patterns that are currently working best for me. This isn't meant to be treated as the gold standard, but rather as a starting point to help you refine your own strategies. Take what works, tweak what doesn't, and keep iterating.
\nGemini 3 favors directness over persuasion and logic over verbosity. To maximize performance, adhere to these core principles:
\nExplicit Planning & Decomposition
\nBefore providing the final answer, please:\n1. Parse the stated goal into distinct sub-tasks.\n2. Is the input information complete? If not, stop and ask for it.\n3. Are there tools, shortcuts, or "power user" methods that solve this problem better than the standard approach? (e.g., "Don't just list specs, suggest a workaround").\n4. Create a structured outline to achieve the goal.\n5. Validate your understanding before proceeding.\nSelf-updating TODO Tracker
\nCreate a TODO list to track progress:\n \n- [ ] Primary objective\n- [ ] Task 1\n- [ ] Task 2\n....\n- [ ] Review\nCritique its own output
\nBefore returning your final response, review your generated output against the user's original constraints. \n \n1. Did I answer the user's *intent*, not just their literal words?\n2. Is the tone authentic to the requested persona?\n3. If I made an assumption due to missing data, did I flag it?\nUse XML-style tagging or Markdown to structure prompts. This provides unambiguous boundaries that help the model distinguish between instructions and data. Don't mix XML or Markdown, choose one format for consistency.
\nXML Example:
\nXml
\n\n 1. Be objective.\n 2. Cite sources.\n\n \n\n 1. Analyze the Request: Identify the core goal and all explicit constraints.\n 2. Decompose: Break the problem into logical sub-tasks or variables.\n 3. Strategize: Outline the step-by-step methodology to solve each sub-task.\n 4. Verify: Check your plan for logical gaps or edge cases.\n\n \n\n IF is empty, missing code, or lacks necessary data:\n DO NOT attempt to generate a solution.\n DO NOT make up data.\n Output a polite request for the missing information.\n\n \n\n [Insert User Input Here - The model knows this is data, not instructions]\n\nMarkdown Example:
\n# Identity\nYou are a senior solution architect.\n \n# Constraints\n- No external libraries allowed.\n- Python 3.11+ syntax only.\n \n# Output Format\nReturn a single code block.\nThe Persistence Directive
\nYou are an autonomous agent.\n- Continue working until the user's query is COMPLETELY resolved.\n- If a tool fails, analyze the error and try a different approach.\n- Do NOT yield control back to the user until you have verified the solution.\nPre-Computation Reflection
\nBefore calling any tool, explicitly state:\n1. Why you are calling this tool.\n2. What specific data you expect to retrieve.\n3. How this data helps solve the user's problem.\nResearch and Analysis
\n1. Decompose the topic into key research questions\n2. Search for/Analyze provided sources for each question independently\n3. Synthesize findings into a cohesive report\n4. CITATION RULE: If you make a specific claim, you must cite a source. If no source is available, state that it is a general estimate. Every claim must be immediately followed by a reference [Source ID]\nCreative Writing
\n1. Identify the target audience and the specific goal (e.g., empathy vs. authority).\n2. If the task requires empathy or casualness, strictly avoid corporate jargon (e.g., "synergy," "protocols," "ensure").\n3. Draft the content.\n4. Read the draft internally. Does this sound like a human or a template? If it sounds robotic, rewrite it.\nProblem-Solving
\n1. Restate the problem in your own words.\n2. Identify the "Standard Solution."\n3. Identify the "Power User Solution" (Is there a trick, a specific tool, or a nuance most people miss?).\n4. Present the solution, prioritizing the most effective method, even if it deviates slightly from the user's requested format.\n5. Sanity check: Does this solve the root problem?\nEducation Content
\n1. Assess the user's current knowledge level based on their query.\n2. Define key terms before using them.\n3. Explain the concept using a relevant analogy.\n4. Provide a "Check for Understanding" question at the end.\nThis template combines best practices (Caching-friendly structure, Planning, and XML delimiters) into a reusable baseline.
\n⚠️ Note: The Engineering Mindset
\nThere is no "perfect" template or context structure. Context engineering is an empirical effort, not a fixed syntax. The optimal structure depends heavily on your specific data, latency constraints, and domain complexity. Treat the patterns below as robust baselines, but expect to iterate, measure, and refine based on your specific use case.
System Instruction
\n<role>\nYou are Gemini 3, a specialized assistant for [Insert Domain, e.g., Data Science].\nYou are precise, analytical, and persistent.\n</role>\n \n<instructions>\n1. **Plan**: Analyze the task and create a step-by-step plan into distinct sub tasks. tags. \n2. **Execute**: Carry out the plan. If using tools, reflect before every call. Track you progress in TODO List use [ ] for pending, [x] for complete. \n3. **Validate**: Review your output against the user's task. \n4. **Format**: Present the final answer in the requested structure.\n</instructions>\n \n<constraints>\n- Verbosity: [Low/Medium/High]\n- Tone: [Formal/Casual/Technical]\n- Handling Ambiguity: Ask clarifying questions ONLY if critical info is missing; otherwise, make reasonable assumptions and state them.\n</constraints>\n \n<output_format>\nStructure your response as follows:\n2. **Executive Summary**: [2 sentence overview]\n3. **Detailed Response**: [The main content]\n</output_format>\nUser Prompt
\n \n<context>\n[Insert relevant documents, code snippets, or background info here]\n</context>\n \n<task>\n[Insert specific user request here]\n</task>\n \n<final_instruction>\nRemember to think step-by-step before answering.\n</final_instruction>\n \nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2025-11-19T00:00:00+00:00", "date_published": "2025-11-19T00:00:00+00:00", "external_url": "https://www.philschmid.de/gemini-3-prompt-practices", "id": "urn:aggr:item:ac8f1a7786dee7a416dcd748198704d603013d83", "summary": "A comprehensive guide on best practices for prompting Gemini 3, focusing on clarity, structure, reasoning, and agentic tool use to maximize model performance across various domains.", "tags": [ "programming" ], "title": "Gemini 3 Prompting: Best Practices for General Usage", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-11-19-gemini-3-prompting-best-practices-for-general-usage/" }, { "content_html": "The Gemini API File Search tool is a fully managed RAG (Retrieval-Augmented Generation) system built directly into the Gemini API. It automatically manages file storage, chunks your data, creates embeddings, and seamlessly injects the most relevant context into your prompts.
\nThis must be expensive? No, file storage and embedding generation at query time are completely free of charge. You only pay for the initial indexing of your files at a fixed rate of $0.15 (based on the embedding) per 1 million tokens.
\nThis tutorial will walk you through the complete lifecycle of using File Search with the JavaScript/TypeScript, you will learn how to:
\nBefore we start, make sure you have an API key from Google AI Studio and have installed the latest SDK:
\nBash
\n\nInitialize your client in your JavaScript environment:
\nJavaScript
\n;\n;\n;\n \nconst ai = ;\nA File Search Store is a persistent container for your document chunks and embeddings. It's distinct from raw file storage and can hold gigabytes of data.
\nJavaScript
\nconst fileStoreName = 'my-example-store';\n \nconst createStoreOp = await ai..;\n \n;\nOften the creation of a store and the use are in a different application session. Since the API assigns a unique ID (fileSearchStores/xyz...), you need to look it up by the human-readable displayName.
JavaScript
\nlet fileStore = null;\n// List stores with a page size limit\nconst pager = await ai..;\nlet page = pager.;\n \n// Iterate through pages until we find a match\nsearchLoop: \n \n\n;\nSpeed matters. When ingesting a folder of documents, don't process them sequentially. The API supports concurrent operations, so we can use Promise.all to upload and process multiple files at once.
We'll use the helper method uploadToFileSearchStore, which handles uploading the raw file and initiating the indexing process in one step. We then monitor operation.done to ensure processing completes before moving on.
JavaScript
\nconst docsDir = "docs"; // Ensure you have a 'docs' folder with some text files\nconst files = fs..;\n \nawait Promise.;\nBy default, Gemini handles chunking intelligently. However, for specific use cases you might want tighter control over how your data is split.
\nYou can define a chunkingConfig during upload to specify parameters like maxTokensPerChunk and maxOverlapTokens and customMetadata to attach key-value pairs to the document.
JavaScript
\nconst specialDocPath = 'special-docs/technical-manual.txt';\n \nlet advancedUploadOp = await ai..;\n \n// Wait for the file to process\n\n;\nWe don't need to manually retrieve chunks. We just tell the Gemini model to use the fileSearch tool and point it to our store name. Gemini understands it needs more information, searches the store, and grounds its response automatically.
JavaScript
\nconst response = await ai..;\n \n;\n// Optionally check response.candidates[0].groundingMetadata for citations!\nBecause we tagged our technical manual in Step 4, we can now force Gemini to only look at documents matching that tag by using a metadataFilter.
\nJavaScript
\nconst responseFiltered = await ai..;\n \n;\nYou'll often need to manage individual documents within your store. You can find a specific document by its display name.
\nJavaScript
\nconst docToFind = 'doc1.txt';\nlet targetDoc = null;\n \nlet documentPager = await ai...;\n \n// Iterate through the store's document list\nsearchDocsLoop: \n \nthrow ;\nCurrently, the standard flow to update a document in File Search is to delete the old version and upload a new one.
\nJavaScript
\nawait ai...;\nFile Search documents are immutable once indexed. To "update" a document, you must find it, delete it, and upload the new version. In this step, we will automate this entire loop to update doc1.txt with new information.
JavaScript
\nconst docToUpdate = 'doc1.txt'; // assuming it has new content\nconst localDocPath = path.;\n \n// 1. Find the existing document ID in the store based on its display name\nlet documentPager = await ai...;\nlet foundDoc = null;\nfindLoop: \n \n// 2. If we found it, delete it\n\n \n// 3. Upload the new version\nlet updateOp = await ai..;\n\n \n;\nYou are currently limited to 10 File Search Stores per project, so it's important to clean up resources when you are finished with development.
\nJavaScript
\nawait ai..;\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2025-11-07T00:00:00+00:00", "date_published": "2025-11-07T00:00:00+00:00", "external_url": "https://www.philschmid.de/gemini-file-search-javascript", "id": "urn:aggr:item:4fa1fdce3813f181bdf171cad07d35bb3bed2ce8", "summary": "Learn how to use the Gemini API File Search tool with JavaScript/TypeScript to build a Retrieval-Augmented Generation (RAG) system.", "tags": [ "programming" ], "title": "Gemini API File Search: A Web Developer Tutorial", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-11-07-gemini-api-file-search-a-web-developer-tutorial/" }, { "content_html": "n8n is a powerful, open-source workflow automation tool that allows you to connect and automate different services and applications. This guide will walk you through deploying n8n on Google Cloud Run with a PostgreSQL database for persistent storage and creating a simple AI Agent using Google Gemini 2.5.
\nBefore we start you need a Google Cloud Platform (GCP) account with a billing account configured.
\nTo interact with Google Cloud from your terminal, you need the Google Cloud CLI.
\nInstall the gcloud CLI: You can install the Google Cloud CLI using the following command, or follow the official installation instructions for your specific operating system.
\nBash
\n | \n\n\nRun gcloud version to verify the installation.
Login to Google Cloud: Once installed, authenticate with your Google Cloud account:
\nBash
\n\nA browser window will open asking you to log in and allow the Google Cloud SDK to access your account. Copy the authentication code and paste it into the terminal.
\nWe need to set up a new Google Cloud project and enable the necessary services.
\nCreate a new project:
\nBash
\n\n\n\nLink Billing Account: You need to enable billing for your new project. Run the command below and open the URL in your browser to link your billing account.
\nBash
\n\nSet environment variables:
\nBash
\n # Change to your preferred region, e.g. us-east1, us-west1, etc.\nEnable necessary APIs:
\nBash
\n\nn8n requires a database to store its data. We'll use Cloud SQL for PostgreSQL and Secret Manager to securely store credentials. We are using a db-f1-micro tier for cost-effectiveness, but you can adjust the parameters like tier, storage size, and region based on your needs. For more configuration options, refer to the official documentation.
Note: This can take ~10-15 minutes.
\nBash
\n\n\n \n# Create Cloud SQL instance\n\n \n# Create database and user\n\n\n \n# Store secrets in Secret Manager\n | \n | \nNow we'll deploy n8n to Cloud Run, connecting it to our database and secrets. We need to create a service account for n8n and grant it access to the secrets, then deploy the n8n container to Cloud Run with the necessary environment variables and secrets.
\nCreate a service account for n8n:
\nBash
\n\n\nGrant necessary permissions:
\nBash
\n\n\n\nDeploy to Cloud Run:
\nBash
\nDB_CONNECTION_NAME="::n8n-db"\n\n\nNote: We use --no-cpu-throttling to ensure n8n's background processes (like schedules and wait nodes) continue running even when not actively handling an HTTP request.
After a successful deployment, Cloud Run will output the URL of your n8n instance. This should look like:
\nDone.\nService [n8n] revision [n8n-00001-qqn] has been deployed and is serving 100 percent of traffic.\nService URL: https://n8n-780597096402.us-central1.run.app\nAfter you access your n8n instance for the first time, you will be prompted to set up an owner account. Follow the instructions to create an owner account. Here you can also add your enterprise license if you have one. Awesome, once done you should see a screen like this:
\n![\"n8n\"](\"https://www.philschmid.de/static/blog/n8n-cloud-run-gemini/n8n.png\")
We are going to create an Agent from scratch. But before we create the Agent we need to add our Gemini API Key. Click on "Credentials" and then "Add first credential" and search for "Google Gemini (PaLM) API". Click on "Create" and follow the instructions to create a new credential. If you don't have a Gemini API Key, you can get one from the AI Studio.
\n![\"api-key\"](\"https://www.philschmid.de/static/blog/n8n-cloud-run-gemini/api-key.png\")
Once validated go back to "Workflows" and click on "Start from scratch". You should now land in the "editor". Click "Add first step" and search for "AI Agent". Now, you should be in the in configuration of the AI Agent. At the bottom in the middle of the screen you should see "Chat Model", "Memory" and "Tool". Click on "Chat Model"
\n![\"agent-editor\"](\"https://www.philschmid.de/static/blog/n8n-cloud-run-gemini/agent-editor.png\")
Then search for "Google Gemini Chat Model". There you should be able to select your credentials and the model you want to use. Once defined click "Back to canvas" (top left corner). You should now see a "Basic AI Agent" with Gemini.
\n![\"agent\"](\"https://www.philschmid.de/static/blog/n8n-cloud-run-gemini/agent.png\")
Lets try it! Click on "Open Chat" and ask it a question.
\n![\"agent-chat\"](\"https://www.philschmid.de/static/blog/n8n-cloud-run-gemini/agent-chat.png\")
Awesome, it works! Now, lets add some tools and make it an Agent. Therefore click on "+" at "tool" under the AI Agent. Now, you should see a list of tools you can add to your Agent, including MCP servers, vector databases, or hundreds of other tools.
\nLet's try to add the "MCP (Model Context Protocol) Client". MCP is a standard that allows LLMs to connect to external data sources without custom integration code. We will use it to connect Gemini to a dummy weather service. As endpoint add https://gemini-api-demos.uc.r.appspot.com/mcp (Note: There is no guarantee that this endpoint is always available.) Once you have added your MCP server click on "back to canvas".
Your Agent should now have a tool connection to a "MCP Client".
\n![\"agent-mcp\"](\"https://www.philschmid.de/static/blog/n8n-cloud-run-gemini/agent-mcp.png\")
Okay, lets test it. Click on "Open Chat" and ask it about the weather in New York, you have to include the "date" in your question, e.g. What is the weather in New York on 2025-10-29?
![\"agent-mcp-chat\"](\"https://www.philschmid.de/static/blog/n8n-cloud-run-gemini/agent-mcp-chat.png\")
Awesome, it works! During the execution should have seen the Agent first calling Gemini, which outputs a structured call, to call the MCP Server, the MCP Server response was then sent back to Gemini, which generates the final response.
\nDon't forget to delete your Google Cloud resources when you are done. You can delete the resources using the following command:
\nBash
\n\n\n\n\n\n\nCongratz! You have successfully deployed n8n to Google Cloud Run and created your first AI Agent using Gemini 2.5. Now it's time to explore more advanced features, add more tools and build complex workflows to automate your processes. As you move from a proof-of-concept to production, consider these resources:
\nThis post provides a high-level overview to over 50 of modern benchmarks, grouped into four key categories Function Calling & Tool Use (1), General Assistant & Reasoning (2), Coding & Software Engineering (3), and Computer Interactions (4).
The compendium is also available as separate Github repository. Would love to keep this up to date and extend when need benchmarks are coming up. Please Open PRs or Issues.
\nBFCL is a comprehensive benchmark designed to evaluate the function calling (also known as tool use) capabilities of Large Language Models (LLMs) in a wide range of real-world settings. It assesses models across various scenarios, including serial (simple), parallel, and multi-turn interactions, and evaluates agentic capabilities such as reasoning in stateful multi-step environments, memory, web search, and format sensitivity.
\nLinks: Paper | GitHub | Leaderboard | Dataset
\nA massive-scale benchmark designed for evaluating and facilitating large language models in mastering over 16,000 real-world RESTful APIs. It functions as an instruction-tuning dataset for tool use, which was automatically generated using ChatGPT to enhance the general tool-use capabilities of large language models.
\nLinks: Paper | GitHub | Leaderboard | Dataset
\nA benchmark specifically designed for the evaluation of complex function calling in LLMs. It addresses challenging scenarios across five key aspects: multi-step function calls within a single turn, function calls involving user-provided constraints, parameter value reasoning, calls with long parameter values, and calls requiring a 128k long-context length.
\nLinks: Paper | GitHub | Dataset
\nA conversational benchmark designed to test AI agents in dynamic, open-ended real-world scenarios. It specifically evaluates an agent's ability to interact with simulated human users and programmatic APIs while strictly adhering to domain-specific policies and maintaining consistent behavior, with domains in e-commerce and airline reservations.
\nLinks: Paper | GitHub | Leaderboard
\nTests the ability of LLMs to correctly call functions based on given prompts. It comprises 50 function calling problems, each designed to be solved using one of eight provided function schemas inspired by real-world API structures from ClickUp's integration endpoints.
\nLinks: GitHub
\nEvaluates an agent's ability to plan step-by-step API calls, retrieve relevant APIs, and correctly execute API calls to meet human needs based on understanding real-world API documentation. It features over 2,200 dialogues utilizing thousands of APIs.
\n\nA novel benchmark designed to evaluate the function-calling capabilities of LLMs in realistic, multi-turn human-agent interactions, particularly simulating mobile assistant use cases. It tests models under challenging circumstances like imperfect instructions and shifts in user intent.
\nLinks: Paper | GitHub | Dataset
\nThe Dria Pythonic Agent Benchmark is a comprehensive benchmark designed to evaluate the function calling capabilities of LLMs. It specifically compares the performance of models using Pythonic function calling versus traditional JSON-based methods across 100 problems.
\nLinks: Blog
\nA benchmark designed to evaluate the proficiency of LLMs in single-turn function calling tasks. It assesses various complexities, including simple, parallel, and nested function calls, where the output of one function serves as an input for another.
\nLinks: GitHub | Leaderboard
\nA series of large action models (LLMs) developed by Salesforce AI Research, specifically optimized for function calling and AI agent tasks. These models are designed to enhance the generalizability and performance of AI agents across diverse environments.
\n\nAn automatic agentic pipeline meticulously designed to generate accurate, complex, and diverse tool-learning data, specifically tailored to enhance the function-calling capabilities of LLMs.
\nLinks: Paper
\nA comprehensive benchmark designed to evaluate the ability of LLM agents to navigate and effectively utilize a large-scale Model Context Protocol (MCP) toolset in real-world scenarios, overcoming limitations of single-server environments.
\nLinks: Paper | GitHub | Leaderboard | Dataset
\nA comprehensive framework and benchmark for developing, testing, and evaluating AI agents and LLMs through direct interaction with real-world Model Context Protocol (MCP) servers, rather than relying on simulations, covering domains like financial analysis and browser automation.
\nLinks: Paper | GitHub | Leaderboard
\nA landmark benchmark designed to evaluate General AI Assistants, posing real-world questions that are conceptually simple for humans but significantly challenging for most advanced AI systems. It requires AI models to demonstrate a combination of fundamental abilities, including reasoning, multi-modality handling, web browsing, and proficient tool use.
\nLinks: Paper | Leaderboard | Dataset
\nA multi-dimensional, evolving benchmark designed to thoroughly assess the reasoning and decision-making capabilities of LLMs when functioning as autonomous agents. It encompasses eight distinct environments, including Operating System, Database, and Web Shopping.
\nLinks: Paper | GitHub | Leaderboard | Dataset
\nA challenging benchmark designed to evaluate the ability of web agents to automatically solve realistic and time-consuming tasks. It comprises 214 tasks that require navigating the open web, spanning multiple domains, and interacting with over 525 pages from 258 different websites.
\nLinks: Paper | GitHub | Leaderboard | Dataset
\nA challenging, contamination-free benchmark for LLMs that regularly releases new questions from recent information sources to ensure models are tested on novel problems rather than memorized answers.
\nLinks: Paper | GitHub | Leaderboard | Dataset
\nA highly challenging, multi-modal benchmark with 2,500 expert-level academic questions across a broad range of disciplines, designed to test models at the absolute frontier of human knowledge and require genuine reasoning capabilities rather than simple factual recall.
\nLinks: Paper | GitHub | Leaderboard | Dataset
\nA benchmark designed to evaluate the robustness of LLM safeguards against potential misuse relevant to national security and public safety, using expert-crafted adversarial prompts across domains like CBRNE and political violence.
\nLinks: Paper | Leaderboard
\nAims to evaluate the honesty of LLMs by disentangling it from factual accuracy. The benchmark measures whether models will knowingly contradict their established beliefs when subjected to pressure to lie.
\nLinks: Paper | GitHub | Leaderboard
\nA factuality benchmark designed to evaluate the ability of LLMs to answer short, fact-seeking questions. It aims to measure how well models "know what they know" and to identify "hallucinations" or factually incorrect outputs with single, indisputable answers.
\nLinks: Paper | GitHub | Dataset
\nA 1,000-prompt benchmark designed for evaluating the short-form factuality of LLMs, developed to address limitations in the original SimpleQA benchmark, such as noisy labels and topical biases, through a rigorous filtering process.
\nLinks: Paper | Leaderboard | Dataset
\nEvaluates LLMs' ability to generate long-form responses that are factually accurate and strictly "grounded" in provided context documents, thereby mitigating hallucination. Tasks require models to generate responses based exclusively on documents up to 32,000 tokens long.
\nLinks: Paper | GitHub | Leaderboard | Dataset
\nProvides comprehensive performance metrics for LLM agents across business domains.
\n\nA benchmark for evaluating LLMs and AI agents on their ability to resolve real-world software engineering issues. It comprises 2,294 problems sourced from GitHub issues across 12 popular Python repositories. The task is to generate a patch that resolves the issue.
\nLinks: Paper | GitHub | Leaderboard | Dataset
\nSWE-bench Verified is a human-validated subset of the original SWE-bench dataset, containing 500 samples that assess the capability of AI models to resolve real-world software engineering issues. To improve the reliability of evaluation, SWE-bench Verified was created in collaboration with OpenAI and involved professional software developers who screened each sample to ensure well-specified issue descriptions and appropriate unit tests.
\nLinks: Blog | Leaderboard
\nA benchmark for evaluating LLMs and AI agents on their ability to resolve real-world software engineering issues. It comprises 1,865 problems sourced from 41 diverse professional repositories. The task is to generate a patch that resolves the issue. Hidden Test set with 276 additional private tasks.
\nLinks: Paper | GitHub | Leaderboard | Dataset
\nA holistic and contamination-free benchmark for evaluating LLMs for code-related tasks. It continuously collects new problems from competitive programming platforms and assesses capabilities like self-repair, code execution, and test output prediction.
\nLinks: Paper | GitHub | Leaderboard
\nA multi-language benchmark designed to evaluate AI coding agents across diverse programming tasks and languages. It contains over 2,000 curated issues from 21 real-world repositories, covering Java, JavaScript, TypeScript, and Python.
\nLinks: Paper | GitHub | Leaderboard
\nA set of practical evaluations designed to measure how effectively LLMs can edit, refactor, and contribute to an existing codebase. These benchmarks include a code editing benchmark, a challenging refactoring benchmark, and a polyglot benchmark.
\nLinks: GitHub | Leaderboard
\nEvaluates coding and self-correction abilities of LLMs by testing them on 225 challenging Exercism coding exercises across multiple languages, including C++, Go, Java, JavaScript, Python, and Rust.
\nLinks: GitHub | Leaderboard
\nA standalone, self-hostable web environment for building autonomous agents. WebArena creates websites from four popular categories with functionality and data mimicking their real-world equivalents and introduces a benchmark on interpreting high-level commands.
\n\nA benchmark designed to assess the performance of multimodal agents on realistic, visually grounded web tasks. It extends WebArena with 910 new, diverse, and complex tasks that require agents to accurately process image-text inputs and execute actions on websites.
\n\nA benchmark designed to evaluate the performance of AI browser agents. It differentiates agent capabilities on information retrieval (READ) tasks from state-changing (WRITE) tasks across 452 live websites, encompassing 5,750 tasks.
\nLinks: Paper | GitHub | Dataset
\nA foundational benchmark designed for evaluating Large Multimodal Models (LMMs) and web agents on end-to-end, real-world navigation tasks across a diverse set of popular, live websites, integrating both textual (HTML) and visual (screenshots) information.
\nLinks: Paper | GitHub | Leaderboard
\nA simple yet challenging benchmark to measure the ability of agents to browse the web. It consists of 1,266 questions that demand persistent navigation of the internet to find hard-to-find, entangled information.
\n\nA comprehensive benchmark for developing and evaluating generalist web agents. The original dataset includes over 2,000 open-ended tasks collected from 137 real-world websites, with variants for evaluating performance on live websites.
\nLinks: Paper | GitHub | Leaderboard | Dataset
\nA comprehensive benchmark suite to evaluate general-purpose web-browsing AI agents, featuring over 50 interactive challenges crafted to be straightforward for humans but challenging for AI. It operates in a self-contained, hermetic testing environment.
\nLinks: Paper | GitHub | Leaderboard | Dataset
\nA benchmarking platform specifically designed to evaluate the safety and trustworthiness of autonomous web agents in realistic enterprise contexts, where policy compliance and safety are paramount.
\n\nA first-of-its-kind scalable, real computer environment for benchmarking multimodal agents on open-ended tasks within genuine operating systems, including Windows, macOS, and Ubuntu, featuring 369 real-world tasks.
\n\nA benchmark for evaluating advanced GUI-navigation AI agents on complex, multimodal, desktop-oriented tasks. It features 160 tasks across five levels of complexity and nine categories, designed to be easy for humans but challenging for AI.
\n\nA comprehensive suite of 13 benchmarks for evaluating Graphical User Interface (GUI) agents, focusing on the Vision Language Models (VLMs) that power them. It uses a vision-only evaluation stack without relying on accessibility trees or DOM information.
\nLinks: GitHub
\nA novel benchmark to rigorously evaluate AI agents in performing complex, realistic enterprise workflows. It expands upon the original WorkArena benchmark with 682 tasks that mimic the intricate operations of knowledge workers on the ServiceNow platform.
\nLinks: Paper
\nA dynamic benchmarking environment for autonomous agents that control mobile devices. It operates on a live Android emulator and features 116 hand-crafted tasks across 20 real-world Android applications, with millions of unique task variations.
\n\nA comprehensive Graphical User Interface (GUI) benchmark designed to evaluate AI agents across ten widely used desktop and web applications (e.g., PowerPoint, VSCode). It features 315 tasks with diverse initial states to simulate authentic human-computer interactions.
\n\nThe first comprehensive, multilingual, and interactive benchmark to evaluate GUI agents operating within the macOS environment. It features 202 multilingual tasks across 30 applications, with instructions and interfaces in five languages.
\n\nA pioneering benchmark to evaluate an LLM agent's ability to automate complex office workflows across multiple applications, such as Word, Excel, and email. It assesses long-horizon planning and proficiency in switching between applications.
\n\nEvaluates AI agents in realistic enterprise scenarios that require them to go beyond simple browser interaction, intelligently selecting tools like APIs and combining web UI interaction with API calls.
\nLinks: Paper | GitHub | Dataset
\nA benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks. It features long, compositional tasks in a simulated 3D environment.
\nLinks: Paper | GitHub | Leaderboard
\nA comprehensive benchmark designed to evaluate Multi-modal Large Language Models (MLLMs) as embodied agents. It spans diverse tasks in navigation, manipulation, and high-level planning across four simulated environments.
\n\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2025-10-15T00:00:00+00:00", "date_published": "2025-10-15T00:00:00+00:00", "external_url": "https://www.philschmid.de/benchmark-compedium", "id": "urn:aggr:item:10315b20dcb93abf029583e8e1cd084c758e0155", "summary": "An extensive compendium of over 50 benchmarks for evaluating AI agents, categorized into Function Calling and Tool Use, General Assistant and Reasoning, Coding and Software Engineering, and Computer Interaction.", "tags": [ "programming" ], "title": "AI Agent Benchmark Compendium", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-10-15-ai-agent-benchmark-compendium/" }, { "content_html": "For the past year, building an AI agent usually meant one thing: setting up a while loop, take a user prompt, send it to an LLM, parse a tool call, execute the tool, send the result back, and repeat. This is what we call a Shallow Agent or Agent 1.0.
\nThis architecture is fantastically simple for transactional tasks like "What's the weather in Tokyo and what should I wear?", but when asked to perform a task that requires 50 steps over three days, and they invariably get distracted, lose context, enter infinite loops, or hallucinates because the task requires too many steps for a single context window.
\nWe are seeing an architectural shift towards Deep Agents or Agents 2.0. These systems do not just react in a loop. They combine agentic patterns to plan, manage a persistent memory/state, and delegate work to specialized sub-agents to solve multi-step, complex problems.
\n![\"overview\"](\"https://www.philschmid.de/static/blog/agents-2.0-deep-agents/overview.png\")
To understand where we are going, we must understand where we are. Most agents today are "shallow". This means rely entirely on the LLM's context window (conversation history) as their state.
\nsearch("AAPL stock price")This architecture is stateless and ephemeral. The agent's entire "brain" is within the context window. When a task becomes complex, e.g. "Research 10 competitors, analyze their pricing models, build a comparison spreadsheet, and write a strategic summary" it will fail due to:
\nShallow agents are great at tasks that take 5-15 steps. They are terrible at tasks that take 500.
\nDeep Agents decouple planning from execution and manage memory external to the context window. The architecture consists of four pillars.
\nShallow agents plan implicitly via chain-of-thought ("I should do X, then Y"). Deep agents use tools to create and maintain an explicit plan, which can be To-Do list in a markdown document.
\nBetween every step, the agent reviews and updates this plan, marking steps as pending, in_progress, or completed or add notes. If a step fails, it doesn't just retry blindly, it updates the plan to accommodate the failure. This keeps the agent focused on the high-level task.
\nComplex tasks require specialization. Shallow Agents tries to be a jack-of-all-trades in one prompt. Deep Agents utilize an Orchestrator → Sub-Agent pattern.
\nThe Orchestrator delegates task(s) to sub-agent(s) each with a clean context. The sub-agent (e.g., a "Researcher," a "Coder," a "Writer") performs its tool call loops (searching, erroring, retrying), compiles the final answer, and returns only the synthesized answer to the Orchestrator.
\nTo prevent context window overflow, Deep Agents utilize external memory sources, like filesystem or vector databases as their source of truth. Frameworks like Claude Code and Manus give agents read/write access to them. An agent writes intermediate results (code, draft text, raw data). Subsequent agents reference file paths or queries to only retrieve what is necessary. This shifts the paradigm from "remembering everything" to "knowing where to find information."
Smarter models do not require less prompting, they require better context. You cannot get Agent 2.0 behavior with a prompt that says, "You are a helpful AI.". Deep Agents rely on highly detailed instructions sometimes thousands of tokens long. These define:
\nHow do these pillars come together? Let's look at a sequence diagram for a Deep Agent handling a complex request: "Research Quantum Computing and write a summary to a file."
\n![\"sequence\"](\"https://www.philschmid.de/static/blog/agents-2.0-deep-agents/sequence.png\")
Moving from Shallow Agents to Deep Agents (Agent 1.0 to Agent 2.0) isn't just about connecting an LLM to more tools. It is a shift from reactive loops to proactive architecture. It is about better engineering around the model.
\nImplementing explicit planning, hierarchical delegation via sub-agents, and persistent memory, allow us to control the context and by controlling the context, we control the complexity, unlocking the ability to solve problems that take hours or days, not just seconds.
\nThis overview was created with the help of deep and manual research. The term “Deep Agents” was notably popularized by the LangChain team to describe this architectural evolution.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2025-10-12T00:00:00+00:00", "date_published": "2025-10-12T00:00:00+00:00", "external_url": "https://www.philschmid.de/agents-2.0-deep-agents", "id": "urn:aggr:item:2cbed5041e3379ce2a9d1f1c7fbe04eb10d3db33", "summary": "An overview of the architectural shift from Shallow Agents (Agent 1.0) to Deep Agents (Agent 2.0) and how to build complex AI agents that can handle multi-step tasks over extended periods.", "tags": [ "programming" ], "title": "Agents 2.0: From Shallow Loops to Deep Agents", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-10-12-agents-2-0-from-shallow-loops-to-deep-agents/" }, { "content_html": "There is an increasing use of Subagents to reliably handle specific user goals. We can see this in tools like Claude Code and Poke.com. A subagent is a specialized Agent that is purpose-built for a single, well-defined task. Subagents can be explicit, where a user or model defines them for reuse or implicit and dynamically defined. This addresses a key limitation of monolithic AI agents context pollution. When a single, big and complex agent handles many tasks, its context window, number of tools, can become cluttered and less reliable.
\nSubagents are specialized AI agents. They are most of the time used in combination with an orchestrator, which delegates tasks to them. A subagent is just like a normal agent and has the same components. This includes a name for identification, a description of its capabilities, system instructions and a set of tools to interact with environments. It also has its own isolated context window.
\nPython
\n ┌───────────────────┐\n │ │\n └─────────┬─────────┘\n │\n ▼\n ┌───────────────────────────┐\n │ │\n │- │\n │- │\n │- │\n └───────────┬───────────────┘\n │\n┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐\n │ \n │\n ┌───────────┴───────────┬──────────────────┐\n ▼ ▼ ▼\n┌────────────────┐ ┌────────────────┐ ┌────────────────┐\n│ 1 │ │ 2 │ │ │\n│- │ │- │ │- │\n│- │ │- │ │- │\n│- │ │- │ │- │\n└───────┬────────┘ └─────────┬──────┘ └──────┬─────────┘\n │ │ │\n └───────────┬───────────┴──────────────────┘\n │\n ▼\n ┌───────────────────────────┐\n │ │\n │- │\n └─────────────┬─────────────┘\n │\n ▼\n ┌───────────────────┐\n │ │\n └───────────────────┘\nExplicit, User-defined Subagents are a permanent team of reusable specialists. Subagent can be defined in a static file or directly in code and the orchestrator uses them trough their name or when a prompt matches its description. Claude Code uses this method. The tools are manually listed in the definition, and the agent is stateless, meaning it starts fresh on every run.
Pros:
\nCons:
\nHere is a pseudo-code snippet of a definition file, inspired from Claude Code:
\nYAML
\n---\nname: "Code-Reviewer"\ndescription: "MUST BE USED for reviewing code against style guides and security practices."\ntools: \n---\nYou are an expert security code reviewer. Your purpose is to analyze code and identify vulnerabilities. Provide feedback in a numbered list.\n \nImplicit, On-the-Fly Subagents can be created temporary by an orchestrator to handle tasks as they come up. The orchestrator uses a tool (send_message_to_agent) to create and interact with the agent. The system dynamically assigns tools based on the user's natural language request from pre-defined pool which are needed to solve the task. Poke.com uses this method to created unlimited agents for specific user request. A key feature is that these agents can be stateful, e.g. keep context from previous runs when called with the same agent_name.
Here is a pseudo-code snippet for how the orchestrator calls the agent:
\nPython
\n# The orchestrator calls this tool to create or reuse a subagent\n\nBoth predefined and on-the-fly subagents will see increased usage. Dynamic agents might find more adoption in general B2C applications. Predefined agents are a great fit for more structured and repeatable enterprise workflows.
\nContext Engineering is everything. The key to success is giving the LLM the right information and tools at the right time. Using Subagents allows us to create a focused environment for the LLM with a clear, isolated context, specific system instructions, and a limited set of tools for the task. This improves performance and can reduce the cost of reaching the goal.
\nBut even with a subagent architecture, reliability is still a challenge for agentic systems, breaking down a complex task into smaller subagent functions can make them easier to test and evaluate in isolation.
\nModels are improving very fast. Don’t over-engineer a solution today that a simpler or better model can solve tomorrow.
\n", "date_modified": "2025-09-15T00:00:00+00:00", "date_published": "2025-09-15T00:00:00+00:00", "external_url": "https://www.philschmid.de/the-rise-of-subagents", "id": "urn:aggr:item:0bc559ef9933d39439550e849b5a9d4b762cda8b", "summary": "The rise of subagents is a trend in the AI community. We are seeing more and more use of subagents to reliably handle specific user goals.", "tags": [ "programming" ], "title": "The Rise of Subagents", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-09-15-the-rise-of-subagents/" }, { "content_html": "The most significant challenge in using AI for a product catalog is visual consistency. If the hero shot looks slightly different from the one in your detail shots, it erodes customer trust. Gemini 2.5 Flash Image Generation is our latest, fastest, and most efficient natively multimodal model. What makes Gemini special is its ability to understand both text and images together. This allows you to go beyond just creating pictures from words. You can upload an image and give text instructions to edit it, combine several pictures into one, or even apply the style of one image to another.
\nThe Core Capabilities of Gemini 2.5 Flash for E-Commerce:
\nThis guide will walk you through the 10 e-commerce use cases for Gemini 2.5 Flash, transforming your visual content from a costly bottleneck into a creative asset.
\nFirst, we create our one perfect hero image using a detailed text prompt. This image will serve as our consistent visual anchor for all the steps that follow.
\nNote: This might be the only step you should do manually by taking a real photo. All other steps can be done with AI. If you don't have a high-quality photo of your product, you can use Gemini to edit it.
\n\n\nA high-resolution, studio-lit product photograph of a [product description] on a [background surface]. The lighting is a [lighting setup] to [lighting purpose]. The camera angle is a [angle type] to showcase [specific feature]. Ultra-realistic, with sharp focus on [key detail].
\n
![\"master\"](\"https://www.philschmid.de/static/blog/gemini-image-generation-ecommerce/master.jpg\")
Using our master image ensures the sneaker in this new photo is identical to the one on the product page, reinforcing authenticity.
\n\n\nUsing the provided master image of [product], create a top-down, neatly arranged "flat lay" photograph. Place the exact sneaker from the image alongside all its included items: [item 1], [item 2], [item 3]. The items should be on a [surface description].
\n
![\"unboxing\"](\"https://www.philschmid.de/static/blog/gemini-image-generation-ecommerce/unboxing.jpg\")
We instruct Gemini to use our master image as the foundation, guaranteeing the lighting, colors, and textures are a perfect match.
\n\n\nUsing the provided master image of [product], re-frame the shot to be an extreme macro photograph. Focus exclusively on the [specific feature], making it the hero of the new image. The lighting and style should be preserved from the original image.
\n
![\"macro\"](\"https://www.philschmid.de/static/blog/gemini-image-generation-ecommerce/macro.jpg\")
Displaying all options in a single image helps customers compare and choose. We edit our master shot to create variations.
\n\n\nUsing the provided master image of [product], create a single composite image showing the original product side-by-side with its new variations: [variation 1 description] and [variation 2 description], all arranged against a clean background.
\n
![\"lineup\"](\"https://www.philschmid.de/static/blog/gemini-image-generation-ecommerce/lineup.jpg\")
Showing the sneaker on different feet is a powerful tool to prevent returns.
\n\n\nUsing the provided image of [product], create a single composite image showing it on three different sized feet: one small, one medium, and one large. The shots should be from the same angle to make comparison easy.
\n
![\"fit](\"https://www.philschmid.de/static/blog/gemini-image-generation-ecommerce/fit_guide.jpg\")
To ensure the highest consistency, we can generate our brand's model in a separate step before adding the product. This gives us precise control over the model's appearance and pose.
\n\n\nUsing the provided image of [product], create a close-up photograph of a [model description] actively using it. The focus is on the action of [describe the action], demonstrating the [specific feature].
\n
![\"in](\"https://www.philschmid.de/static/blog/gemini-image-generation-ecommerce/in_use.jpg\")
Now we'll place our consistent model and product into a full lifestyle scene. By referencing the master product shot, we ensure the sneaker is perfectly rendered, while the text prompt builds the complete environment around it. This is more efficient than creating a separate background and trying to composite images together.
\n\n\nUsing the provided image of [product], create a photorealistic lifestyle scene. The shot should feature the [model description] wearing the product while [performing an action] in a [location description]. The lighting and mood should be [lighting/mood description].
\n
![\"lifestyle\"](\"https://www.philschmid.de/static/blog/gemini-image-generation-ecommerce/lifestyle.jpg\")
Generate authentic-looking "customer photos" to build trust and relatability on social media.
\n\n\nUsing the provided image of [product], generate a realistic, user-generated style photo of it being used in an everyday situation. The style should look like a high-quality smartphone photo, slightly casual, with natural lighting.
\n
![\"ugc\"](\"https://www.philschmid.de/static/blog/gemini-image-generation-ecommerce/ugc.jpg\")
Create images designed for ad copy instead of slapping text over a busy photo.
\n\n\nUsing the provided image of [product], create a minimalist composition featuring the sneaker positioned in the [bottom-right/top-left/etc.] of the frame. The background should be a [background description], creating significant negative space for text.
\n
![\"ad](\"https://www.philschmid.de/static/blog/gemini-image-generation-ecommerce/ad_template.jpg\")
To create a "Shop the Look" image, providing exact images of other products is far better than just describing them. This ensures the composite photo is an accurate representation of the specific items you actually sell, creating a perfectly curated upsell opportunity.
\n\n\nCreate a new composite product photo by combining the items from the provided images. Take the [product 1 from image 1], the [product 2 from image 2], and the [product 3 from image 3]. Arrange them in a [style of arrangement, e.g., clean flat lay] on a [surface description].
\n
![\"upsell](\"https://www.philschmid.de/static/blog/gemini-image-generation-ecommerce/upsell_composite.jpg\")
In ten steps, we have built a comprehensive visual asset library for a single product that covers the entire customer journey. By starting with a "single source of truth" master image, we can ensure product consistency across every shot—from technical details to aspirational lifestyle scenes.
\nAI image generation with models like Gemini 2.5 Flash is fundamentally changing the e-commerce landscape. It democratizes professional-quality product photography, allowing brands of all sizes to create stunning, diverse, and on-brand visuals at a fraction of the cost and time.
\n", "date_modified": "2025-08-27T00:00:00+00:00", "date_published": "2025-08-27T00:00:00+00:00", "external_url": "https://www.philschmid.de/gemini-image-generation-product", "id": "urn:aggr:item:6e54873fc8fe61a4a04602828e1c63f180b86bea", "summary": "Learn how to use Gemini 2.5 Flash for product image generation.", "tags": [ "programming" ], "title": "The 10 Steps for product AI generation with Gemini 2.5 Flash", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-08-27-the-10-steps-for-product-ai-generation-with-gemini-2-5-flash/" }, { "content_html": "Imagine hiring a brilliant co-worker. They can reason, write, and research with incredible skill. But there’s a catch: every day, they forget everything they ever did, learned or said. This is the reality of most Agents today. They are powerful but are inherently stateless. We are making significant progress in reasoning and tool use, but the ability for Agents to remember past interactions, preferences, and learned skills remains heavily under explored.
\nThe context window of LLMs is a (currently) limited space where it processes information. This window has a limited bandwith, but should include the right information and tools, in the right format, at the right time for an LLM to perform a task. This is now known under Context Engineering.
\nIt is a delicate balance “of filling the context window with just the right information for the next step" - AK. Too little context, and the agent fails. Too much or irrelevant context, and costs rise while performance can decrease and the most powerful source for that "right information" is the agent's own memory/context.
\n![\"context-engineering.png\"](\"https://www.philschmid.de/static/blog/memory-in-agents/context-engineering.png\")
Since LLMs lack inherent memory, it must be intentionally engineered into the architecture. Like human cognition, agent memory operates on two different timelines, short-term memory and long-term memory.
\nIntegrating memory unlocks capabilities that are impossible for stateless models:
\nShort-term memory is the context window itself. It holds the system instructions, recent conversation history, current instructions, tool definitions and information relevant to the current interaction. It's “fast” and essential for the current task, but temporary and limited in size. It needs to be re-constructed for every call to the LLM.
\nLong-term memory requires external data stores, such as vector databases. It allows the agent to store and recall information across multiple sessions and extended periods. This is where true personalization and “learning” happens. The long-term memory is loaded into the short-term memory when relevant/helpful. Long-term memory can be specialized into three types:
\nThere are two strategies for how and when an agent writes or updates its memory: explicitly during interaction or implicitly via a background process.
\n![\"updates.png\"](\"https://www.philschmid.de/static/blog/memory-in-agents/updates.png\")
The agent updates its memory as a direct part of the ongoing interaction flow, before or after responding to the user using tool use.
\n✅ Pros: Memory is updated in real-time and immediately available for subsequent turns; can be transparent to the user.
\n❌ Cons: Can add latency to responses; increases complexity by mixing memory logic with agent logic; agent juggles memory tasks with response generation.
\nMemory updates occur as a separate, asynchronous background process. The agent responds to the user first, and a background task updates the memory later.
\n✅ Pros: No added latency to user-facing responses; keeps memory logic separate from core agent logic; allows the agent to focus on responding.
\n❌ Cons: Memory isn't updated instantly, potentially leading to outdated context for a short period; requires careful design of when and how often the background process runs.
\nImplementing Memory successfully in Agents is challenging as there is no direct feedback or impact of it. Memory system will grow over time and you must find the right balance between performance, accuracy, and operational cost to solve these key problems:
\nThe challenge alone are already hard to solve but what makes it even harder is the delayed feedback loop of memory systems. The consequences of a flawed memory strategy only surface over time, making it tough to measure and iterate effectively.
\nThere is a growing ecosystem of tools making it easier to implement Memory features. Frameworks like LangGraph, Mem0, Zep, ADK, Letta and others provide easy to use integrations for adding memory. I built a chatbot using mem0 and Google Gemini 2.5 Flash to remember user preferences.
\nToday's tools solve the core engineering challenges of storing and retrieving data. But for a successful future we will not just need bigger hard drive, but a smarter brain.
\nThis overview was created with the help of deep and manual research, drawing inspiration and information from several excellent resources, including:
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2025-08-04T00:00:00+00:00", "date_published": "2025-08-04T00:00:00+00:00", "external_url": "https://www.philschmid.de/memory-in-agents", "id": "urn:aggr:item:3aa05349c2aabdcb25068119d6ef2c7188785198", "summary": "Learn how to engineer long-term memory into stateless AI agents to overcome their biggest limitation and unlock true personalization.", "tags": [ "programming" ], "title": "Memory in Agents, Make LLMs remember.", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-08-04-memory-in-agents-make-llms-remember/" }, { "content_html": "The Gemini CLI is an open-source AI agent that brings the power of Gemini directly into your terminal, allowing you to perform a wide range of tasks such as coding, problem-solving, and task management using natural language. This cheatsheet provides a quick reference for installing, configuring, and using the Gemini CLI, with a focus on users authenticating via a Gemini API key.
\n![\"Gemini](\"https://www.philschmid.de/static/blog/gemini-cli-cheatsheet/architecture.png\")
Install Globally:
\nBash
\n\nRun without Installing:
\nBash
\n\nAuthenticate with an API key before first use. See the authentication guide for details.
\nGet Your Key: Get an API key from Google AI Studio.
\nSet Your Key: Make the key available to the CLI with one of these methods.
\nMethod 1: Shell Environment Variable Set the GEMINI_API_KEY environment variable. To use it across terminal sessions, add this line to your shell's profile (e.g., ~/.bashrc, ~/.zshrc).
Bash
\n\nMethod 2: Environment File Create a .env file in ~/.gemini/ for global use or ./.gemini/ for a specific project. The CLI automatically loads it.
# In .gemini/.env\nGEMINI_API_KEY="YOUR_GEMINI_API_KEY"\nInteractive Mode (REPL): Start a conversational session.
\nBash
\n\nNon-Interactive Mode: Pass a prompt and get a single response.
\nBash
\n\nPiping to the CLI: Pipe content to the CLI.
\nBash
\n | \nSandbox Mode: Run tools in a secure sandbox (requires Docker or Podman).
\nBash
\n\nOther Flags:
\n-m, --model <model>: Use a specific model.-i, --prompt-interactive <prompt>: Start an interactive session with an initial prompt.-d, --debug: Enable debug output.--yolo: Auto-approve all tool calls.--checkpointing: Save a project snapshot before file modifications. Use /restore to revert changes.settings.json)Customize the CLI by creating a settings.json file. Settings are applied with the following precedence:
.gemini/settings.json (overrides user).~/.gemini/settings.json/etc/gemini-cli/settings.json (applies to all users, has highest precedence).Example settings.json:
JSON
\n\nautoAccept: Auto-approve safe, read-only tool calls.sandbox: Isolate tool execution (e.g., true, "docker", or "podman").vimMode: Enable Vim-style editing for the input prompt.checkpointing: Enable the /restore command to undo file changes.includeDirectories: Define a multi-directory workspace.chatCompression: Configure automatic chat history compression.customThemes: Define your own color themes.usageStatisticsEnabled: Set to false to disable usage statistics.All details in the configuration guide.
\nGEMINI.md)Use GEMINI.md files to provide instructions to the model and tailor it to your project. Use /init to generate a starting GEMINI.md file for your project.
Hierarchical Loading: The CLI combines GEMINI.md files from multiple locations. More specific files override general ones. The loading order is:
~/.gemini/GEMINI.md (for instructions that apply to all your projects).GEMINI.md files.GEMINI.md files, allowing for component-specific instructions.Use /memory show to see the final combined context being sent to the model.
Modularizing Context with Imports: You can organize GEMINI.md files by importing other Markdown files with the @file.md syntax. This only supports .md files.
Example GEMINI.md using imports:
Markdown
\n \n- - \nMore in the Full context file guide.
\n.geminiignoreCreate a .geminiignore file in your project root to exclude files and directories from Gemini's tools, similar to .gitignore.
# .geminiignore\n/backups/\n*.log\nsecret-config.json\nlist_directory(path="/src"), glob(pattern="src/**/*.ts"), read_file(path="/path/to/file.txt"), write_file(file_path="/path/to/new_file.js", content="console.log('hello');"), replace(file_path="...", old_string="...", new_string="..."), search_file_content(pattern="myFunction", include="*.js")excludeTools in settings.json. For example: "excludeTools": ["run_shell_command(rm)"]google_web_search(query="Gemini API rate limits"), web_fetch(prompt="Summarize https://my-blog.com/article")save_memory(fact="My preferred CSS framework is Tailwind CSS.")Extend the CLI with your own tools by running Model Context Protocol (MCP) servers. Manage servers via settings.json or with the gemini mcp <add|list|remove> commands.
Capabilities:
\nExample mcpServers configuration:
JSON
\n"mcpServers": \nTransport (choose one):
\ncommand, args, cwd: Launch a local process via Stdiourl: SSE endpoint (e.g., "http://localhost:8080/sse")httpUrl: HTTP streaming endpoint (e.g., "http://localhost:8080/mcp")Optional:
\nenv: Environment variables. Use $VAR_NAME syntax to reference shell variables.headers: Key-value map of HTTP headers for url/httpUrl transports.timeout: Request timeout in milliseconds (default: 10 minutes).trust: Bypass all tool confirmations for this server.includeTools/excludeTools: Whitelist/blacklist specific tools. excludeTools takes precedence.Using OAuth take a look at mcp-server.md
\n/)| Command | \nDescription | \n
|---|---|
/compress | \nReplace the entire chat context with a summary to save tokens. | \n
/copy | \nCopy the last response to the clipboard. | \n
/mcp | \nList configured MCP servers and their available tools. | \n
/clear | \nClear the terminal screen and context (Ctrl+L also works). | \n
/tools | \nList available tools. | \n
/extensions | \nList active extensions. | \n
/stats | \nShow session token usage and savings. | \n
/memory show | \nShow the combined context from all GEMINI.md files. | \n
/memory refresh | \nReload all GEMINI.md files. | \n
/chat save <tag> | \nSave the current conversation with a tag. | \n
/chat resume <tag> | \nResume a saved conversation. | \n
/chat list | \nList saved conversation tags. | \n
/restore | \nList or restore a project state checkpoint. | \n
/auth | \nChange the current authentication method. | \n
/bug | \nFile an issue or bug report about the Gemini CLI. | \n
/help | \nDisplay help information and available commands. | \n
/theme | \nChange the CLI's visual theme. | \n
/quit | \nExit the Gemini CLI. | \n
/ide | \nManage integration with your IDE (e.g., install, enable). | \n
/settings | \nOpen a friendly editor to change your settings.json file. | \n
/vim | \nToggle Vim mode for input editing. | \n
/init | \nGenerate a starting GEMINI.md context file for your project. | \n
/directory | \nManage directories in a multi-directory workspace (e.g., add, show). | \n
@)Reference files or directories in your prompt. The CLI respects .gitignore and .geminiignore. You can also reference images, PDFs, audio, and video files.
Include a single file:
\n> Explain this code to me. @./src/main.js\nInclude an image:
\n> Describe what you see in this screenshot. @./ux-mockup.png\nInclude a whole directory (recursively):
\n> Refactor the code in this directory to use async/await. @./src/\n!)Run shell commands directly in the CLI.
\nRun a single command:
\n> !git status\nToggle Shell Mode: Enter ! by itself to switch to a persistent shell mode. Type ! again to exit.
| Shortcut | \nDescription | \n
|---|---|
Ctrl+L | \nClear the screen. | \n
Ctrl+V | \nPaste text or an image from the clipboard. | \n
Ctrl+Y | \nToggle YOLO mode (auto-approve all tool calls). | \n
Ctrl+X | \nOpen the current prompt in an external editor. | \n
Connect the CLI to VS Code for a more powerful, context-aware experience.
\n/ide install to set up and /ide enable to connect.Create custom commands using TOML files. Store them in ~/.gemini/commands/ (global) or <project>/.gemini/commands/ (project-specific). See the custom commands guide for more details.
Example: ~/.gemini/commands/test/gen.toml
Toml
\n# Invoked as: /test:gen "Create a test for the login button"\ndescription = "Generates a unit test based on a description."\nprompt = """\nYou are an expert test engineer. Based on the following requirement, please write a comprehensive unit test using the Jest testing framework.\n \nRequirement: {{args}}\n"""\nCreate extensions to add functionality. Place them in <workspace>/.gemini/extensions/ or ~/.gemini/extensions/. Each extension is a directory with a gemini-extension.json file that can configure MCP servers, tools, and context files. For more details, see the extensions guide.
For example:
\n<workspace>/.gemini/extensions/my-extension/gemini-extension.json\nJSON
\n\nWhen checkpointing is on, the CLI saves a project snapshot before tools modify files.
Enable in settings.json or with a flag:
Bash
\n\nRestore to a previous state:
\nBash
\n# List available checkpoints\n\n \n# Restore a specific checkpoint\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2025-07-24T00:00:00+00:00", "date_published": "2025-07-24T00:00:00+00:00", "external_url": "https://www.philschmid.de/gemini-cli-cheatsheet", "id": "urn:aggr:item:19de8e64f561acd5c88b7afc035dd113fbdbc3a9", "summary": "A comprehensive cheatsheet on using Google's Gemini CLI, covering installation, authentication, configuration, and core commands.", "tags": [ "programming" ], "title": "Google Gemini CLI Cheatsheet", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-07-24-google-gemini-cli-cheatsheet/" }, { "content_html": "July 22, 2025 4 minute read View Code
\nCode agents are transforming software development. But how do we safely let them execute code? Code sandboxes have evolved from basic security tools into essential development infrastructure.
\nToday, I'm launching Code Sandbox MCP, a lightweight, STDIO-based Model Context Protocol (MCP) Server, allowing AI assistants and LLM applications to safely execute code snippets using containerized environments. It is uses the llm-sandbox package for the containerization and execution of the code snippets.
\nIt exposes run_python_code and run_javascript_code tools, giving your AI agent the ability to execute code on your own infrastructure.
python3 -u code.py or javascript node -u code.js![\"Code](\"https://www.philschmid.de/static/blog/code-sandbox-mcp/code-sandbox-mcp.png\")
This entire workflow happens on your machine, locally or on your own server, keeping your data and code private.
\nIntegrating Code Sandbox MCP with your Python application using the Gemini SDK is incredibly straightforward. First, install the package:
\nBash
\n\nThen, you can use fastmcp to connect your Gemini client to the local code execution tool:
Python
\n\n\n\n \n# Configure the MCP client to use the local server\n = \n = \n \n\n \n = await \n # The model's response will include the output from the code execution\n \n \n\n \n \nYou can also supercharge the Gemini CLI with code execution capabilities. Simply add the server configuration to your ~/.gemini/settings.json file:
JSON
\n\n \nNow, when you use the Gemini CLI, it can automatically discover and use the run_python_code tool to answer your questions!
By leveraging llm-sandbox, Code Sandbox MCP benefits from multiple layers of security:
llm-sandbox framework supports security policies that can analyze code for dangerous patterns (e.g., os.system, file system access) before it even runs.I built Code Sandbox MCP primarily for my own use. When working with the Gemini CLI or other coding agents, I constantly struggled with testing code snippets in an isolated environment. Especially when you hand over more responsbility to the agent.
\nWhile managed solutions like Daytona and E2B are excellent for scaling or building customer facing agents, they didn't fit my specific needs. In my use cases, I wanted to give agents a way to execute code in an environment I can control and don't have to pay for.
\nI mean:
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2025-07-22T00:00:00+00:00", "date_published": "2025-07-22T00:00:00+00:00", "external_url": "https://www.philschmid.de/code-sandbox-mcp", "id": "urn:aggr:item:9d58da1a47535b20a07bb33b6ad139c77f97f156", "summary": "Code Sandbox MCP is a simple, self-hosted code interpreter for your AI agents. It allows you to execute code snippets in containerized environments.", "tags": [ "programming" ], "title": "Code Sandbox MCP: A Simple Code Interpreter for Your AI Agents", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-07-22-code-sandbox-mcp-a-simple-code-interpreter-for-your-ai/" }, { "content_html": "By default, large language models (LLMs) are stateless, which means they do not remember past conversations. This can make it difficult to create truly personal and helpful AI applications. This guide shows you how to add long-term memory to your Gemini 2.5 chatbot using the Gemini API and Mem0.
\nBy adding a memory system, your chatbot can:
\nIn this example, we will use mem0, an open-source tool for giving AI agents long-term memory and Gemini 2.5 Flash as the LLM. We will build a simple chatbot that saves what you talk about and uses that history to give you better, more personalized answers.
Mem0 is designed to equip AI agents with scalable long-term memory, effectively addressing the limitations of fixed context windows in LLMs. At its core, mem0 works by reactively extracting, consolidating, and retrieving salient information from ongoing conversations.
\n![\"Mem0\"](\"https://www.philschmid.de/static/blog/gemini-with-memory/mem0.png\")
The process is split into four steps:
\nIt uses vector embeddings to store and retrieve semantic information, maintaining user-specific context across sessions, and implementing efficient retrieval mechanisms for relevant past interactions.
\nThe first task is to install the google-genai Python SDK and mem0ai and obtain an API key.
Python
\n! - --\nFor building Memory we need to configure two main components:
\nmem0 to efficiently search and retrieve relevant memories based on their meaning when you ask a question.In this example, we will use Google's Gemini models for both tasks. We will use gemini-2.5-flash as our LLM and text-embedding-004 as our embedding model. We are going to use a local Qdrant instance as our vector store. Mem0 supports multiple vector stores incuding MongoDB, OpenSearch, Elasticsearch, Pgvector and more.
Python
\n\n\n \n# create client\n = \n \n = \n \n = \nTo simulate and simplify the process of adding memory. We will use dummy conversations between me (Philipp Schmid) and Gemini based on what can be seen on my blog. Conversations need to be in the messages format to be added.
Python
\n = \n = \n \n = \n = \n \n = \n = \nWe can manually search throught the memory to find relevant information. Let's try to find information about me.
\nPython
\n = \n \n# Print the first 3 memories\n\n \n \n# Has a professional background\n# Blog post will be personal\n# Wants to write a new blog post\nGreat! Now let's do some test on how including memory into our conversation changes the output. Our memmory should now include that Philipp is going to give a talk about agentic AI in the enterprise and some context about my hobbies. P.S. I prefer weightlifting over mountain biking.
\nLet see what Gemini will do when we ask it to help brainstorm about the upcoming talk.
\nPython
\n = \n \n = \n \n = \n \n\n# This is a fantastic request! .... Once you give me some input, we can refine the titles and visuals even further!\nAs expected, Gemini has no idea about the upcoming talk and my interests. Now lets add the memory to the conversation.
\nPython
\n# retrieve memories\n = \n = \n# extend system prompt\n = f\n \n# generate response\n = \n \n\n# This is a great idea ... Here are a few options ... Both options allow you to subtly connect to your interests in mountain climbing\nAwesome! Instead of needing to follow up with the user, Gemini generated two respones based on the memories, including the user's personal interests.
\nNow, lets combine all of this into an interactive chatbot. We separate the chatbot into two notebook cells, that we chat with Gemini, stop and start the chatbot again, but it has all the memories from the previous chat.
\nPython
\n\n\n \n = \n = \n = \n \n = \nAn example to test is you can start with saying something about you and where you are, e.g. "I am live in Nuremberg and today we have 30°C, how can i cool down?". Then stop (type 'exit'), restart (rerun the cell) the conversation and ask about "what the closest swimming pool is".
\nPython
\n\n # Retrieve relevant memories\n \n = \n = \n \n # Generate Assistant response\n = f\n = \n \n # Create new memories from the conversation we need to convert the history to a list of messages\n = \n \n \n return \n \n\n \n = \n \n = \n \n \n break\n \n = \n \n \n \n\nWe've seen how to overcome the stateless nature of LLMs by integrating long-term memory into a Gemini 2.5. By storing and retrieving information from past conversations, we can build AI assistants that are more personal, context-aware, and genuinely helpful. This allows them to remember user preferences, build on previous interactions, and provide a much richer and more natural conversational experience.
\nThanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.
\n", "date_modified": "2025-07-03T00:00:00+00:00", "date_published": "2025-07-03T00:00:00+00:00", "external_url": "https://www.philschmid.de/gemini-with-memory", "id": "urn:aggr:item:9c622f6143577d9c9ca72b5e0115bd6464d9f78d", "summary": "This guide shows you how to add long-term memory to your Gemini 2.5 chatbot using the Gemini API and Mem0.", "tags": [ "programming" ], "title": "Integrating Long-Term Memory with Gemini 2.5", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-07-03-integrating-long-term-memory-with-gemini-2-5/" } ], "language": "en", "title": "philschmid.de - RSS feed", "version": "https://jsonfeed.org/version/1.1" }