{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-04-23-how-to-use-transformers-js-in-a-chrome-extension/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "We recently released a Transformers.js demo browser extension powered by [Gemma 4 E2B](https://huggingface.co/google/gemma-4-E2B) to help users navigate the web.\n\nWhile building it, we ran into several practical observations about Manifest V3 runtimes, model loading, and messaging that are worth sharing.\n\n## Who this is for\n\nThis guide is for developers who want to run local AI features in a Chrome extension with Transformers.js under Manifest V3 constraints.\n\nBy the end, you will have the same architecture used in this project: a background service worker that hosts models, a side panel chat UI, and a content script for page-level actions.\n\n## What we will build\n\nIn this guide, we will recreate the core architecture of **Transformers.js Gemma 4 Browser Assistant**, using the published extension as a reference and the open-source codebase as the implementation map.\n\n- Live extension: [Chrome Web Store](https://chromewebstore.google.com/detail/transformerjs-gemma-4-bro/dhaknnnkcdkjhcclchmnfdhddoehoool)\n- Source code: [github.com/nico-martin/gemma4-browser-extension](https://github.com/nico-martin/gemma4-browser-extension)\n- End result: a background-hosted Transformers.js engine, a side panel chat UI, and a content script for page extraction and highlighting.\n\n## 1) Chrome extension architecture (MV3)\n\nBefore diving in, a quick scope note: I will not go deep on the React UI layer or Vite build configuration. The focus here is the high-level architecture decisions: what runs in each Chrome runtime and how those pieces are orchestrated.\n\nIf Manifest V3 is new to you, read this short overview first: [What is Manifest V3?](https://developer.chrome.com/docs/extensions/develop/migrate/what-is-mv3).\n\n### 1.1 Runtime contexts and entry points\n\nIn MV3, your architecture starts in [`public/manifest.json`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/public/manifest.json). This project defines three entry points:\n\n- `background.service_worker = background.js`, built from [`src/background/background.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/background.ts).\n- `side_panel.default_path = sidebar.html`, built from [`src/sidebar/index.html`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/sidebar/index.html).\n- `content_scripts[].js = content.js` with `matches: http(s)://*/*` and `run_at: document_idle`, built from [`src/content/content.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/content/content.ts).\n\nThe background service worker also handles `chrome.action.onClicked` to open the side panel for the active tab. Related entry point to know: a popup can be defined with `action.default_popup` and works well for quick actions. This project uses a side panel for persistent chat, but the orchestration pattern is the same.\n\n### 1.2 What runs where\n\nThe key design decision is to keep heavy orchestration in the background and keep UI/page logic thin.\n\n- Background ([`src/background/background.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/background.ts)) is the control plane: agent lifecycle, model initialization, tool execution, and shared services like feature extraction.\n- Side panel ([`src/sidebar/*`](https://github.com/nico-martin/gemma4-browser-extension/tree/main/src/sidebar)) is the interaction layer: chat input/output, streaming updates, and setup controls.\n- Content script ([`src/content/content.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/content/content.ts)) is the page bridge: DOM extraction and highlight actions.\n\nOne practical consequence of this division is that the conversation history also lives in background (`Agent.chatMessages`): the UI sends events like `AGENT_GENERATE_TEXT`, background appends the message, runs inference, then emits `MESSAGES_UPDATE` back to the side panel.\n\nThis split avoids duplicate model loads, keeps the UI responsive, and respects Chrome's security boundaries around DOM access.\n\n### 1.3 Messaging contract\n\nOnce runtimes are separated, messaging becomes the backbone. In this project, all messages are typed through enums in [`src/shared/types.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/shared/types.ts).\n\n- Side panel -> background (`BackgroundTasks`):\n - `CHECK_MODELS`, `INITIALIZE_MODELS`\n - `AGENT_INITIALIZE`, `AGENT_GENERATE_TEXT`, `AGENT_GET_MESSAGES`, `AGENT_CLEAR`\n - `EXTRACT_FEATURES`\n- Background -> side panel (`BackgroundMessages`):\n - `DOWNLOAD_PROGRESS`, `MESSAGES_UPDATE`\n- Background -> content (`ContentTasks`):\n - `EXTRACT_PAGE_DATA`, `HIGHLIGHT_ELEMENTS`, `CLEAR_HIGHLIGHTS`\n\nThe orchestration rule is simple: the background is the single coordinator; side panel and content script are specialized workers that request actions and render results.\n\nTypical request flow:\n\n1. Side panel sends `AGENT_GENERATE_TEXT`.\n2. Background appends to `Agent.chatMessages` and runs model/tool steps.\n3. Background emits `MESSAGES_UPDATE`.\n4. Side panel re-renders from the updated message list.\n\n## 2) Transformers.js integration details\n\n### 2.1 Models and responsibilities\n\nIn [`src/shared/constants.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/shared/constants.ts), this extension uses two model roles:\n\n- TextGeneration / LLM: [`onnx-community/gemma-4-E2B-it-ONNX`](https://huggingface.co/onnx-community/gemma-4-E2B-it-ONNX) (`text-generation`, `q4f16`)\n- VectorEmbeddings: [`onnx-community/all-MiniLM-L6-v2-ONNX`](https://huggingface.co/onnx-community/all-MiniLM-L6-v2-ONNX) (`feature-extraction`, `fp32`)\n\nThe split is intentional: Gemma 4 handles reasoning/tool decisions, while MiniLM generates vector embeddings for the semantic similarity search in `ask_website` and `find_history`.\n\n### 2.2 Where inference runs\n\nAll inference runs in background ([`src/background/background.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/background.ts)):\n\n- text generation via `pipeline(\"text-generation\", ...)` with consistent KV Caching enabled by our new `DynamicCache` class\n- embeddings via `pipeline(\"feature-extraction\", ...)` plus vector normalization\n\nThis gives a single model host for all tabs/sessions, avoids duplicate memory usage, and keeps the side panel UI responsive. Because models are loaded from the background service worker, artifacts are cached under the extension origin (`chrome-extension://`) rather than per-website origins, which gives one shared cache for the whole extension install.\n\nMV3 lifecycle note: service workers can be suspended and restarted, so model runtime state should be treated as recoverable and re-initialized when needed.\n\n### 2.3 Download and cache lifecycle\n\nThe model lifecycle is explicit:\n\n- `CHECK_MODELS` inspects what is already cached and estimates remaining download size.\n- `INITIALIZE_MODELS` downloads/initializes models and emits `DOWNLOAD_PROGRESS` to the UI.\n- Long-lived instances are reused after setup:\n - generation pipeline in [`src/background/agent/Agent.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/agent/Agent.ts)\n - embedding pipeline in [`src/background/utils/FeatureExtractor.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/utils/FeatureExtractor.ts)\n\nPermissions and privacy are part of the architecture, not a checkbox at the end. In this project, [`public/manifest.json`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/public/manifest.json) asks for `sidePanel`, `storage`, `scripting`, and `tabs`, plus `host_permissions` for `http(s)://*/*`:\n\n- `sidePanel`: required to open and control the side panel UX.\n- `storage`: required to persist tool/settings state across sessions.\n- `tabs` + `scripting`: required for tab-aware tools and page-level actions.\n- `host_permissions` on `http(s)://*/*`: required because content extraction/highlighting is designed to work on arbitrary websites.\n\nWhy keep this narrow: permissions define user trust and Chrome Web Store review risk. Request only what your features actually need, and state clearly that inference runs locally in the extension runtime so users understand where their data is processed.\n\n## 3) Agent and tool execution loop\n\n### 3.1 Tool-calling basics (why this layer exists)\n\nBefore the execution loop, it helps to understand how model tool calling works (the basis for any agentic workflow). You pass messages plus a tool schema (`name`, `description`, and `parameters`), and Transformers.js formats the actual prompt from those inputs using the model's chat template. Because chat templates are model-specific, the exact tool-call format depends on the model you use. With Gemma-4-style templates, the model emits a special tool-call token block when it decides to call one.\n\n```ts\nimport { pipeline } from \"@huggingface/transformers\";\n\nconst generator = await pipeline(\n \"text-generation\",\n \"onnx-community/gemma-4-E2B-it-ONNX\",\n {\n dtype: \"q4f16\",\n device: \"webgpu\",\n },\n);\n\nconst messages = [{ role: \"user\", content: \"What's the weather in Bern?\" }];\n\nconst output = await generator(messages, {\n max_new_tokens: 128,\n do_sample: false,\n tools: [\n {\n type: \"function\",\n function: {\n name: \"getWeather\",\n description: \"Get the weather in a location\",\n parameters: {\n type: \"object\",\n properties: {\n location: {\n type: \"string\",\n description: \"The location to get the weather for\",\n },\n },\n required: [\"location\"],\n },\n },\n },\n ],\n});\n```\n\nAt generation time, the model can emit output like:\n\n```\n<|tool_call>call:getWeather{location:<|\"|>Bern<|\"|>}\n```\n\nThat is exactly why this project has a normalization layer (`webMcp`) and a parser (`extractToolCalls`): model output must be converted into deterministic tool executions.\n\n### 3.2 Tool interface in this project\n\n[`src/background/agent/webMcp.tsx`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/agent/webMcp.tsx) normalizes extension tools into a model-friendly shape:\n\n- `name`, `description`, `inputSchema`, `execute`\n\nExample tools include `get_open_tabs`, `go_to_tab`, `open_url`, `close_tab`, `find_history`, `ask_website`, and `highlight_website_element`.\n\n### 3.3 Loop design (`Agent.runAgent`)\n\nThe core design choice here is to separate internal model messages from UI-facing chat messages:\n\n- Internal model transcript (`messages`): system/user/tool/assistant turns used for messages in `generator(...)`.\n- UI transcript (`chatMessages`): what the user sees, including streamed assistant text plus tool execution metadata (`tools`) and performance metrics.\n\nExecution flow:\n\n1. Add user input to `chatMessages`, create a placeholder assistant message, and stream tokens.\n2. Parse streamed/final model output with [`extractToolCalls.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/agent/extractToolCalls.ts) into `{ message, toolCalls }`.\n3. Keep the user-visible assistant message as plain text, while tool calls execute in background.\n4. Append tool results to the assistant tool metadata and feed results back as the next prompt turn.\n5. Repeat until no tool calls remain, then finalize assistant content + metrics.\n\nThis keeps user communication clean while preserving a deterministic tool loop in the background.\n\n## 4) Data boundaries and persistence\n\nState placement is another architectural decision that matters a lot in MV3. In this implementation, state is split by lifecycle and access pattern:\n\n- Conversation state: background memory (`Agent.chatMessages`) for fast turn-by-turn orchestration.\n- Tool preferences: `chrome.storage.local` so settings persist across sessions.\n- Semantic history vectors: IndexedDB (`VectorHistoryDB`) for larger local retrieval data.\n- Extracted page content: background cache (`WebsiteContentManager`) keyed by active URL.\n\nAs described in section 1.2, keeping conversation history in background gives one canonical state across UI updates. This keeps short-lived state in memory, durable settings in extension storage, and heavy retrieval data in a local database.\n\n## 5) Build and packaging notes\n\nYou do not need a complex build setup, but MV3 does require predictable outputs for each runtime.\n\n- Multi-entry build in [`vite.config.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/vite.config.ts):\n - [`src/sidebar/index.html`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/sidebar/index.html)\n - [`src/background/background.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/background.ts)\n - [`src/content/content.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/content/content.ts)\n- Ensure manifest-aligned output names/paths (`sidebar.html`, `background.js`, `content.js`).\n- Keep the content script as a self-contained output to avoid runtime chunk-loading issues.\n\nThe goal is simple: one artifact per Chrome entry point, in the exact place [`public/manifest.json`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/public/manifest.json) expects.\n\n## Final takeaway\n\nThe architecture choice that unlocks this whole project is clear separation of concerns: background owns orchestration and model execution, UI surfaces stay thin, and content scripts handle page access.\n\nThis project uses a side panel, but the same approach works for other setups:\n\n- Popup-first assistant: use `action.default_popup` for quick interactions, with background owning conversation state and model execution.\n- Side-panel copilot: keep long-running conversations in a persistent panel while background handles tool loops and caching.\n- Per-tab agents: keep one agent state per `tabId` in background when each tab should have its own context.\n- Hybrid UI (popup + side panel + options page): all UI entry points talk to the same background coordinator and reuse the same message contracts.\n\nThe practical rule is simple: decide where state lives (`global`, `tabId`, or site-scoped), keep that state and the model inference in background (basically as background services), and let UI/content runtimes act as focused clients.\n", "authors": [], "captured_at": "2026-10-19T01:26:53.998049939+00:00", "category": "ai", "content_html": "

We recently released a Transformers.js demo browser extension powered by Gemma 4 E2B to help users navigate the web.

\n

While building it, we ran into several practical observations about Manifest V3 runtimes, model loading, and messaging that are worth sharing.

\n

Who this is for

\n

This guide is for developers who want to run local AI features in a Chrome extension with Transformers.js under Manifest V3 constraints.

\n

By the end, you will have the same architecture used in this project: a background service worker that hosts models, a side panel chat UI, and a content script for page-level actions.

\n

What we will build

\n

In this guide, we will recreate the core architecture of Transformers.js Gemma 4 Browser Assistant, using the published extension as a reference and the open-source codebase as the implementation map.

\n\n

1) Chrome extension architecture (MV3)

\n

Before diving in, a quick scope note: I will not go deep on the React UI layer or Vite build configuration. The focus here is the high-level architecture decisions: what runs in each Chrome runtime and how those pieces are orchestrated.

\n

If Manifest V3 is new to you, read this short overview first: What is Manifest V3?.

\n

1.1 Runtime contexts and entry points

\n

In MV3, your architecture starts in public/manifest.json. This project defines three entry points:

\n\n

The background service worker also handles chrome.action.onClicked to open the side panel for the active tab. Related entry point to know: a popup can be defined with action.default_popup and works well for quick actions. This project uses a side panel for persistent chat, but the orchestration pattern is the same.

\n

1.2 What runs where

\n

The key design decision is to keep heavy orchestration in the background and keep UI/page logic thin.

\n\n

One practical consequence of this division is that the conversation history also lives in background (Agent.chatMessages): the UI sends events like AGENT_GENERATE_TEXT, background appends the message, runs inference, then emits MESSAGES_UPDATE back to the side panel.

\n

This split avoids duplicate model loads, keeps the UI responsive, and respects Chrome's security boundaries around DOM access.

\n

1.3 Messaging contract

\n

Once runtimes are separated, messaging becomes the backbone. In this project, all messages are typed through enums in src/shared/types.ts.

\n\n

The orchestration rule is simple: the background is the single coordinator; side panel and content script are specialized workers that request actions and render results.

\n

Typical request flow:

\n
    \n
  1. Side panel sends AGENT_GENERATE_TEXT.
    2. \n
  2. Background appends to Agent.chatMessages and runs model/tool steps.
    3. \n
  3. Background emits MESSAGES_UPDATE.
    4. \n
  4. Side panel re-renders from the updated message list.
    5. \n
\n

2) Transformers.js integration details

\n

2.1 Models and responsibilities

\n

In src/shared/constants.ts, this extension uses two model roles:

\n\n

The split is intentional: Gemma 4 handles reasoning/tool decisions, while MiniLM generates vector embeddings for the semantic similarity search in ask_website and find_history.

\n

2.2 Where inference runs

\n

All inference runs in background (src/background/background.ts):

\n\n

This gives a single model host for all tabs/sessions, avoids duplicate memory usage, and keeps the side panel UI responsive. Because models are loaded from the background service worker, artifacts are cached under the extension origin (chrome-extension://<extension-id>) rather than per-website origins, which gives one shared cache for the whole extension install.

\n

MV3 lifecycle note: service workers can be suspended and restarted, so model runtime state should be treated as recoverable and re-initialized when needed.

\n

2.3 Download and cache lifecycle

\n

The model lifecycle is explicit:

\n\n

Permissions and privacy are part of the architecture, not a checkbox at the end. In this project, public/manifest.json asks for sidePanel, storage, scripting, and tabs, plus host_permissions for http(s)://*/*:

\n\n

Why keep this narrow: permissions define user trust and Chrome Web Store review risk. Request only what your features actually need, and state clearly that inference runs locally in the extension runtime so users understand where their data is processed.

\n

3) Agent and tool execution loop

\n

3.1 Tool-calling basics (why this layer exists)

\n

Before the execution loop, it helps to understand how model tool calling works (the basis for any agentic workflow). You pass messages plus a tool schema (name, description, and parameters), and Transformers.js formats the actual prompt from those inputs using the model's chat template. Because chat templates are model-specific, the exact tool-call format depends on the model you use. With Gemma-4-style templates, the model emits a special tool-call token block when it decides to call one.

\n
import { pipeline } from "@huggingface/transformers";\n\nconst generator = await pipeline(\n  "text-generation",\n  "onnx-community/gemma-4-E2B-it-ONNX",\n  {\n    dtype: "q4f16",\n    device: "webgpu",\n  },\n);\n\nconst messages = [{ role: "user", content: "What's the weather in Bern?" }];\n\nconst output = await generator(messages, {\n  max_new_tokens: 128,\n  do_sample: false,\n  tools: [\n    {\n      type: "function",\n      function: {\n        name: "getWeather",\n        description: "Get the weather in a location",\n        parameters: {\n          type: "object",\n          properties: {\n            location: {\n              type: "string",\n              description: "The location to get the weather for",\n            },\n          },\n          required: ["location"],\n        },\n      },\n    },\n  ],\n});\n
\n

At generation time, the model can emit output like:

\n
<|tool_call>call:getWeather{location:<|"|>Bern<|"|>}<tool_call|>\n
\n

That is exactly why this project has a normalization layer (webMcp) and a parser (extractToolCalls): model output must be converted into deterministic tool executions.

\n

3.2 Tool interface in this project

\n

src/background/agent/webMcp.tsx normalizes extension tools into a model-friendly shape:

\n\n

Example tools include get_open_tabs, go_to_tab, open_url, close_tab, find_history, ask_website, and highlight_website_element.

\n

3.3 Loop design (Agent.runAgent)

\n

The core design choice here is to separate internal model messages from UI-facing chat messages:

\n\n

Execution flow:

\n
    \n
  1. Add user input to chatMessages, create a placeholder assistant message, and stream tokens.
    2. \n
  2. Parse streamed/final model output with extractToolCalls.ts into { message, toolCalls }.
    3. \n
  3. Keep the user-visible assistant message as plain text, while tool calls execute in background.
    4. \n
  4. Append tool results to the assistant tool metadata and feed results back as the next prompt turn.
    5. \n
  5. Repeat until no tool calls remain, then finalize assistant content + metrics.
    6. \n
\n

This keeps user communication clean while preserving a deterministic tool loop in the background.

\n

4) Data boundaries and persistence

\n

State placement is another architectural decision that matters a lot in MV3. In this implementation, state is split by lifecycle and access pattern:

\n\n

As described in section 1.2, keeping conversation history in background gives one canonical state across UI updates. This keeps short-lived state in memory, durable settings in extension storage, and heavy retrieval data in a local database.

\n

5) Build and packaging notes

\n

You do not need a complex build setup, but MV3 does require predictable outputs for each runtime.

\n\n

The goal is simple: one artifact per Chrome entry point, in the exact place public/manifest.json expects.

\n

Final takeaway

\n

The architecture choice that unlocks this whole project is clear separation of concerns: background owns orchestration and model execution, UI surfaces stay thin, and content scripts handle page access.

\n

This project uses a side panel, but the same approach works for other setups:

\n\n

The practical rule is simple: decide where state lives (global, tabId, or site-scoped), keep that state and the model inference in background (basically as background services), and let UI/content runtimes act as focused clients.

\n", "content_markdown": "We recently released a Transformers.js demo browser extension powered by [Gemma 4 E2B](https://huggingface.co/google/gemma-4-E2B) to help users navigate the web.\n\nWhile building it, we ran into several practical observations about Manifest V3 runtimes, model loading, and messaging that are worth sharing.\n\n## Who this is for\n\nThis guide is for developers who want to run local AI features in a Chrome extension with Transformers.js under Manifest V3 constraints.\n\nBy the end, you will have the same architecture used in this project: a background service worker that hosts models, a side panel chat UI, and a content script for page-level actions.\n\n## What we will build\n\nIn this guide, we will recreate the core architecture of **Transformers.js Gemma 4 Browser Assistant**, using the published extension as a reference and the open-source codebase as the implementation map.\n\n- Live extension: [Chrome Web Store](https://chromewebstore.google.com/detail/transformerjs-gemma-4-bro/dhaknnnkcdkjhcclchmnfdhddoehoool)\n- Source code: [github.com/nico-martin/gemma4-browser-extension](https://github.com/nico-martin/gemma4-browser-extension)\n- End result: a background-hosted Transformers.js engine, a side panel chat UI, and a content script for page extraction and highlighting.\n\n## 1) Chrome extension architecture (MV3)\n\nBefore diving in, a quick scope note: I will not go deep on the React UI layer or Vite build configuration. The focus here is the high-level architecture decisions: what runs in each Chrome runtime and how those pieces are orchestrated.\n\nIf Manifest V3 is new to you, read this short overview first: [What is Manifest V3?](https://developer.chrome.com/docs/extensions/develop/migrate/what-is-mv3).\n\n### 1.1 Runtime contexts and entry points\n\nIn MV3, your architecture starts in [`public/manifest.json`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/public/manifest.json). This project defines three entry points:\n\n- `background.service_worker = background.js`, built from [`src/background/background.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/background.ts).\n- `side_panel.default_path = sidebar.html`, built from [`src/sidebar/index.html`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/sidebar/index.html).\n- `content_scripts[].js = content.js` with `matches: http(s)://*/*` and `run_at: document_idle`, built from [`src/content/content.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/content/content.ts).\n\nThe background service worker also handles `chrome.action.onClicked` to open the side panel for the active tab. Related entry point to know: a popup can be defined with `action.default_popup` and works well for quick actions. This project uses a side panel for persistent chat, but the orchestration pattern is the same.\n\n### 1.2 What runs where\n\nThe key design decision is to keep heavy orchestration in the background and keep UI/page logic thin.\n\n- Background ([`src/background/background.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/background.ts)) is the control plane: agent lifecycle, model initialization, tool execution, and shared services like feature extraction.\n- Side panel ([`src/sidebar/*`](https://github.com/nico-martin/gemma4-browser-extension/tree/main/src/sidebar)) is the interaction layer: chat input/output, streaming updates, and setup controls.\n- Content script ([`src/content/content.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/content/content.ts)) is the page bridge: DOM extraction and highlight actions.\n\nOne practical consequence of this division is that the conversation history also lives in background (`Agent.chatMessages`): the UI sends events like `AGENT_GENERATE_TEXT`, background appends the message, runs inference, then emits `MESSAGES_UPDATE` back to the side panel.\n\nThis split avoids duplicate model loads, keeps the UI responsive, and respects Chrome's security boundaries around DOM access.\n\n### 1.3 Messaging contract\n\nOnce runtimes are separated, messaging becomes the backbone. In this project, all messages are typed through enums in [`src/shared/types.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/shared/types.ts).\n\n- Side panel -> background (`BackgroundTasks`):\n - `CHECK_MODELS`, `INITIALIZE_MODELS`\n - `AGENT_INITIALIZE`, `AGENT_GENERATE_TEXT`, `AGENT_GET_MESSAGES`, `AGENT_CLEAR`\n - `EXTRACT_FEATURES`\n- Background -> side panel (`BackgroundMessages`):\n - `DOWNLOAD_PROGRESS`, `MESSAGES_UPDATE`\n- Background -> content (`ContentTasks`):\n - `EXTRACT_PAGE_DATA`, `HIGHLIGHT_ELEMENTS`, `CLEAR_HIGHLIGHTS`\n\nThe orchestration rule is simple: the background is the single coordinator; side panel and content script are specialized workers that request actions and render results.\n\nTypical request flow:\n\n1. Side panel sends `AGENT_GENERATE_TEXT`.\n2. Background appends to `Agent.chatMessages` and runs model/tool steps.\n3. Background emits `MESSAGES_UPDATE`.\n4. Side panel re-renders from the updated message list.\n\n## 2) Transformers.js integration details\n\n### 2.1 Models and responsibilities\n\nIn [`src/shared/constants.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/shared/constants.ts), this extension uses two model roles:\n\n- TextGeneration / LLM: [`onnx-community/gemma-4-E2B-it-ONNX`](https://huggingface.co/onnx-community/gemma-4-E2B-it-ONNX) (`text-generation`, `q4f16`)\n- VectorEmbeddings: [`onnx-community/all-MiniLM-L6-v2-ONNX`](https://huggingface.co/onnx-community/all-MiniLM-L6-v2-ONNX) (`feature-extraction`, `fp32`)\n\nThe split is intentional: Gemma 4 handles reasoning/tool decisions, while MiniLM generates vector embeddings for the semantic similarity search in `ask_website` and `find_history`.\n\n### 2.2 Where inference runs\n\nAll inference runs in background ([`src/background/background.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/background.ts)):\n\n- text generation via `pipeline(\"text-generation\", ...)` with consistent KV Caching enabled by our new `DynamicCache` class\n- embeddings via `pipeline(\"feature-extraction\", ...)` plus vector normalization\n\nThis gives a single model host for all tabs/sessions, avoids duplicate memory usage, and keeps the side panel UI responsive. Because models are loaded from the background service worker, artifacts are cached under the extension origin (`chrome-extension://`) rather than per-website origins, which gives one shared cache for the whole extension install.\n\nMV3 lifecycle note: service workers can be suspended and restarted, so model runtime state should be treated as recoverable and re-initialized when needed.\n\n### 2.3 Download and cache lifecycle\n\nThe model lifecycle is explicit:\n\n- `CHECK_MODELS` inspects what is already cached and estimates remaining download size.\n- `INITIALIZE_MODELS` downloads/initializes models and emits `DOWNLOAD_PROGRESS` to the UI.\n- Long-lived instances are reused after setup:\n - generation pipeline in [`src/background/agent/Agent.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/agent/Agent.ts)\n - embedding pipeline in [`src/background/utils/FeatureExtractor.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/utils/FeatureExtractor.ts)\n\nPermissions and privacy are part of the architecture, not a checkbox at the end. In this project, [`public/manifest.json`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/public/manifest.json) asks for `sidePanel`, `storage`, `scripting`, and `tabs`, plus `host_permissions` for `http(s)://*/*`:\n\n- `sidePanel`: required to open and control the side panel UX.\n- `storage`: required to persist tool/settings state across sessions.\n- `tabs` + `scripting`: required for tab-aware tools and page-level actions.\n- `host_permissions` on `http(s)://*/*`: required because content extraction/highlighting is designed to work on arbitrary websites.\n\nWhy keep this narrow: permissions define user trust and Chrome Web Store review risk. Request only what your features actually need, and state clearly that inference runs locally in the extension runtime so users understand where their data is processed.\n\n## 3) Agent and tool execution loop\n\n### 3.1 Tool-calling basics (why this layer exists)\n\nBefore the execution loop, it helps to understand how model tool calling works (the basis for any agentic workflow). You pass messages plus a tool schema (`name`, `description`, and `parameters`), and Transformers.js formats the actual prompt from those inputs using the model's chat template. Because chat templates are model-specific, the exact tool-call format depends on the model you use. With Gemma-4-style templates, the model emits a special tool-call token block when it decides to call one.\n\n```ts\nimport { pipeline } from \"@huggingface/transformers\";\n\nconst generator = await pipeline(\n \"text-generation\",\n \"onnx-community/gemma-4-E2B-it-ONNX\",\n {\n dtype: \"q4f16\",\n device: \"webgpu\",\n },\n);\n\nconst messages = [{ role: \"user\", content: \"What's the weather in Bern?\" }];\n\nconst output = await generator(messages, {\n max_new_tokens: 128,\n do_sample: false,\n tools: [\n {\n type: \"function\",\n function: {\n name: \"getWeather\",\n description: \"Get the weather in a location\",\n parameters: {\n type: \"object\",\n properties: {\n location: {\n type: \"string\",\n description: \"The location to get the weather for\",\n },\n },\n required: [\"location\"],\n },\n },\n },\n ],\n});\n```\n\nAt generation time, the model can emit output like:\n\n```\n<|tool_call>call:getWeather{location:<|\"|>Bern<|\"|>}\n```\n\nThat is exactly why this project has a normalization layer (`webMcp`) and a parser (`extractToolCalls`): model output must be converted into deterministic tool executions.\n\n### 3.2 Tool interface in this project\n\n[`src/background/agent/webMcp.tsx`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/agent/webMcp.tsx) normalizes extension tools into a model-friendly shape:\n\n- `name`, `description`, `inputSchema`, `execute`\n\nExample tools include `get_open_tabs`, `go_to_tab`, `open_url`, `close_tab`, `find_history`, `ask_website`, and `highlight_website_element`.\n\n### 3.3 Loop design (`Agent.runAgent`)\n\nThe core design choice here is to separate internal model messages from UI-facing chat messages:\n\n- Internal model transcript (`messages`): system/user/tool/assistant turns used for messages in `generator(...)`.\n- UI transcript (`chatMessages`): what the user sees, including streamed assistant text plus tool execution metadata (`tools`) and performance metrics.\n\nExecution flow:\n\n1. Add user input to `chatMessages`, create a placeholder assistant message, and stream tokens.\n2. Parse streamed/final model output with [`extractToolCalls.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/agent/extractToolCalls.ts) into `{ message, toolCalls }`.\n3. Keep the user-visible assistant message as plain text, while tool calls execute in background.\n4. Append tool results to the assistant tool metadata and feed results back as the next prompt turn.\n5. Repeat until no tool calls remain, then finalize assistant content + metrics.\n\nThis keeps user communication clean while preserving a deterministic tool loop in the background.\n\n## 4) Data boundaries and persistence\n\nState placement is another architectural decision that matters a lot in MV3. In this implementation, state is split by lifecycle and access pattern:\n\n- Conversation state: background memory (`Agent.chatMessages`) for fast turn-by-turn orchestration.\n- Tool preferences: `chrome.storage.local` so settings persist across sessions.\n- Semantic history vectors: IndexedDB (`VectorHistoryDB`) for larger local retrieval data.\n- Extracted page content: background cache (`WebsiteContentManager`) keyed by active URL.\n\nAs described in section 1.2, keeping conversation history in background gives one canonical state across UI updates. This keeps short-lived state in memory, durable settings in extension storage, and heavy retrieval data in a local database.\n\n## 5) Build and packaging notes\n\nYou do not need a complex build setup, but MV3 does require predictable outputs for each runtime.\n\n- Multi-entry build in [`vite.config.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/vite.config.ts):\n - [`src/sidebar/index.html`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/sidebar/index.html)\n - [`src/background/background.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/background/background.ts)\n - [`src/content/content.ts`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/src/content/content.ts)\n- Ensure manifest-aligned output names/paths (`sidebar.html`, `background.js`, `content.js`).\n- Keep the content script as a self-contained output to avoid runtime chunk-loading issues.\n\nThe goal is simple: one artifact per Chrome entry point, in the exact place [`public/manifest.json`](https://github.com/nico-martin/gemma4-browser-extension/blob/main/public/manifest.json) expects.\n\n## Final takeaway\n\nThe architecture choice that unlocks this whole project is clear separation of concerns: background owns orchestration and model execution, UI surfaces stay thin, and content scripts handle page access.\n\nThis project uses a side panel, but the same approach works for other setups:\n\n- Popup-first assistant: use `action.default_popup` for quick interactions, with background owning conversation state and model execution.\n- Side-panel copilot: keep long-running conversations in a persistent panel while background handles tool loops and caching.\n- Per-tab agents: keep one agent state per `tabId` in background when each tab should have its own context.\n- Hybrid UI (popup + side panel + options page): all UI entry points talk to the same background coordinator and reuse the same message contracts.\n\nThe practical rule is simple: decide where state lives (`global`, `tabId`, or site-scoped), keep that state and the model inference in background (basically as background services), and let UI/content runtimes act as focused clients.\n", "dateCreated": "2026-10-19T01:26:53.998049939+00:00", "date_created": "2026-04-23T00:00:00+00:00", "date_modified": "2026-04-23T00:00:00+00:00", "date_published": "2026-04-23T00:00:00+00:00", "description": "We recently released a Transformers.js demo browser extension powered by Gemma 4 E2B to help users navigate the web. While building it, we ran into several practical observations about Manifest V3 runtimes, model loading, and messaging…", "external_url": "https://huggingface.co/blog/transformersjs-chrome-extension", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2026/04/2026-04-23-how-to-use-transformers-js-in-a-chrome-extension.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/04/2026-04-23-how-to-use-transformers-js-in-a-chrome-extension.md" }, "id": "urn:aggr:item:d654e5f1d41943957a523eb1bb48815264990a61", "isBasedOn": { "@id": "https://huggingface.co/blog/transformersjs-chrome-extension" }, "mainEntity": { "@id": "https://huggingface.co/blog/transformersjs-chrome-extension", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-04-23-how-to-use-transformers-js-in-a-chrome-extension/#webpage" }, "dateModified": "2026-04-23T00:00:00+00:00", "datePublished": "2026-04-23T00:00:00+00:00", "headline": "How to Use Transformers.js in a Chrome Extension", "image": { "@type": "ImageObject", "caption": null, "height": 90, "url": "https://huggingface.co/blog/assets/transformersjs-chrome-extension/thumbnail.jpg", "width": 160 }, "timeRequired": "PT8M", "url": "https://huggingface.co/blog/transformersjs-chrome-extension", "wordCount": 1616 }, "name": "Archived snapshot: How to Use Transformers.js in a Chrome Extension", "preview": { "alt": null, "color": null, "height": 90, "placeholder": { "data_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", "hash": "" }, "url": "https://huggingface.co/blog/assets/transformersjs-chrome-extension/thumbnail.jpg", "width": 160 }, "reading_minutes": 8, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-19T01:26:53.998049939+00:00", "title": "How to Use Transformers.js in a Chrome Extension", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-04-23-how-to-use-transformers-js-in-a-chrome-extension/", "word_count": 1616 }