{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-01-17-how-to-use-anthropic-mcp-server-with-open-llms-openai-or/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "AI agents are evolving from simple chatbots to (semi-)autonomous systems capable of complex reasoning, planning, and interaction with the real world. These agents are not just theoretical concepts; they will be deployed in production across various verticals, tackling increasingly more complex and longer-running tasks. As their capabilities grow, so does the challenge of managing and scaling them efficiently.\n\nCurrently, AI agents are designed with all tools, structures, and resources consolidated in a single monolithic application. This could become increasingly challenging as the system grows, leading to maintenance difficulties, hindering innovation, and creating bottlenecks in development processes.\n\nTo address these challenges, the Model Context Protocol (MCP), an open standard, was proposed by Anthropic. It tries to decouple components and define clear interfaces. In this blog post, you will learn how to use MCP Servers with any open LLM, OpenAI, or Google Gemini. You will learn how to build a simple CLI Agent that can control and interact with a SQLite database.\n\n## What is the Model Context Protocol (MCP)?\n\nThe [Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) is an open standard that enables developers to build secure agents and complex workflows on top of LLMs. MCP allows AI Agents to connect seamlessly with various external data sources and tools by standardizing how applications provide context to LLMs. MCP implements a Client-Server Architecture where the AI Agents initiate clients and communicate with servers. The current components of MCP servers include:\n\n1. **Tools**: These are functions that LLMs can call to perform specific actions, e.g. weather API\n2. **Resources**: These are data sources that LLMs can access, similar to GET endpoints in a REST API. Resources provide data without performing significant computation.\n3. **Prompts**: These are pre-defined templates to use tools or resources in the most optimal way.\n\nMCP allows AI Agents to interact with multiple resources through a unified protocol. Tools and resources can be independently updated, tested, extended, and reused across different platforms without the need to copy code.\n\nHowever, there are still limitations and missing features, including remote MCP Servers or how to handle authentication correctly. The good news is that this is being worked on!\n\n*Note: If you want to see a Google Gemini example, check out the [repository](https://github.com/philschmid/mcp-openai-gemini-llama-example/blob/master/sqlite_gemini_mcp_agent.py).*\n\n## How to use Anthropic MCP Server with open LLMs, OpenAI, or Google Gemini\n\nWe are going to build a simple CLI Agent that can control an SQLite database through an MCP Server. We will use the official [SQLite MCP Server](https://github.com/modelcontextprotocol/servers/tree/main/src/sqlite) using Docker. The architecture is quite simple:\n\n1. First, we need to create our LLM client, e.g. `OpenAI` or `GenerativeModel`\n2. Initiate an MCP client and connect to our SQLite MCP Server\n3. Load existing tools, resources, and prompts provided by the MCP Server\n4. Convert tools into LLM-compatible function calling tools (JSON Schema) with callable to our MCP Server\n5. Create a custom System Message based on available MCP features.\n6. Start interactive loop waiting for user inputs.\n\nIn our example, we are going to use [Meta Llama 3.3 70B instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) hosted on Hugging Face Inference API with the `openai` SDK.\n\nNote: Before we can start make sure you have `docker` setup and are logged into your Hugging Face account:\n\nBash\n\n```bash\nhuggingface-cli login --token YOUR_TOKEN\n```\n\nBefore we can start coding we need to install our required libraries. We are going to use the `openai` SDK to interact with the Hugging Face Inference API and we need the official python `mcp` SDK.\n\nBash\n\n```bash\npip install huggingface_hub openai \"mcp==1.1.2\"\n```\n\n**Note: Different from other blog posts, you will find the full code below as a file, including detailed code comments on the different sections.**\n\nPython\n\n```python\nimport json\nfrom huggingface_hub import get_token\nfrom openai import AsyncOpenAI\nfrom mcp import ClientSession, StdioServerParameters\nfrom mcp.client.stdio import stdio_client\nfrom typing import Any, List\nimport asyncio\n \nMODEL_ID = \"meta-llama/Llama-3.3-70B-Instruct\"\n \n# System prompt that guides the LLM's behavior and capabilities\n# This helps the model understand its role and available tools\nSYSTEM_PROMPT = \"\"\"You are a helpful assistant capable of accessing external functions and engaging in casual chat. Use the responses from these function calls to provide accurate and informative answers. The answers should be natural and hide the fact that you are using tools to access real-time information. Guide the user about available tools and their capabilities. Always utilize tools to access real-time information when required. Engage in a friendly manner to enhance the chat experience.\n \n# Tools\n \n{tools}\n \n# Notes \n \n- Ensure responses are based on the latest information available from function calls.\n- Maintain an engaging, supportive, and friendly tone throughout the dialogue.\n- Always highlight the potential of available tools to assist users comprehensively.\"\"\"\n \n \n# Initialize OpenAI client with HuggingFace inference API\n# This allows us to use Llama models through HuggingFace's API\nclient = AsyncOpenAI(\n base_url=\"https://api-inference.huggingface.co/v1/\",\n api_key=get_token(),\n)\n \n \nclass MCPClient:\n \"\"\"\n A client class for interacting with the MCP (Model Control Protocol) server.\n This class manages the connection and communication with the SQLite database through MCP.\n \"\"\"\n \n def __init__(self, server_params: StdioServerParameters):\n \"\"\"Initialize the MCP client with server parameters\"\"\"\n self.server_params = server_params\n self.session = None\n self._client = None\n \n async def __aenter__(self):\n \"\"\"Async context manager entry\"\"\"\n await self.connect()\n return self\n \n async def __aexit__(self, exc_type, exc_val, exc_tb):\n \"\"\"Async context manager exit\"\"\"\n if self.session:\n await self.session.__aexit__(exc_type, exc_val, exc_tb)\n if self._client:\n await self._client.__aexit__(exc_type, exc_val, exc_tb)\n \n async def connect(self):\n \"\"\"Establishes connection to MCP server\"\"\"\n self._client = stdio_client(self.server_params)\n self.read, self.write = await self._client.__aenter__()\n session = ClientSession(self.read, self.write)\n self.session = await session.__aenter__()\n await self.session.initialize()\n \n async def get_available_tools(self) -> List[Any]:\n \"\"\"\n Retrieve a list of available tools from the MCP server.\n \"\"\"\n if not self.session:\n raise RuntimeError(\"Not connected to MCP server\")\n \n tools = await self.session.list_tools()\n _, tools_list = tools\n _, tools_list = tools_list\n return tools_list\n \n def call_tool(self, tool_name: str) -> Any:\n \"\"\"\n Create a callable function for a specific tool.\n This allows us to execute database operations through the MCP server.\n \n Args:\n tool_name: The name of the tool to create a callable for\n \n Returns:\n A callable async function that executes the specified tool\n \"\"\"\n if not self.session:\n raise RuntimeError(\"Not connected to MCP server\")\n \n async def callable(*args, **kwargs):\n response = await self.session.call_tool(tool_name, arguments=kwargs)\n return response.content[0].text\n \n return callable\n \n \nasync def agent_loop(query: str, tools: dict, messages: List[dict] = None):\n \"\"\"\n Main interaction loop that processes user queries using the LLM and available tools.\n \n This function:\n 1. Sends the user query to the LLM with context about available tools\n 2. Processes the LLM's response, including any tool calls\n 3. Returns the final response to the user\n \n Args:\n query: User's input question or command\n tools: Dictionary of available database tools and their schemas\n messages: List of messages to pass to the LLM, defaults to None\n \"\"\"\n messages = (\n [\n {\n \"role\": \"system\",\n \"content\": SYSTEM_PROMPT.format(\n tools=\"\\n- \".join(\n [\n f\"{t['name']}: {t['schema']['function']['description']}\"\n for t in tools.values()\n ]\n )\n ), # Creates System prompt based on available MCP server tools\n },\n ]\n if messages is None\n else messages # reuse existing messages if provided\n )\n # add user query to the messages list\n messages.append({\"role\": \"user\", \"content\": query})\n \n # Query LLM with the system prompt, user query, and available tools\n first_response = await client.chat.completions.create(\n model=MODEL_ID,\n messages=messages,\n tools=([t[\"schema\"] for t in tools.values()] if len(tools) > 0 else None),\n max_tokens=4096,\n temperature=0,\n )\n # detect how the LLM call was completed:\n # tool_calls: if the LLM used a tool\n # stop: If the LLM generated a general response, e.g. \"Hello, how can I help you today?\"\n stop_reason = (\n \"tool_calls\"\n if first_response.choices[0].message.tool_calls is not None\n else first_response.choices[0].finish_reason\n )\n \n if stop_reason == \"tool_calls\":\n # Extract tool use details from response\n for tool_call in first_response.choices[0].message.tool_calls:\n arguments = (\n json.loads(tool_call.function.arguments)\n if isinstance(tool_call.function.arguments, str)\n else tool_call.function.arguments\n )\n # Call the tool with the arguments using our callable initialized in the tools dict\n tool_result = await tools[tool_call.function.name][\"callable\"](**arguments)\n # Add the tool result to the messages list\n messages.append(\n {\n \"role\": \"tool\",\n \"tool_call_id\": tool_call.id,\n \"name\": tool_call.function.name,\n \"content\": json.dumps(tool_result),\n }\n )\n \n # Query LLM with the user query and the tool results\n new_response = await client.chat.completions.create(\n model=MODEL_ID,\n messages=messages,\n )\n \n elif stop_reason == \"stop\":\n # If the LLM stopped on its own, use the first response\n new_response = first_response\n \n else:\n raise ValueError(f\"Unknown stop reason: {stop_reason}\")\n \n # Add the LLM response to the messages list\n messages.append(\n {\"role\": \"assistant\", \"content\": new_response.choices[0].message.content}\n )\n \n # Return the LLM response and messages\n return new_response.choices[0].message.content, messages\n \n \nasync def main():\n \"\"\"\n Main function that sets up the MCP server, initializes tools, and runs the interactive loop.\n The server is run in a Docker container to ensure isolation and consistency.\n \"\"\"\n # Configure Docker-based MCP server for SQLite\n server_params = StdioServerParameters(\n command=\"docker\",\n args=[\n \"run\",\n \"--rm\", # Remove container after exit\n \"-i\", # Interactive mode\n \"-v\", # Mount volume\n \"mcp-test:/mcp\", # Map local volume to container path\n \"mcp/sqlite\", # Use SQLite MCP image\n \"--db-path\",\n \"/mcp/test.db\", # Database file path inside container\n ],\n env=None,\n )\n \n # Start MCP client and create interactive session\n async with MCPClient(server_params) as mcp_client:\n # Get available database tools and prepare them for the LLM\n mcp_tools = await mcp_client.get_available_tools()\n # Convert MCP tools into a format the LLM can understand and use\n tools = {\n tool.name: {\n \"name\": tool.name,\n \"callable\": mcp_client.call_tool(\n tool.name\n ), # returns a callable function for the rpc call\n \"schema\": {\n \"type\": \"function\",\n \"function\": {\n \"name\": tool.name,\n \"description\": tool.description,\n \"parameters\": tool.inputSchema,\n },\n },\n }\n for tool in mcp_tools\n if tool.name\n != \"list_tables\" # Excludes list_tables tool as it has an incorrect schema\n }\n \n # Start interactive prompt loop for user queries\n messages = None\n while True:\n try:\n # Get user input and check for exit commands\n user_input = input(\"\\nEnter your prompt (or 'quit' to exit): \")\n if user_input.lower() in [\"quit\", \"exit\", \"q\"]:\n break\n \n # Process the prompt and run agent loop\n response, messages = await agent_loop(user_input, tools, messages)\n print(\"\\nResponse:\", response)\n # print(\"\\nMessages:\", messages)\n except KeyboardInterrupt:\n print(\"\\nExiting...\")\n break\n except Exception as e:\n print(f\"\\nError occurred: {e}\")\n \n \nif __name__ == \"__main__\":\n asyncio.run(main())\n```\n\nYou can run the agent and start chatting with\n\nPython\n\n```python\npython sqlite_llama_mcp_agent.py\n```\n\nBelow is an example conversation of what you can do and how it response.\n\n```plaintext\nEnter your prompt (or 'quit' to exit): HEllo \nResponse: Hello! How can 1 assist you today?\n \nEnter your prompt (or 'quit' to exit): Who are you?\nResponse: I am a helpful assistant capable of accessing external functions and engaging in casual chat. I can use the available tools to provide accurate and infor mative answers. The available tools include read_query,\nwrite_query, create_table, describe_table, and append_insight. I can guide you about these tools and their capabilities, and I will utilize them to access real-time information when required. \n \nEnter your prompt (or 'quit' to exit): Cool! Create a new table for fake products. Each product should have a name and a price. Generate 10 fake products based on video games.\nResponse: The table has been created successfully with the following schema:\n- name (TEXT): The name of the product.\n- price (REAL): The price of the product.\n \nHere are 10 fake products based on video games:\n1. \"The Last of Us\" T-shirt - $19.99\n2. Minecraft Creeper Plush Toy - $14.99\n3. Grand Theft Auto V Poster - $9,99\n4. The Legend of Zelda: Breath of the Wild Strategy Guide - $24.99\n5. Call of Duty: Modern Warfare Gaming Keyboard - $69.99\n6. World of Warcraft Subscription Card (3 months) - $39.99\n7. Assassin's Creed Odyssey Action Figure - $29.99\n8. Fortnite Monopoly Board Game - $29.99 \n9. Resident Evil 2 Remake Collector's Edition - $99.99\n10. Pokemon Sword and Shield Nintendo Switch Bundle - $399.99\n \nLet me know if you'd like to add more products or perform any other operations on the table!\n```\n\nFeel free to change the code, include debugging statements to better understand the outputs from each step.\n\n- [Google Gemini Example](https://github.com/philschmid/mcp-openai-gemini-llama-example/blob/master/sqlite_gemini_mcp_agent.py)\n\n## Conclusion\n\nThis simple example illustrates how you can get started with building an AI Agent using MCP Servers alongside open LLMs, OpenAI, or Google Gemini. It demonstrates the core principles of how an agent might connect to a server, interact with tools, and respond to user commands. It is not a framework or a reliable implementation—its purpose is to help you understand the building blocks of a distributed AI Agent architecture using MCP Server.\n\nI’m working on a small toolkit to make implementing AI Agents via MCP easier, so stay tuned for more updates.\n\nIf you have any questions, feedback, or ideas, please dm me on [X](https://x.com/_philschmid) or [LinkedIn](https://www.linkedin.com/in/philipp-schmid-a6a2bb196/). I am excited to hear about how you are experimenting and pushing the boundaries of AI agents.\n", "authors": [], "captured_at": "2026-09-27T19:29:15.293681927+00:00", "category": "programming", "content_html": "

AI agents are evolving from simple chatbots to (semi-)autonomous systems capable of complex reasoning, planning, and interaction with the real world. These agents are not just theoretical concepts; they will be deployed in production across various verticals, tackling increasingly more complex and longer-running tasks. As their capabilities grow, so does the challenge of managing and scaling them efficiently.

\n

Currently, AI agents are designed with all tools, structures, and resources consolidated in a single monolithic application. This could become increasingly challenging as the system grows, leading to maintenance difficulties, hindering innovation, and creating bottlenecks in development processes.

\n

To address these challenges, the Model Context Protocol (MCP), an open standard, was proposed by Anthropic. It tries to decouple components and define clear interfaces. In this blog post, you will learn how to use MCP Servers with any open LLM, OpenAI, or Google Gemini. You will learn how to build a simple CLI Agent that can control and interact with a SQLite database.

\n

What is the Model Context Protocol (MCP)?

\n

The Model Context Protocol (MCP) is an open standard that enables developers to build secure agents and complex workflows on top of LLMs. MCP allows AI Agents to connect seamlessly with various external data sources and tools by standardizing how applications provide context to LLMs. MCP implements a Client-Server Architecture where the AI Agents initiate clients and communicate with servers. The current components of MCP servers include:

\n
    \n
  1. Tools: These are functions that LLMs can call to perform specific actions, e.g. weather API
    2. \n
  2. Resources: These are data sources that LLMs can access, similar to GET endpoints in a REST API. Resources provide data without performing significant computation.
    3. \n
  3. Prompts: These are pre-defined templates to use tools or resources in the most optimal way.
    4. \n
\n

MCP allows AI Agents to interact with multiple resources through a unified protocol. Tools and resources can be independently updated, tested, extended, and reused across different platforms without the need to copy code.

\n

However, there are still limitations and missing features, including remote MCP Servers or how to handle authentication correctly. The good news is that this is being worked on!

\n

Note: If you want to see a Google Gemini example, check out the repository.

\n

How to use Anthropic MCP Server with open LLMs, OpenAI, or Google Gemini

\n

We are going to build a simple CLI Agent that can control an SQLite database through an MCP Server. We will use the official SQLite MCP Server using Docker. The architecture is quite simple:

\n
    \n
  1. First, we need to create our LLM client, e.g. OpenAI or GenerativeModel
    2. \n
  2. Initiate an MCP client and connect to our SQLite MCP Server
    3. \n
  3. Load existing tools, resources, and prompts provided by the MCP Server
    4. \n
  4. Convert tools into LLM-compatible function calling tools (JSON Schema) with callable to our MCP Server
    5. \n
  5. Create a custom System Message based on available MCP features.
    6. \n
  6. Start interactive loop waiting for user inputs.
    7. \n
\n

In our example, we are going to use Meta Llama 3.3 70B instruct hosted on Hugging Face Inference API with the openai SDK.

\n

Note: Before we can start make sure you have docker setup and are logged into your Hugging Face account:

\n

Bash

\n
huggingface-cli login --token YOUR_TOKEN\n
\n

Before we can start coding we need to install our required libraries. We are going to use the openai SDK to interact with the Hugging Face Inference API and we need the official python mcp SDK.

\n

Bash

\n
pip install huggingface_hub openai "mcp==1.1.2"\n
\n

Note: Different from other blog posts, you will find the full code below as a file, including detailed code comments on the different sections.

\n

Python

\n
import json\nfrom huggingface_hub import get_token\nfrom openai import AsyncOpenAI\nfrom mcp import ClientSession, StdioServerParameters\nfrom mcp.client.stdio import stdio_client\nfrom typing import Any, List\nimport asyncio\n \nMODEL_ID = "meta-llama/Llama-3.3-70B-Instruct"\n \n# System prompt that guides the LLM's behavior and capabilities\n# This helps the model understand its role and available tools\nSYSTEM_PROMPT = """You are a helpful assistant capable of accessing external functions and engaging in casual chat. Use the responses from these function calls to provide accurate and informative answers. The answers should be natural and hide the fact that you are using tools to access real-time information. Guide the user about available tools and their capabilities. Always utilize tools to access real-time information when required. Engage in a friendly manner to enhance the chat experience.\n \n# Tools\n \n{tools}\n \n# Notes \n \n- Ensure responses are based on the latest information available from function calls.\n- Maintain an engaging, supportive, and friendly tone throughout the dialogue.\n- Always highlight the potential of available tools to assist users comprehensively."""\n \n \n# Initialize OpenAI client with HuggingFace inference API\n# This allows us to use Llama models through HuggingFace's API\nclient = AsyncOpenAI(\n    base_url="https://api-inference.huggingface.co/v1/",\n    api_key=get_token(),\n)\n \n \nclass MCPClient:\n    """\n    A client class for interacting with the MCP (Model Control Protocol) server.\n    This class manages the connection and communication with the SQLite database through MCP.\n    """\n \n    def __init__(self, server_params: StdioServerParameters):\n        """Initialize the MCP client with server parameters"""\n        self.server_params = server_params\n        self.session = None\n        self._client = None\n \n    async def __aenter__(self):\n        """Async context manager entry"""\n        await self.connect()\n        return self\n \n    async def __aexit__(self, exc_type, exc_val, exc_tb):\n        """Async context manager exit"""\n        if self.session:\n            await self.session.__aexit__(exc_type, exc_val, exc_tb)\n        if self._client:\n            await self._client.__aexit__(exc_type, exc_val, exc_tb)\n \n    async def connect(self):\n        """Establishes connection to MCP server"""\n        self._client = stdio_client(self.server_params)\n        self.read, self.write = await self._client.__aenter__()\n        session = ClientSession(self.read, self.write)\n        self.session = await session.__aenter__()\n        await self.session.initialize()\n \n    async def get_available_tools(self) -> List[Any]:\n        """\n        Retrieve a list of available tools from the MCP server.\n        """\n        if not self.session:\n            raise RuntimeError("Not connected to MCP server")\n \n        tools = await self.session.list_tools()\n        _, tools_list = tools\n        _, tools_list = tools_list\n        return tools_list\n \n    def call_tool(self, tool_name: str) -> Any:\n        """\n        Create a callable function for a specific tool.\n        This allows us to execute database operations through the MCP server.\n \n        Args:\n            tool_name: The name of the tool to create a callable for\n \n        Returns:\n            A callable async function that executes the specified tool\n        """\n        if not self.session:\n            raise RuntimeError("Not connected to MCP server")\n \n        async def callable(*args, **kwargs):\n            response = await self.session.call_tool(tool_name, arguments=kwargs)\n            return response.content[0].text\n \n        return callable\n \n \nasync def agent_loop(query: str, tools: dict, messages: List[dict] = None):\n    """\n    Main interaction loop that processes user queries using the LLM and available tools.\n \n    This function:\n    1. Sends the user query to the LLM with context about available tools\n    2. Processes the LLM's response, including any tool calls\n    3. Returns the final response to the user\n \n    Args:\n        query: User's input question or command\n        tools: Dictionary of available database tools and their schemas\n        messages: List of messages to pass to the LLM, defaults to None\n    """\n    messages = (\n        [\n            {\n                "role": "system",\n                "content": SYSTEM_PROMPT.format(\n                    tools="\\n- ".join(\n                        [\n                            f"{t['name']}: {t['schema']['function']['description']}"\n                            for t in tools.values()\n                        ]\n                    )\n                ),  # Creates System prompt based on available MCP server tools\n            },\n        ]\n        if messages is None\n        else messages  # reuse existing messages if provided\n    )\n    # add user query to the messages list\n    messages.append({"role": "user", "content": query})\n \n    # Query LLM with the system prompt, user query, and available tools\n    first_response = await client.chat.completions.create(\n        model=MODEL_ID,\n        messages=messages,\n        tools=([t["schema"] for t in tools.values()] if len(tools) > 0 else None),\n        max_tokens=4096,\n        temperature=0,\n    )\n    # detect how the LLM call was completed:\n    # tool_calls: if the LLM used a tool\n    # stop: If the LLM generated a general response, e.g. "Hello, how can I help you today?"\n    stop_reason = (\n        "tool_calls"\n        if first_response.choices[0].message.tool_calls is not None\n        else first_response.choices[0].finish_reason\n    )\n \n    if stop_reason == "tool_calls":\n        # Extract tool use details from response\n        for tool_call in first_response.choices[0].message.tool_calls:\n            arguments = (\n                json.loads(tool_call.function.arguments)\n                if isinstance(tool_call.function.arguments, str)\n                else tool_call.function.arguments\n            )\n            # Call the tool with the arguments using our callable initialized in the tools dict\n            tool_result = await tools[tool_call.function.name]["callable"](**arguments)\n            # Add the tool result to the messages list\n            messages.append(\n                {\n                    "role": "tool",\n                    "tool_call_id": tool_call.id,\n                    "name": tool_call.function.name,\n                    "content": json.dumps(tool_result),\n                }\n            )\n \n        # Query LLM with the user query and the tool results\n        new_response = await client.chat.completions.create(\n            model=MODEL_ID,\n            messages=messages,\n        )\n \n    elif stop_reason == "stop":\n        # If the LLM stopped on its own, use the first response\n        new_response = first_response\n \n    else:\n        raise ValueError(f"Unknown stop reason: {stop_reason}")\n \n    # Add the LLM response to the messages list\n    messages.append(\n        {"role": "assistant", "content": new_response.choices[0].message.content}\n    )\n \n    # Return the LLM response and messages\n    return new_response.choices[0].message.content, messages\n \n \nasync def main():\n    """\n    Main function that sets up the MCP server, initializes tools, and runs the interactive loop.\n    The server is run in a Docker container to ensure isolation and consistency.\n    """\n    # Configure Docker-based MCP server for SQLite\n    server_params = StdioServerParameters(\n        command="docker",\n        args=[\n            "run",\n            "--rm",  # Remove container after exit\n            "-i",  # Interactive mode\n            "-v",  # Mount volume\n            "mcp-test:/mcp",  # Map local volume to container path\n            "mcp/sqlite",  # Use SQLite MCP image\n            "--db-path",\n            "/mcp/test.db",  # Database file path inside container\n        ],\n        env=None,\n    )\n \n    # Start MCP client and create interactive session\n    async with MCPClient(server_params) as mcp_client:\n        # Get available database tools and prepare them for the LLM\n        mcp_tools = await mcp_client.get_available_tools()\n        # Convert MCP tools into a format the LLM can understand and use\n        tools = {\n            tool.name: {\n                "name": tool.name,\n                "callable": mcp_client.call_tool(\n                    tool.name\n                ),  # returns a callable function for the rpc call\n                "schema": {\n                    "type": "function",\n                    "function": {\n                        "name": tool.name,\n                        "description": tool.description,\n                        "parameters": tool.inputSchema,\n                    },\n                },\n            }\n            for tool in mcp_tools\n            if tool.name\n            != "list_tables"  # Excludes list_tables tool as it has an incorrect schema\n        }\n \n        # Start interactive prompt loop for user queries\n        messages = None\n        while True:\n            try:\n                # Get user input and check for exit commands\n                user_input = input("\\nEnter your prompt (or 'quit' to exit): ")\n                if user_input.lower() in ["quit", "exit", "q"]:\n                    break\n \n                # Process the prompt and run agent loop\n                response, messages = await agent_loop(user_input, tools, messages)\n                print("\\nResponse:", response)\n                # print("\\nMessages:", messages)\n            except KeyboardInterrupt:\n                print("\\nExiting...")\n                break\n            except Exception as e:\n                print(f"\\nError occurred: {e}")\n \n \nif __name__ == "__main__":\n    asyncio.run(main())\n
\n

You can run the agent and start chatting with

\n

Python

\n
python sqlite_llama_mcp_agent.py\n
\n

Below is an example conversation of what you can do and how it response.

\n
Enter your prompt (or 'quit' to exit): HEllo \nResponse: Hello! How can 1 assist you today?\n \nEnter your prompt (or 'quit' to exit): Who are you?\nResponse: I am a helpful assistant capable of accessing external functions and engaging in casual chat. I can use the available tools to provide accurate and infor mative answers. The available tools include read_query,\nwrite_query, create_table, describe_table, and append_insight. I can guide you about these tools and their capabilities, and I will utilize them to access real-time information when required. \n \nEnter your prompt (or 'quit' to exit): Cool! Create a new table for fake products. Each product should have a name and a price. Generate 10 fake products based on video games.\nResponse: The table has been created successfully with the following schema:\n- name (TEXT): The name of the product.\n- price (REAL): The price of the product.\n \nHere are 10 fake products based on video games:\n1. "The Last of Us" T-shirt - $19.99\n2. Minecraft Creeper Plush Toy - $14.99\n3. Grand Theft Auto V Poster - $9,99\n4. The Legend of Zelda: Breath of the Wild Strategy Guide - $24.99\n5. Call of Duty: Modern Warfare Gaming Keyboard - $69.99\n6. World of Warcraft Subscription Card (3 months) - $39.99\n7. Assassin's Creed Odyssey Action Figure - $29.99\n8. Fortnite Monopoly Board Game - $29.99    \n9. Resident Evil 2 Remake Collector's Edition - $99.99\n10. Pokemon Sword and Shield Nintendo Switch Bundle - $399.99\n \nLet me know if you'd like to add more products or perform any other operations on the table!\n
\n

Feel free to change the code, include debugging statements to better understand the outputs from each step.

\n\n

Conclusion

\n

This simple example illustrates how you can get started with building an AI Agent using MCP Servers alongside open LLMs, OpenAI, or Google Gemini. It demonstrates the core principles of how an agent might connect to a server, interact with tools, and respond to user commands. It is not a framework or a reliable implementation—its purpose is to help you understand the building blocks of a distributed AI Agent architecture using MCP Server.

\n

I’m working on a small toolkit to make implementing AI Agents via MCP easier, so stay tuned for more updates.

\n

If you have any questions, feedback, or ideas, please dm me on X or LinkedIn. I am excited to hear about how you are experimenting and pushing the boundaries of AI agents.

\n", "content_markdown": "AI agents are evolving from simple chatbots to (semi-)autonomous systems capable of complex reasoning, planning, and interaction with the real world. These agents are not just theoretical concepts; they will be deployed in production across various verticals, tackling increasingly more complex and longer-running tasks. As their capabilities grow, so does the challenge of managing and scaling them efficiently.\n\nCurrently, AI agents are designed with all tools, structures, and resources consolidated in a single monolithic application. This could become increasingly challenging as the system grows, leading to maintenance difficulties, hindering innovation, and creating bottlenecks in development processes.\n\nTo address these challenges, the Model Context Protocol (MCP), an open standard, was proposed by Anthropic. It tries to decouple components and define clear interfaces. In this blog post, you will learn how to use MCP Servers with any open LLM, OpenAI, or Google Gemini. You will learn how to build a simple CLI Agent that can control and interact with a SQLite database.\n\n## What is the Model Context Protocol (MCP)?\n\nThe [Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) is an open standard that enables developers to build secure agents and complex workflows on top of LLMs. MCP allows AI Agents to connect seamlessly with various external data sources and tools by standardizing how applications provide context to LLMs. MCP implements a Client-Server Architecture where the AI Agents initiate clients and communicate with servers. The current components of MCP servers include:\n\n1. **Tools**: These are functions that LLMs can call to perform specific actions, e.g. weather API\n2. **Resources**: These are data sources that LLMs can access, similar to GET endpoints in a REST API. Resources provide data without performing significant computation.\n3. **Prompts**: These are pre-defined templates to use tools or resources in the most optimal way.\n\nMCP allows AI Agents to interact with multiple resources through a unified protocol. Tools and resources can be independently updated, tested, extended, and reused across different platforms without the need to copy code.\n\nHowever, there are still limitations and missing features, including remote MCP Servers or how to handle authentication correctly. The good news is that this is being worked on!\n\n*Note: If you want to see a Google Gemini example, check out the [repository](https://github.com/philschmid/mcp-openai-gemini-llama-example/blob/master/sqlite_gemini_mcp_agent.py).*\n\n## How to use Anthropic MCP Server with open LLMs, OpenAI, or Google Gemini\n\nWe are going to build a simple CLI Agent that can control an SQLite database through an MCP Server. We will use the official [SQLite MCP Server](https://github.com/modelcontextprotocol/servers/tree/main/src/sqlite) using Docker. The architecture is quite simple:\n\n1. First, we need to create our LLM client, e.g. `OpenAI` or `GenerativeModel`\n2. Initiate an MCP client and connect to our SQLite MCP Server\n3. Load existing tools, resources, and prompts provided by the MCP Server\n4. Convert tools into LLM-compatible function calling tools (JSON Schema) with callable to our MCP Server\n5. Create a custom System Message based on available MCP features.\n6. Start interactive loop waiting for user inputs.\n\nIn our example, we are going to use [Meta Llama 3.3 70B instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) hosted on Hugging Face Inference API with the `openai` SDK.\n\nNote: Before we can start make sure you have `docker` setup and are logged into your Hugging Face account:\n\nBash\n\n```bash\nhuggingface-cli login --token YOUR_TOKEN\n```\n\nBefore we can start coding we need to install our required libraries. We are going to use the `openai` SDK to interact with the Hugging Face Inference API and we need the official python `mcp` SDK.\n\nBash\n\n```bash\npip install huggingface_hub openai \"mcp==1.1.2\"\n```\n\n**Note: Different from other blog posts, you will find the full code below as a file, including detailed code comments on the different sections.**\n\nPython\n\n```python\nimport json\nfrom huggingface_hub import get_token\nfrom openai import AsyncOpenAI\nfrom mcp import ClientSession, StdioServerParameters\nfrom mcp.client.stdio import stdio_client\nfrom typing import Any, List\nimport asyncio\n \nMODEL_ID = \"meta-llama/Llama-3.3-70B-Instruct\"\n \n# System prompt that guides the LLM's behavior and capabilities\n# This helps the model understand its role and available tools\nSYSTEM_PROMPT = \"\"\"You are a helpful assistant capable of accessing external functions and engaging in casual chat. Use the responses from these function calls to provide accurate and informative answers. The answers should be natural and hide the fact that you are using tools to access real-time information. Guide the user about available tools and their capabilities. Always utilize tools to access real-time information when required. Engage in a friendly manner to enhance the chat experience.\n \n# Tools\n \n{tools}\n \n# Notes \n \n- Ensure responses are based on the latest information available from function calls.\n- Maintain an engaging, supportive, and friendly tone throughout the dialogue.\n- Always highlight the potential of available tools to assist users comprehensively.\"\"\"\n \n \n# Initialize OpenAI client with HuggingFace inference API\n# This allows us to use Llama models through HuggingFace's API\nclient = AsyncOpenAI(\n base_url=\"https://api-inference.huggingface.co/v1/\",\n api_key=get_token(),\n)\n \n \nclass MCPClient:\n \"\"\"\n A client class for interacting with the MCP (Model Control Protocol) server.\n This class manages the connection and communication with the SQLite database through MCP.\n \"\"\"\n \n def __init__(self, server_params: StdioServerParameters):\n \"\"\"Initialize the MCP client with server parameters\"\"\"\n self.server_params = server_params\n self.session = None\n self._client = None\n \n async def __aenter__(self):\n \"\"\"Async context manager entry\"\"\"\n await self.connect()\n return self\n \n async def __aexit__(self, exc_type, exc_val, exc_tb):\n \"\"\"Async context manager exit\"\"\"\n if self.session:\n await self.session.__aexit__(exc_type, exc_val, exc_tb)\n if self._client:\n await self._client.__aexit__(exc_type, exc_val, exc_tb)\n \n async def connect(self):\n \"\"\"Establishes connection to MCP server\"\"\"\n self._client = stdio_client(self.server_params)\n self.read, self.write = await self._client.__aenter__()\n session = ClientSession(self.read, self.write)\n self.session = await session.__aenter__()\n await self.session.initialize()\n \n async def get_available_tools(self) -> List[Any]:\n \"\"\"\n Retrieve a list of available tools from the MCP server.\n \"\"\"\n if not self.session:\n raise RuntimeError(\"Not connected to MCP server\")\n \n tools = await self.session.list_tools()\n _, tools_list = tools\n _, tools_list = tools_list\n return tools_list\n \n def call_tool(self, tool_name: str) -> Any:\n \"\"\"\n Create a callable function for a specific tool.\n This allows us to execute database operations through the MCP server.\n \n Args:\n tool_name: The name of the tool to create a callable for\n \n Returns:\n A callable async function that executes the specified tool\n \"\"\"\n if not self.session:\n raise RuntimeError(\"Not connected to MCP server\")\n \n async def callable(*args, **kwargs):\n response = await self.session.call_tool(tool_name, arguments=kwargs)\n return response.content[0].text\n \n return callable\n \n \nasync def agent_loop(query: str, tools: dict, messages: List[dict] = None):\n \"\"\"\n Main interaction loop that processes user queries using the LLM and available tools.\n \n This function:\n 1. Sends the user query to the LLM with context about available tools\n 2. Processes the LLM's response, including any tool calls\n 3. Returns the final response to the user\n \n Args:\n query: User's input question or command\n tools: Dictionary of available database tools and their schemas\n messages: List of messages to pass to the LLM, defaults to None\n \"\"\"\n messages = (\n [\n {\n \"role\": \"system\",\n \"content\": SYSTEM_PROMPT.format(\n tools=\"\\n- \".join(\n [\n f\"{t['name']}: {t['schema']['function']['description']}\"\n for t in tools.values()\n ]\n )\n ), # Creates System prompt based on available MCP server tools\n },\n ]\n if messages is None\n else messages # reuse existing messages if provided\n )\n # add user query to the messages list\n messages.append({\"role\": \"user\", \"content\": query})\n \n # Query LLM with the system prompt, user query, and available tools\n first_response = await client.chat.completions.create(\n model=MODEL_ID,\n messages=messages,\n tools=([t[\"schema\"] for t in tools.values()] if len(tools) > 0 else None),\n max_tokens=4096,\n temperature=0,\n )\n # detect how the LLM call was completed:\n # tool_calls: if the LLM used a tool\n # stop: If the LLM generated a general response, e.g. \"Hello, how can I help you today?\"\n stop_reason = (\n \"tool_calls\"\n if first_response.choices[0].message.tool_calls is not None\n else first_response.choices[0].finish_reason\n )\n \n if stop_reason == \"tool_calls\":\n # Extract tool use details from response\n for tool_call in first_response.choices[0].message.tool_calls:\n arguments = (\n json.loads(tool_call.function.arguments)\n if isinstance(tool_call.function.arguments, str)\n else tool_call.function.arguments\n )\n # Call the tool with the arguments using our callable initialized in the tools dict\n tool_result = await tools[tool_call.function.name][\"callable\"](**arguments)\n # Add the tool result to the messages list\n messages.append(\n {\n \"role\": \"tool\",\n \"tool_call_id\": tool_call.id,\n \"name\": tool_call.function.name,\n \"content\": json.dumps(tool_result),\n }\n )\n \n # Query LLM with the user query and the tool results\n new_response = await client.chat.completions.create(\n model=MODEL_ID,\n messages=messages,\n )\n \n elif stop_reason == \"stop\":\n # If the LLM stopped on its own, use the first response\n new_response = first_response\n \n else:\n raise ValueError(f\"Unknown stop reason: {stop_reason}\")\n \n # Add the LLM response to the messages list\n messages.append(\n {\"role\": \"assistant\", \"content\": new_response.choices[0].message.content}\n )\n \n # Return the LLM response and messages\n return new_response.choices[0].message.content, messages\n \n \nasync def main():\n \"\"\"\n Main function that sets up the MCP server, initializes tools, and runs the interactive loop.\n The server is run in a Docker container to ensure isolation and consistency.\n \"\"\"\n # Configure Docker-based MCP server for SQLite\n server_params = StdioServerParameters(\n command=\"docker\",\n args=[\n \"run\",\n \"--rm\", # Remove container after exit\n \"-i\", # Interactive mode\n \"-v\", # Mount volume\n \"mcp-test:/mcp\", # Map local volume to container path\n \"mcp/sqlite\", # Use SQLite MCP image\n \"--db-path\",\n \"/mcp/test.db\", # Database file path inside container\n ],\n env=None,\n )\n \n # Start MCP client and create interactive session\n async with MCPClient(server_params) as mcp_client:\n # Get available database tools and prepare them for the LLM\n mcp_tools = await mcp_client.get_available_tools()\n # Convert MCP tools into a format the LLM can understand and use\n tools = {\n tool.name: {\n \"name\": tool.name,\n \"callable\": mcp_client.call_tool(\n tool.name\n ), # returns a callable function for the rpc call\n \"schema\": {\n \"type\": \"function\",\n \"function\": {\n \"name\": tool.name,\n \"description\": tool.description,\n \"parameters\": tool.inputSchema,\n },\n },\n }\n for tool in mcp_tools\n if tool.name\n != \"list_tables\" # Excludes list_tables tool as it has an incorrect schema\n }\n \n # Start interactive prompt loop for user queries\n messages = None\n while True:\n try:\n # Get user input and check for exit commands\n user_input = input(\"\\nEnter your prompt (or 'quit' to exit): \")\n if user_input.lower() in [\"quit\", \"exit\", \"q\"]:\n break\n \n # Process the prompt and run agent loop\n response, messages = await agent_loop(user_input, tools, messages)\n print(\"\\nResponse:\", response)\n # print(\"\\nMessages:\", messages)\n except KeyboardInterrupt:\n print(\"\\nExiting...\")\n break\n except Exception as e:\n print(f\"\\nError occurred: {e}\")\n \n \nif __name__ == \"__main__\":\n asyncio.run(main())\n```\n\nYou can run the agent and start chatting with\n\nPython\n\n```python\npython sqlite_llama_mcp_agent.py\n```\n\nBelow is an example conversation of what you can do and how it response.\n\n```plaintext\nEnter your prompt (or 'quit' to exit): HEllo \nResponse: Hello! How can 1 assist you today?\n \nEnter your prompt (or 'quit' to exit): Who are you?\nResponse: I am a helpful assistant capable of accessing external functions and engaging in casual chat. I can use the available tools to provide accurate and infor mative answers. The available tools include read_query,\nwrite_query, create_table, describe_table, and append_insight. I can guide you about these tools and their capabilities, and I will utilize them to access real-time information when required. \n \nEnter your prompt (or 'quit' to exit): Cool! Create a new table for fake products. Each product should have a name and a price. Generate 10 fake products based on video games.\nResponse: The table has been created successfully with the following schema:\n- name (TEXT): The name of the product.\n- price (REAL): The price of the product.\n \nHere are 10 fake products based on video games:\n1. \"The Last of Us\" T-shirt - $19.99\n2. Minecraft Creeper Plush Toy - $14.99\n3. Grand Theft Auto V Poster - $9,99\n4. The Legend of Zelda: Breath of the Wild Strategy Guide - $24.99\n5. Call of Duty: Modern Warfare Gaming Keyboard - $69.99\n6. World of Warcraft Subscription Card (3 months) - $39.99\n7. Assassin's Creed Odyssey Action Figure - $29.99\n8. Fortnite Monopoly Board Game - $29.99 \n9. Resident Evil 2 Remake Collector's Edition - $99.99\n10. Pokemon Sword and Shield Nintendo Switch Bundle - $399.99\n \nLet me know if you'd like to add more products or perform any other operations on the table!\n```\n\nFeel free to change the code, include debugging statements to better understand the outputs from each step.\n\n- [Google Gemini Example](https://github.com/philschmid/mcp-openai-gemini-llama-example/blob/master/sqlite_gemini_mcp_agent.py)\n\n## Conclusion\n\nThis simple example illustrates how you can get started with building an AI Agent using MCP Servers alongside open LLMs, OpenAI, or Google Gemini. It demonstrates the core principles of how an agent might connect to a server, interact with tools, and respond to user commands. It is not a framework or a reliable implementation—its purpose is to help you understand the building blocks of a distributed AI Agent architecture using MCP Server.\n\nI’m working on a small toolkit to make implementing AI Agents via MCP easier, so stay tuned for more updates.\n\nIf you have any questions, feedback, or ideas, please dm me on [X](https://x.com/_philschmid) or [LinkedIn](https://www.linkedin.com/in/philipp-schmid-a6a2bb196/). I am excited to hear about how you are experimenting and pushing the boundaries of AI agents.\n", "dateCreated": "2026-09-27T19:29:15.293681927+00:00", "date_created": "2025-01-17T00:00:00+00:00", "date_modified": "2025-01-17T00:00:00+00:00", "date_published": "2025-01-17T00:00:00+00:00", "description": "How to use Anthropic MCP Server with open LLMs, OpenAI or Google Gemini", "external_url": "https://www.philschmid.de/mcp-example-llama", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/philschmid-de/2025/01/2025-01-17-how-to-use-anthropic-mcp-server-with-open-llms-openai-or.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/philschmid-de/2025/01/2025-01-17-how-to-use-anthropic-mcp-server-with-open-llms-openai-or.md" }, "id": "urn:aggr:item:abf7293f710ccfb9fa7d9ebfcb5d0b62869fd607", "isBasedOn": { "@id": "https://www.philschmid.de/mcp-example-llama" }, "mainEntity": { "@id": "https://www.philschmid.de/mcp-example-llama", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-01-17-how-to-use-anthropic-mcp-server-with-open-llms-openai-or/#webpage" }, "dateModified": "2025-01-17T00:00:00+00:00", "datePublished": "2025-01-17T00:00:00+00:00", "headline": "How to use Anthropic MCP Server with open LLMs, OpenAI or Google Gemini", "timeRequired": "PT9M", "url": "https://www.philschmid.de/mcp-example-llama", "wordCount": 2015 }, "name": "Archived snapshot: How to use Anthropic MCP Server with open LLMs, OpenAI or Google Gemini", "preview": null, "reading_minutes": 9, "replicated_at": null, "source": "philschmid-de", "source_name": "philschmid.de - RSS feed", "summary": "How to use Anthropic MCP Server with open LLMs, OpenAI or Google Gemini", "tags": [], "temporalCoverage": "2026-09-27T19:29:15.293681927+00:00", "title": "How to use Anthropic MCP Server with open LLMs, OpenAI or Google Gemini", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2025-01-17-how-to-use-anthropic-mcp-server-with-open-llms-openai-or/", "word_count": 2015 }