open_llm_mcp_client_server_colab.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "machine_shape": "hm",
      "gpuType": "L4",
      "authorship_tag": "ABX9TyPYyRilsuRHYywS6aL1hGAo",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mrm8488/shared_colab_notebooks/blob/master/open_llm_mcp_client_server_colab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Create an `MPC` client and server with OS LLM\n",
        "\n",
        "In this tutorial we will create and **MCP server** that offers tools for weather forecasting and an **MCP client** that uses an Open Source LLM (`Qwen-2.5-7B-Instruct`) with **HuggingFace** `transformers`\n",
        "\n",
        "> Based on the Anthropic [example](https://modelcontextprotocol.io/introduction)\n",
        "\n",
        "> Created by [Manu Romero](http://twitter.com/mrm8488)"
      ],
      "metadata": {
        "id": "jz-QXqByFGaE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **1. Introduction to MCP**\n",
        "\n",
        "The Model Context Protocol (MCP) is an open standard developed by Anthropic that enables AI applications to interact seamlessly with external data sources and tools. By implementing MCP, AI models can access real-time information and perform tasks beyond their initial training data.\n",
        "\n",
        "---\n",
        "\n",
        "## **2. Setting Up the Environment**\n",
        "\n",
        "In this section, we'll set up the necessary environment by installing required libraries and configuring the runtime to support asynchronous operations.\n"
      ],
      "metadata": {
        "id": "2QuuEC96XDzX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! pip install -qU transformers bitsandbytes \"mcp[cli]\" httpx"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BzpO2c08Y3wN",
        "outputId": "8982552a-a8e6-4b4c-8e79-44b5eea20c9b"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m76.1/76.1 MB\u001b[0m \u001b[31m26.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m363.4/363.4 MB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.8/13.8 MB\u001b[0m \u001b[31m97.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m24.6/24.6 MB\u001b[0m \u001b[31m73.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m883.7/883.7 kB\u001b[0m \u001b[31m40.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m664.8/664.8 MB\u001b[0m \u001b[31m2.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.5/211.5 MB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.3/56.3 MB\u001b[0m \u001b[31m40.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m127.9/127.9 MB\u001b[0m \u001b[31m18.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m207.5/207.5 MB\u001b[0m \u001b[31m4.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.1/21.1 MB\u001b[0m \u001b[31m92.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will use an **L4** GPU, you can try with the free **T4** and maybe quantize the model to 4 bits (in this example we quantize it to 8 bits)"
      ],
      "metadata": {
        "id": "bLiktnJmX-nT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! nvidia-smi"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mxwTH7mFYnhU",
        "outputId": "eaa4a587-f4ee-43c1-aa4d-dff26d30deec"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sat Mar 22 20:54:17 2025       \n",
            "+-----------------------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |\n",
            "|-----------------------------------------+------------------------+----------------------+\n",
            "| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |\n",
            "|                                         |                        |               MIG M. |\n",
            "|=========================================+========================+======================|\n",
            "|   0  NVIDIA L4                      Off |   00000000:00:03.0 Off |                    0 |\n",
            "| N/A   76C    P0             35W /   72W |       0MiB /  23034MiB |      0%      Default |\n",
            "|                                         |                        |                  N/A |\n",
            "+-----------------------------------------+------------------------+----------------------+\n",
            "                                                                                         \n",
            "+-----------------------------------------------------------------------------------------+\n",
            "| Processes:                                                                              |\n",
            "|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |\n",
            "|        ID   ID                                                               Usage      |\n",
            "|=========================================================================================|\n",
            "|  No running processes found                                                             |\n",
            "+-----------------------------------------------------------------------------------------+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Apply nest_asyncio to allow nested use of asyncio.run in notebooks.\n",
        "import nest_asyncio\n",
        "nest_asyncio.apply()"
      ],
      "metadata": {
        "id": "zlf_Q-YHKR8Q"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **3. Implementing the MCP Server**\n",
        "\n",
        "We'll create an MCP server that provides tools to fetch weather alerts and forecasts. This server will interact with external weather APIs and expose functionalities via MCP tools.\n",
        "\n",
        "**Key Components:**\n",
        "- **make_nws_request:** A helper function to make requests to the National Weather Service (NWS) API.\n",
        "- **format_alert:** A function to format weather alert data.\n",
        "- **get_alerts:** An MCP tool to retrieve active weather alerts for a given state.\n",
        "- **get_forecast:** An MCP tool to fetch weather forecasts for a specific location based on latitude and longitude.\n"
      ],
      "metadata": {
        "id": "IVLh7tCAFuEl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile weather_mcp_server.py\n",
        "\n",
        "from typing import Any\n",
        "import httpx\n",
        "from mcp.server.fastmcp import FastMCP\n",
        "\n",
        "# Initialize FastMCP server\n",
        "mcp = FastMCP(\"weather\")\n",
        "\n",
        "# Constants\n",
        "NWS_API_BASE = \"https://api.weather.gov\"\n",
        "USER_AGENT = \"weather-app/1.0\"\n",
        "\n",
        "\n",
        "# Helper functions for formatting\n",
        "async def make_nws_request(url: str) -> dict[str, Any] | None:\n",
        "    \"\"\"Make a request to the NWS API with proper error handling.\"\"\"\n",
        "    headers = {\n",
        "        \"User-Agent\": USER_AGENT,\n",
        "        \"Accept\": \"application/geo+json\"\n",
        "    }\n",
        "    async with httpx.AsyncClient() as client:\n",
        "        try:\n",
        "            response = await client.get(url, headers=headers, timeout=30.0)\n",
        "            response.raise_for_status()\n",
        "            return response.json()\n",
        "        except Exception:\n",
        "            return None\n",
        "\n",
        "def format_alert(feature: dict) -> str:\n",
        "    \"\"\"Format an alert feature into a readable string.\"\"\"\n",
        "    props = feature[\"properties\"]\n",
        "    return f\"\"\"\n",
        "Event: {props.get('event', 'Unknown')}\n",
        "Area: {props.get('areaDesc', 'Unknown')}\n",
        "Severity: {props.get('severity', 'Unknown')}\n",
        "Description: {props.get('description', 'No description available')}\n",
        "Instructions: {props.get('instruction', 'No specific instructions provided')}\n",
        "\"\"\"\n",
        "\n",
        "# Implementing tool execution\n",
        "@mcp.tool()\n",
        "async def get_alerts(state: str) -> str:\n",
        "    \"\"\"Get weather alerts for a US state.\n",
        "\n",
        "    Args:\n",
        "        state: Two-letter US state code (e.g. CA, NY)\n",
        "    \"\"\"\n",
        "    url = f\"{NWS_API_BASE}/alerts/active/area/{state}\"\n",
        "    data = await make_nws_request(url)\n",
        "\n",
        "    if not data or \"features\" not in data:\n",
        "        return \"Unable to fetch alerts or no alerts found.\"\n",
        "\n",
        "    if not data[\"features\"]:\n",
        "        return \"No active alerts for this state.\"\n",
        "\n",
        "    alerts = [format_alert(feature) for feature in data[\"features\"]]\n",
        "    return \"\\n---\\n\".join(alerts)\n",
        "\n",
        "@mcp.tool()\n",
        "async def get_forecast(latitude: float, longitude: float) -> str:\n",
        "    \"\"\"Get weather forecast for a location.\n",
        "\n",
        "    Args:\n",
        "        latitude: Latitude of the location\n",
        "        longitude: Longitude of the location\n",
        "    \"\"\"\n",
        "    # First get the forecast grid endpoint\n",
        "    points_url = f\"{NWS_API_BASE}/points/{latitude},{longitude}\"\n",
        "    points_data = await make_nws_request(points_url)\n",
        "\n",
        "    if not points_data:\n",
        "        return \"Unable to fetch forecast data for this location.\"\n",
        "\n",
        "    # Get the forecast URL from the points response\n",
        "    forecast_url = points_data[\"properties\"][\"forecast\"]\n",
        "    forecast_data = await make_nws_request(forecast_url)\n",
        "\n",
        "    if not forecast_data:\n",
        "        return \"Unable to fetch detailed forecast.\"\n",
        "\n",
        "    # Format the periods into a readable forecast\n",
        "    periods = forecast_data[\"properties\"][\"periods\"]\n",
        "    forecasts = []\n",
        "    for period in periods[:5]:  # Only show next 5 periods\n",
        "        forecast = f\"\"\"\n",
        "{period['name']}:\n",
        "Temperature: {period['temperature']}°{period['temperatureUnit']}\n",
        "Wind: {period['windSpeed']} {period['windDirection']}\n",
        "Forecast: {period['detailedForecast']}\n",
        "\"\"\"\n",
        "        forecasts.append(forecast)\n",
        "\n",
        "    return \"\\n---\\n\".join(forecasts)\n",
        "\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "  # Initialize and run the server\n",
        "  mcp.run(transport='stdio')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SLWkOu2rFPN2",
        "outputId": "46cd136b-d2a1-42ff-94d0-cf029698d15f"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting weather_mcp_server.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **4. Implementing the MCP Client**\n",
        "\n",
        "We'll develop a client that connects to the MCP server and utilizes the tools provided. This client will use Anthropic's API to interact with the model and process user queries.\n",
        "\n",
        "**Key Components:**\n",
        "- **MCPClient Class:** Manages the connection to the MCP server, lists available tools, and facilitates interaction between the user and the server.\n",
        "- **connect_to_server:** Establishes a connection to the specified MCP server.\n",
        "- **list_tools:** Retrieves and displays the tools available on the connected server.\n",
        "- **chat_loop:** Handles the interactive loop where the user inputs queries, and the client processes them using the model and server tools.\n",
        "- **cleanup:** Ensures proper closure of the client session.\n"
      ],
      "metadata": {
        "id": "g8QMf9iRHK53"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile mcp_client.py\n",
        "\n",
        "import re\n",
        "import json\n",
        "import asyncio\n",
        "from typing import Optional\n",
        "from contextlib import AsyncExitStack\n",
        "import warnings\n",
        "\n",
        "from mcp import ClientSession, StdioServerParameters\n",
        "from mcp.client.stdio import stdio_client\n",
        "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
        "\n",
        "warnings.filterwarnings(\"ignore\")\n",
        "\n",
        "# Use there the model you wish (it must support tool calling)\n",
        "MODEL_ID = \"Qwen/Qwen2.5-7B-Instruct\"\n",
        "\n",
        "\n",
        "# Load the model and tokenizer (quantized to 8bit)\n",
        "tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)\n",
        "model = AutoModelForCausalLM.from_pretrained(\n",
        "    MODEL_ID,\n",
        "    torch_dtype=\"auto\",\n",
        "    device_map=\"auto\",\n",
        "    load_in_8bit=True,\n",
        ")\n",
        "\n",
        "# Helper function becaus in transformers the tool calls should be a field of assistant messages.\n",
        "def try_parse_tool_calls(content: str):\n",
        "    \"\"\"Try parse the tool calls.\"\"\"\n",
        "    tool_calls = []\n",
        "    offset = 0\n",
        "    for i, m in enumerate(re.finditer(r\"<tool_call>\\n(.+)?\\n</tool_call>\", content)):\n",
        "        if i == 0:\n",
        "            offset = m.start()\n",
        "        try:\n",
        "            func = json.loads(m.group(1))\n",
        "            tool_calls.append({\"type\": \"function\", \"function\": func})\n",
        "            if isinstance(func[\"arguments\"], str):\n",
        "                func[\"arguments\"] = json.loads(func[\"arguments\"])\n",
        "        except json.JSONDecodeError as e:\n",
        "            print(f\"Failed to parse tool calls: the content is {m.group(1)} and {e}\")\n",
        "            pass\n",
        "    if tool_calls:\n",
        "        if offset > 0 and content[:offset].strip():\n",
        "            c = content[:offset]\n",
        "        else:\n",
        "            c = \"\"\n",
        "        return {\"role\": \"assistant\", \"content\": c, \"tool_calls\": tool_calls}\n",
        "    return {\"role\": \"assistant\", \"content\": re.sub(r\"<\\|im_end\\|>$\", \"\", content)}\n",
        "\n",
        "\n",
        "class MCPClient:\n",
        "    def __init__(self):\n",
        "        # Initialize session and client objects\n",
        "        self.session: Optional[ClientSession] = None\n",
        "        self.exit_stack = AsyncExitStack()\n",
        "        self.llm = model\n",
        "        self.tokenizer = tokenizer\n",
        "\n",
        "    async def connect_to_server(self, server_script_path: str):\n",
        "        \"\"\"Connect to an MCP server\n",
        "\n",
        "        Args:\n",
        "            server_script_path: Path to the server script (.py or .js)\n",
        "        \"\"\"\n",
        "        is_python = server_script_path.endswith('.py')\n",
        "        is_js = server_script_path.endswith('.js')\n",
        "        if not (is_python or is_js):\n",
        "            raise ValueError(\"Server script must be a .py or .js file\")\n",
        "\n",
        "        command = \"python\" if is_python else \"node\"\n",
        "        server_params = StdioServerParameters(\n",
        "            command=command,\n",
        "            args=[server_script_path],\n",
        "            env=None\n",
        "        )\n",
        "\n",
        "        stdio_transport = await self.exit_stack.enter_async_context(stdio_client(server_params))\n",
        "        self.stdio, self.write = stdio_transport\n",
        "        self.session = await self.exit_stack.enter_async_context(ClientSession(self.stdio, self.write))\n",
        "\n",
        "        await self.session.initialize()\n",
        "\n",
        "        # List available tools\n",
        "        response = await self.session.list_tools()\n",
        "        tools = response.tools\n",
        "        print(\"\\nConnected to server with tools:\", [tool.name for tool in tools])\n",
        "\n",
        "    async def process_query(self, query: str) -> str:\n",
        "      \"\"\"Process a query using Claude and available tools\"\"\"\n",
        "      messages = [\n",
        "          {\n",
        "              \"role\": \"user\",\n",
        "              \"content\": query\n",
        "          }\n",
        "      ]\n",
        "\n",
        "      response = await self.session.list_tools()\n",
        "      available_tools = [{\n",
        "          \"name\": tool.name,\n",
        "          \"description\": tool.description,\n",
        "          \"input_schema\": tool.inputSchema\n",
        "      } for tool in response.tools]\n",
        "\n",
        "\n",
        "      # Initial LLM Call\n",
        "      text = self.tokenizer.apply_chat_template(messages, tools=available_tools, add_generation_prompt=True, tokenize=False)\n",
        "      inputs = self.tokenizer(text, return_tensors=\"pt\").to(model.device)\n",
        "      outputs = self.llm.generate(**inputs, max_new_tokens=512)\n",
        "      output_text = tokenizer.batch_decode(outputs)[0][len(text):]\n",
        "\n",
        "\n",
        "      # Processing response and handel tool calls with the LLM `output_text`\n",
        "      final_text = []\n",
        "\n",
        "      parsed_message = try_parse_tool_calls(output_text)\n",
        "      messages.append(parsed_message)\n",
        "\n",
        "      final_text.append(parsed_message[\"content\"])\n",
        "\n",
        "      if tool_calls := messages[-1].get(\"tool_calls\", None):\n",
        "          for tool_call in tool_calls:\n",
        "              if fn_call := tool_call.get(\"function\"):\n",
        "                  fn_name: str = fn_call[\"name\"]\n",
        "                  fn_args: dict = fn_call[\"arguments\"]\n",
        "\n",
        "                  print(f\"Calling tool: {fn_name} with args: {fn_args}\")\n",
        "                  final_text.append(f\"Calling tool: {fn_name} with args: {fn_args}\")\n",
        "                  result = await self.session.call_tool(fn_name, fn_args)\n",
        "                  #print(result)\n",
        "                  fn_res = result.content\n",
        "                  #print(f\"Tool result: {fn_res}\")\n",
        "\n",
        "                  messages.append({\n",
        "                      \"role\": \"tool\",\n",
        "                      \"name\": fn_name,\n",
        "                      \"content\": fn_res,\n",
        "                  })\n",
        "\n",
        "              # Get next response from Claude\n",
        "              text = self.tokenizer.apply_chat_template(messages, tools=available_tools, add_generation_prompt=True, tokenize=False)\n",
        "              inputs = self.tokenizer(text, return_tensors=\"pt\").to(model.device)\n",
        "              outputs = self.llm.generate(**inputs, max_new_tokens=512)\n",
        "              output_text = self.tokenizer.batch_decode(outputs)[0][len(text):]\n",
        "\n",
        "              final_text.append(output_text)\n",
        "\n",
        "\n",
        "\n",
        "      return \"\\n\".join(final_text)\n",
        "\n",
        "    async def chat_loop(self):\n",
        "      \"\"\"Run an interactive chat loop\"\"\"\n",
        "      print(\"\\nMCP Client Started!\")\n",
        "      print(\"Type your queries or 'quit' to exit.\")\n",
        "\n",
        "      while True:\n",
        "          try:\n",
        "              query = input(\"\\nQuery: \").strip()\n",
        "\n",
        "              if query.lower() == 'quit':\n",
        "                  break\n",
        "\n",
        "              response = await self.process_query(query)\n",
        "              print(\"\\n\" + response)\n",
        "\n",
        "          except Exception as e:\n",
        "              print(f\"\\nError: {str(e)}\")\n",
        "\n",
        "    async def cleanup(self):\n",
        "      \"\"\"Clean up resources\"\"\"\n",
        "      await self.exit_stack.aclose()\n",
        "\n",
        "\n",
        "async def main():\n",
        "  if len(sys.argv) < 2:\n",
        "      print(\"Usage: python client.py <path_to_server_script>\")\n",
        "      sys.exit(1)\n",
        "\n",
        "  client = MCPClient()\n",
        "  try:\n",
        "      await client.connect_to_server(sys.argv[1])\n",
        "      await client.chat_loop()\n",
        "  finally:\n",
        "      await client.cleanup()\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "    import sys\n",
        "    asyncio.run(main())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QHWyo3FaHwHg",
        "outputId": "bce1eabb-b6c4-42a8-e04a-22a39dc299ac"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting mcp_client.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Runing it"
      ],
      "metadata": {
        "id": "B2wQiNr5JG44"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! python mcp_client.py weather_mcp_server.py"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LFfjvqpHJRe7",
        "outputId": "ef9c5624-72a4-4c0f-a776-bf85123ae632"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2025-03-22 21:26:59.376107: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n",
            "2025-03-22 21:26:59.396282: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
            "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
            "E0000 00:00:1742678819.416234   27153 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "E0000 00:00:1742678819.421799   27153 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "2025-03-22 21:26:59.440865: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
            "To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
            "The `load_in_4bit` and `load_in_8bit` arguments are deprecated and will be removed in the future versions. Please, pass a `BitsAndBytesConfig` object in `quantization_config` argument instead.\n",
            "Sliding Window Attention is enabled but not implemented for `sdpa`; unexpected results may be encountered.\n",
            "Loading checkpoint shards: 100% 4/4 [00:07<00:00,  1.78s/it]\n",
            "\n",
            "Connected to server with tools: ['get_alerts', 'get_forecast']\n",
            "\n",
            "MCP Client Started!\n",
            "Type your queries or 'quit' to exit.\n",
            "\n",
            "Query: weather in SF, CA\n",
            "/usr/local/lib/python3.11/dist-packages/bitsandbytes/autograd/_functions.py:315: UserWarning: MatMul8bitLt: inputs will be cast from torch.bfloat16 to float16 during quantization\n",
            "  warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
            "Calling tool: get_forecast with args: {'latitude': 37.7749, 'longitude': -122.4194}\n",
            "\n",
            "To provide you with the weather forecast for San Francisco, CA, I will use the coordinates for San Francisco. The latitude and longitude for San Francisco are approximately 37.7749 and -122.4194, respectively.\n",
            "\n",
            "I will now fetch the weather forecast using these coordinates.\n",
            "\n",
            "Calling tool: get_forecast with args: {'latitude': 37.7749, 'longitude': -122.4194}\n",
            "Here's the weather forecast for San Francisco, CA:\n",
            "\n",
            "- **This Afternoon:** \n",
            "  - Temperature: 62°F\n",
            "  - Wind: 10 mph from the west\n",
            "  - Forecast: A slight chance of rain before 5pm. Mostly sunny, with a high near 62. West wind around 10 mph. Chance of precipitation is 20%.\n",
            "\n",
            "- **Tonight:**\n",
            "  - Temperature: 49°F\n",
            "  - Wind: 3 to 10 mph from the west\n",
            "  - Forecast: Patchy fog after 2am. Mostly clear, with a low around 49. West wind 3 to 10 mph.\n",
            "\n",
            "- **Sunday:**\n",
            "  - Temperature: 68°F during the day, falling to around 66 in the afternoon\n",
            "  - Wind: 2 to 7 mph from the southwest\n",
            "  - Forecast: Areas of fog before 9am. Mostly sunny. High near 68.\n",
            "\n",
            "- **Sunday Night:**\n",
            "  - Temperature: 52°F\n",
            "  - Wind: 2 to 7 mph from the southwest\n",
            "  - Forecast: Mostly clear, with a low around 52.\n",
            "\n",
            "- **Monday:**\n",
            "  - Temperature: 78°F\n",
            "  - Wind: 2 to 6 mph from the north-northeast\n",
            "  - Forecast: Sunny, with a high near 78.\n",
            "\n",
            "Stay prepared for some changes in temperature and a slight chance of rain today!<|im_end|>\n",
            "\n",
            "Query: any alert in CA?\n",
            "Calling tool: get_alerts with args: {'state': 'CA'}\n",
            "\n",
            "\n",
            "Calling tool: get_alerts with args: {'state': 'CA'}\n",
            "There are a couple of alerts currently active in California:\n",
            "\n",
            "1. **Beach Hazards Statement**:\n",
            "   - **Area**: Coastal Del Norte, Northern Humboldt Coast, Southwestern Humboldt, and Mendocino Coast.\n",
            "   - **Severity**: Moderate.\n",
            "   - **Description**: Increased threat of sneaker waves is expected. Large, unexpected waves can sweep across the beach without warning, posing a risk to people near the water. Instructions: Don't be fooled by an ocean that looks calm. There can be 30 minutes of small waves before a sneaker wave strikes. Avoid rocks and jetties, stay far back from the water, and never turn your back on the ocean.\n",
            "\n",
            "2. **Wind Advisory**:\n",
            "   - **Area**: Santa Barbara County Southwestern Coast and Santa Ynez Mountains Western Range.\n",
            "   - **Severity**: Moderate.\n",
            "   - **Description**: North winds are expected to be between 20 to 30 mph with gusts up to 50 mph. Winds will diminish in strength this morning but will increase again late this afternoon and evening. Instructions: Strong winds can make driving difficult, especially for high-profile vehicles. Use extra caution.\n",
            "\n",
            "Please take necessary precautions based on these alerts.<|im_end|>\n",
            "\n",
            "Query: what is 1+1?\n",
            "\n",
            "1 + 1 equals 2. If you have any other questions or need further assistance, feel free to ask!\n",
            "\n",
            "Query: quit\n"
          ]
        }
      ]
    }
  ]
}