The Agent Deployment Co.
toolsmcplitellm

LiteLLM Quickstart Notes

A mostly polished experience, about as good as could be

Mando Escamilla·

LiteLLM is a thing that is a couple of things:

  1. a python library that gives your code a single interface to like 100+ different provider models so you don’t have to write that code yourself (or really, so you don’t have to review the code that Claude writes)
  2. an AI gateway/proxy/router, um, thing (this thing does a bunch of things too, which seems to be a theme)

These notes are specifically about the gateway and more specifically about getting it running locally for testing, evaluation, etc. I’m not setting this up in production yet.

The quickstart

The quickstart asks you to curl and pipe to docker-compose:

curl -sSL https://docs.litellm.ai/docker-compose.yml | docker compose -f - up -d

which, I mean, fine — not to brag, but I read yaml at a high-school level. For those that don’t, there’s some nice comments in the yaml that aren’t lies:

# LiteLLM quickstart stack: the gateway plus a Postgres database that stores
# models, virtual keys, and spend logs. Used by https://docs.litellm.ai/docs/proxy/docker_quick_start
#
# Start it directly:
#   curl -sSL https://docs.litellm.ai/docker-compose.yml | docker compose -f - up -d
#
# The credentials below are placeholders for local evaluation. Before any real
# use, download this file and set LITELLM_MASTER_KEY and LITELLM_SALT_KEY to
# long random values, and pin the image to a release tag instead of `latest`.

One of the classic Docker blunders

So the first thing I wanted to do was try and use LiteLLM to proxy to Argus’ MCP server. No super real reason other than I know this MCP and can test it pretty easily. First pass: no dice :(

LiteLLM's Add New MCP Server form. The MCP Server URL is http://localhost:4242/mcp and the Connection Status card reads: Connection Failed — failed to connect to MCP server: the server is unreachable.
Connection failed. Note the URL. (Click for full size.)

Of course, I fell victim to one of the classic Docker blunders. The most famous of which is ‘never run docker inside of docker’ but only slightly well-known is this: ‘never try to connect to localhost when docker is involved’ (aka, ‘you keep using that word, i do not think it means what you think it means’).

localhost to my browser, is, of course, not localhost to litellm running inside of docker 🤦🏽‍♂️ (see the Docker Desktop networking docs for slightly more info).

A quick config change and voila! Connected!

The same LiteLLM form, now with the MCP Server URL set to http://host.docker.internal:4242/mcp, a Bearer Token auth value, a Connected status, and a Tool Configuration panel listing six tools sorted into Read, Update, and Other risk groups.
host.docker.internal instead of localhost, and we’re connected. (Click for full size.)

The auto-classification of the tools is interesting, if slightly incorrect (health_summary only shows up once in the interface — my dumb screenshot tool grabbed it twice when it scrolled).

Cool note here: you can toggle individual tools off/on here if you want.

Now to test this out

First things first:

cp ~/.claude.json ~/.claude.json.backup  # or whatever harness you're using

Then add the mcpServers stanza:

"mcpServers": {
  "LiteLLM": {
    "url": "http://localhost:4000/mcp",
    "type": "http",
    "headers": {
      "x-litellm-api-key": "Bearer sk-1234"
    }
  }
}

More notes:

I couldn’t find in the docs where the api key header is the same thing as the LITELLM_MASTER_KEY env var (for the quickstart it’s defined in the docker-compose.yaml, defaults to sk-1234 plz don’t hack me). Also, the docs leave out type: http but at least claude won’t work without it.

Once you’ve got that in there, boom!

Claude Code's Manage MCP servers screen listing LiteLLM as connected with 6 tools.
Claude Code sees the gateway: connected, 6 tools.
The LiteLLM MCP Server detail screen in Claude Code: status connected, authenticated, URL http://localhost:4000/mcp, capabilities tools, 6 tools.
Connected and authenticated against localhost:4000/mcp.
The tools list for the LiteLLM server in Claude Code, showing six argus-prefixed tools: search_sessions, get_session, get_session_transcript, usage_summary, and tool_usage.
Argus’ tools, proxied through the gateway and namespaced argus-*.

Relatively easy-peasy!

Ready to deploy your own agents?

Book a discovery call and let's talk through your use case.

Book a Discovery Call