![\"\"](\"https://steipete.me/assets/img/2025/self-hosting-ai-models/hero.png\")
When Anthropic changed Claude Max’s subscription model from a 5-hour usage window to weekly limits, it hit me hard.
\nTL;DR: Claude Code is still king, Qwen 3 Coder + opencode is a strong contender tho. Have high hopes for Gemini cli and Crush.
\nI’m definitely part of the 1% that has got a ton of benefit out of their payment model, and I’m not entirely surprised that they’re changing the deal, given the incredible demand for Claude Code. Much like Cursor’s recent pricing changes, this caused a lot of anger in the community, and it also triggered my curiosity on what other options are out there.
\nJust to set the record straight tho: While I often worked 16h-days, I never automated Claude Code and was well within their Terms of Service. My Anthropic bill for July is at ~6000$, since I’ve been using their GitHub Review bot, and while some people have been extracting the Max API Token to run on GitHub in the background, I opted to not break the rules and pay up… so Anthropic, we cool?
\nSo what are the alternatives? I spend some time testing every promising tool I could find. Claude Code is insofar hard to replace, as it’s this genius blend of amazing model & tooling. But the landscape is changing fast! There’s quite a few contenders out there that are worth a look:
\nopencode In my tests, opencode is the most promising alternative currently. It supports pretty much all providers and gets better every day. It has an optimized prompt for Qwen 3 Coder, and produces good results. I’ll definitely be using this more.
\ncharm crush I really want to love this one. It’s gorgeous and fun to use. No surprise, it’s backed by a company that makes fun cli tools and that shows. Crush was released just yesterday and it’s still early. It has problems with the edit tool and sometimes just gets stuck, at least when using it with Qwen 3 Coder. I had to send a PR to make it work with self hosted inference. There’s a whole team working on it so my hopes are up. If you use Anthropic’s models, I’m sure it performs much better. Personal deal-breaker: The non-working text selection. You can’t click on links either. I use this a lot so it’d really disruptive for my workflow.
\nClaude Code with claude-code-router Clever hack that patches Claude Code to use different models. Since CC is optimized for Anthropic’s models, tool use and overall results aren’t as good as with Sonnet or Opus. I didn’t explore this further.
\nCline I used Cline for testing the model, but since it’s a VS Code extension and not a cli, it doesn’t fit my workflow, and generally I haven’t been impressed by it.
\namp Amp is a very opinionated CLI. It uses Claude Sonnet and has a tool that can invoke OpenAI’s o3. Pricing is forwarded without markup. You can’t use your subscription here, so until now, it wasn’t interesting. It’s hyped, in my tests it didn’t impress me tho. Some people swear by it tho, and it’s certaily more token efficient than Claude Code.
\nGemini CLI / Qwen Code I have high hopes for Gemini CLI, and with it, the Qwen Code fork. Gemini is REALLY FAST. I use it to debug tricky issues that need a large context space. Tool calling is still very broken “I’m just a file” and it can be really lazy, however it’s open source and a lot of folks are working on making it better. As for the Quen Code fork, we’ll see if they keep up with the fast-paced development of Gemini or abandon it. I worry it’ll be the latter, at least from looking at the current pace of updates.
\nThere are so many more tools out there in all colors and sizes, and it’s really impressive how important tooling & the right system prompt are (e.g. RooCode drawing diagrams!). IMO that’s Claude Code’s strength: it’s optimized for just one model, and it shows.
\nMy main goals for this exploration are twofold: I don’t wanna be too dependent on one company & tool (Anthropic), and I want to have a less costy solution for my heavy use.
\nA few months ago I bought a Mac Studio with 512GB ram, top of the line, to experiment with models (and because it’s prettyyh). Turns out, back then there just wasn’t anything great out there that could compete with Opus & Gemini. And then came Deepseek, and with that the realization that even the 512GB won’t get me very far.
\nYes, I can run Deepseek Coder V2 at ~25 tok/s or a quantified version of R1 at ~8–15 tok/s, but that’s not fun, and 128k context size is quite a bit of a downgrade compared to Claude’s 200k or Gemini’s 1Mio context. There’s ways to connect multiple Studio’s for more performance and/or less compressed models, but then we’re talking 30k$+ for the setup.
\nHardware is one thing - we also need capable models. These days, chinese labs releases better and more capable models on an almost weekly basis. The release of Qwen3-Coder-480B got my attention, as it’s the first model that achieves a similar score on SWE-bench as Claude Sonnet 4. Plus, it has a native 256k token context window that can stretch to 1M tokens with YaRN.
\nRunning this needs some beefy hardware. NVidia’s H200 (Hopper refresh) is currently the best top-of-the-line choice. There’s also B200 (Blackwell), but they are usually even more expensive to rent, and software support is still quite weak.
\nI’ve been experimenting with Prime Intellect, Vast.ai and DataCrunch. Prices for 8xH200 machines range from $5-30/hour, depending on location, availability and if you rent a spot instance or a regular one. Now spot instances are instances that are available temporarily. Whenever data centers need more capacity, they will simply deprovision it and use it again.
\nIn my tests, if you get a H200 spot instance it’ll likely disappear before you are finished setting it up. They can be incredibly cheap, but it aint fun when you suddenly have to stop working and set up a new rack because of peak demand. We’re talking $6/h for spot instances vs $14/h for a regular 8xH200 rack, at least on DataCrunch, which seems to be one of the cheapest (while still reliable in my tests) options out there. On vast.ai, a similar setup is more like ~$26/h, and double that again if you look at AWS.
\nThe setup I’m currently running:
\n\n\n\n\nWith my rig I could set up the FP81-version of Qwen3 Coder 480B with a 400k context. FP8 means smaller float’s, so you lose a few percent accuracy compared to the FP16-version, which is used for the benchmarks. For the Qwen model these losses are insignificant though, and it’s a big performance win, plus squeezing in the FP16 model would leave us with maybe 32k tokens of context.
\nMemory is absolutely everything at this scale. The KV cache2 (that’s what stores the conversation context) eats about 4.2MB per token when using FP16 precision. If you want a 1M token context window, you’d need 17-30 H200 GPUs. That’s why I “only” got 400k tokens with 8 GPUs.
\nSetup is tricky, but since Claude Code is my computer, it was mostly a bit of prompting and about half an hour waiting until the rig was ready - at least for H200.
\nCurrently sparse instances of B200 are an incredibly good deal (~$4/h) and they also been extremely stable in my tests. I also understand why: this hardware is so new that you’ll unlikely succeed running models efficiently on it. I spent all day on this and got it partially running, but at no point was it faster than a 8xH200 rig. That will change in a few weeks tho as software catches up.
\nInsane if you think about it: The 3 8xB200 rack hardware I rented here is worth about 2 Mio Dollar.
\nWith the technical setup sorted, let’s get to the question everyone’s really asking about: Money.
\nRunning this setup costs $15/h per hour. That’s ~$360 per day if you run it 24/7, or about $11,000/month. You could recreate the machine every day to save money, but setting this up also takes 30-60 minutes each morning. That would drive cost down to ~$120 a day and ~$2,600 a month (if we just count work days, but who doesn’t work on the weekends?)
\nThose numbers probably made you wince. So here’s the honest assessment after burning through all that cash:
\nSimple answer: No. You can’t stop and easily restart instances, at least not with the hardware provider I tested. Restarted instances will be scheduled, but there’s no guarantee you get a spot in that data center, so it might take days or weeks until your setup reboots.
\nWhat about performance? I naively thought that I’d get better performance with my personal 8xH200 rig, so I ran some tests. And since I couldn’t find a simple benchmark tool, I vibed one in go. Turns out, Alibaba is slightly faster than my best optimized 8xH200 rig. Maybe that changes once someone gets the B200 racks working.
\nIf you want REAL performance and also a damn good deal, check out Cerebra’s new offering that came out today. Cerebras Code Max, 5000 messages per day for $200/month. About 20x faster than Claude 4 Sonnet.
\nYou can rent the 1 Mio-context version of Qwen 3 Coder on Alibaba for $1-$6 In / $5-$60 Out per Million tokens, depending how much context you’re using. That’s a lot of tokens you have to burn to make self-hosting cost effective. And yes, a self-hosted runner could be shared, but the economics are still against you, and you quickly run into bottlenecks if more than one person accesses your server concurrently.
\nCompare to Anthropic: $15 / $75 for Opus, $3 / $15 for Sonnet
\nGoogle’s Gemini 2.5 Pro: $1.25-$2.50 / $10-$15
\nTo give you an idea, on an intense day I need maybe 500 Mio tokens. Calculating the price is difficult tho, since many tokens are cached as you work through a session. The cost for that, when using Opus, would be about $1000; for Sonnet about $200 - that’s similar to running my rig for 8h. Using other models is significantly cheaper.
\nRealistically, using Qwen 3 Coder with opencode is now a very usable and cost effective alternative today. And in a month when the subscription nerving takes effect, things will look different again.
\nIt’s great to know that open-source models are a merely 6-12 month behind the best commercial ones, and that it’s absolutely possible to host them yourself. Commercially tho, paying per token is the economically saner choice.
\nVerdict: Claude Code as terminal + coding, opencode + Qwen on Alibaba (or Cerebas for speeeeeed!) as alternative coding agent for simpler tasks (or when the subscription is used up) and Gemini for debugging.
\n🤖 Setup Guide for AI Agents to Self-Hosting Qwen3-Coder-480B on 8xH200
\nThis guide provides step-by-step instructions for AI agents to set up Qwen3-Coder-480B on Vast.ai with maximum context window.
\nGo to Vast.ai
\nSearch for instances with:
\nSelect instance and configure:
\nDocker Image: Leave default vLLM image\nJupyter: Optional (uses port 8080)\nSSH: Required\nDirect SSH: Recommended\nDeploy instance and note:
\nAdd to ~/.ssh/config:
\n \n \n \n \n \nCreate SSH tunnel:
\n\nSSH into instance:
\n\nCheck GPU status:
\n\nCreate deployment script:
\n\n\n\n\nModel download takes 1-2 hours for 480GB:
\n# Watch download progress\n | \n\n# Check disk usage\n\nVast.ai uses Caddy proxy with auth. Disable it:
\n# Stop Caddy to remove authentication\n\n\n# Verify direct access works\n\nInstall Cline extension in VS Code
\nOpen Cline settings
\nConfigure:
\nAPI Provider: OpenAI Compatible\nBase URL: http://localhost:8000/v1\nAPI Key: not-needed\nModel: qwen3-coder\nOpen Cursor settings
\nAdd custom model:
\n\nCreate config at ~/.config/qwen/config.json:
\nTest with curl:
\n\nTest context window:
\n# Create large context test\n\nMonitor GPU utilization:
\n# Real-time GPU monitoring\n\n\n# Check vLLM metrics\n\nOptimize for your use case:
\nCommon issues and solutions:
\n# Check available memory\n\n# Solution: Reduce --gpu-memory-utilization to 0.90\n# Ensure Caddy is stopped\n\n\n# Reduce max_model_len in deployment script\n# 4x H200: max 190000\n# 8x H200: max 400000\n# Check batch settings\n# Reduce --max-num-batched-tokens to 16384\n# Enable streaming in client\nTrack usage and costs:
\n# Create usage tracker\n\n\n\n\nFor different context windows, adjust these parameters:
\n\n\n\n\nRegular maintenance tasks:
\n# Check logs for errors\n | \n\n# Monitor disk space\n\n\n# Restart vLLM if needed\n\n\n\n# Clean old model files\n\nThis completes the detailed setup guide for AI agents. The entire process should take 2-3 hours including model download time.
\nFP8/FP16 - Floating-point precision formats. FP16 uses 16 bits per number (higher accuracy), FP8 uses 8 bits (lower memory usage, slightly less accurate). ↩
\nKV Cache - Key-Value cache stores the model’s “memory” of previous tokens in the conversation, enabling efficient context handling without recomputing everything. ↩
\n