If you don't run your own LLM proxy, how do you know whats going on?

voodooEntity

Well that was one of the reasons why i build my own agent/harness - to have actual full observability of what it is doing and why it is doing what it is doing. Don't really understand why other agents to give those insights but well found my solution.

hoppp

Yup. Setting up liteLLMProxy on some servers as we speak and will start playing around. I can't orchestrate agents in sandboxes without it. 1. It can issue virtual keys, so provider API keys never go near the sandbox 2. I can see everything 3. Guardrails And much more Looking forward to it.

stavros

Does anyone here use LiteLLM for this? How well does it do? Can I use it with my Claude subscription?

impure

OpenRouter does all this for you. It's pretty nice. Although for apps I use another server to lock down the requests even more and prevent the client from seeing my API key.

soared

One man’s fun is another man’s “idk what any of these words mean, I pay $20/mo for some company to abstract this complexity away” For clarity, I do like blog posts like these.

sroerick

Seeing some good stuff in the comments, but how are people doing this?

jubilanti

Yeah you should run an LLM proxy, but imagine a world where LiteLLM and all the other LLM shovels weren't buggy vibecoded messes. And with the velocity of updates I feel there has to be vulnerabilities there. Or am I just holding it wrong? Anyone got a good alternative?

NeutralWanted

I'd love to do something like this, or use API keys, but I still don't understand how people justify the cost of using API keys vs. subscription. I tested it once, I'm in on the 20x claude max ($200/month) and I either max my weekly limit every week or come really close. But it only took 2-3 days to hit $200 spend with api keys.

variety8675

I’m not sure what people are doing with these subscriptions to feel the need to monitor what’s going on this closely. A $100 OpenAI subscription feels effectively infinite for my usage at home

ok_major_9889

Why not just grab it from the kernel?

kennywinker

> Own your infrastructure ...proceeds to use codex. With just a 12gb VRAM GPU + 64gb of CPU RAM, and you can actually own your own infrastructure. I can't wait for hardware prices to fall so that becomes even more accessible.

passive

Tailscale makes Aperture available for free, and while it's not something you own, it is in your tailnet, so at least you have a certain amount of control over how things access it. It gives a nice view of what agents are doing, with lots of configurability and extensions, and the team behind it has been updating pretty steadily. I happen to be working on an extension to Aperture, so I have my own reason to promote it. But I do think it's a nice easy solution, and I promoted it to people before I was extending it.

chopete3

This is the best one I found https://github.com/kenn-io/agentsview It is a not proxy. It can clearly show whats going on across all agents. The proxy's are generally inconvenient and yet another point of failure. I think the author got to the resort option quickly

garou

https://github.com/gumieri/nenya my own ai gateway/proxy

joshheitzman

By running my own agent harness that provides detailed API metrics in the GUI and logs very verbosely.

left-struck

I like the content but I have to say the Mark Manson writing style is off putting.

quixoticelixer-

What is the use case for these sorts of things?

roosterIllusi0n

I am boggled how the first step of an llm after you make an orchestrator isn't some kind of proxy that records your personal internet surfing to be used by your personal AI later. Every college should be running in house AI with their own in house search engines and in house any service. Don't go to any college that wants you to use chatgpt or other corporate nonsense. If they can't make local LLMs work, they are not a real college.

jsphweid

I don't do this but I do have a system that imports all my conversations with AI (claude code, chatgpt, claude app, etc.) to a storage system I call Chatacombs: a unified storage of most conversations I've had with AI, as detailed as I can easily get.

jakozaur

Proxies are good if you pay API prices. For subscriptions, it is easier to collect the JSONL files from ~/.claude/projects/ and similar directories. They are better organized (e.g., subagents) than what the proxy sees. However, the best value comes from collecting agent sessions across the whole organization. You'll need to scrub credentials and handle security.