I’ve been working on a project to solve two big problems in the LLM space: unpredictable costs and data privacy.

Unlike passive monitoring tools, this acts as a proxy/interceptor. When an event is ingested, it checks the cumulative cost against a user-defined threshold. If the threshold is breached, it flags the key for immediate suspension.

Key Architecture Decisions:

Local-first: Designed to run as a sidecar or a local gateway to minimize latency. Regex-based PII stripping: Before the request reaches the provider, we strip sensitive tokens. Recharts Dashboard: High-granularity cost tracking per feature/model. I'd love to hear the community's thoughts on the latency overhead of adding an interceptor layer like this.

Demo/Waitlist: https://tally.so/r/2EbRGA