Search and Filter

Llmbridge: Reducing Costs To Access Llms In A Prompt-Centric Internet

Date

AI Overview

Why it matters

Accessing large language models can be very costly, creating financial barriers for users in developing regions and education. Cost-saving proxy middleboxes can make prompt-based communication more affordable.

Study at a glance
  • Question: Can an intelligent proxy system reduce costs and manage trade-offs when accessing large language models?
  • Setting and data: WhatsApp Q&A service (100+ users, 14.7K requests) and three university computer science courses (60 students, 75K requests over 145 days).
  • Method: System development, real-world deployments, benchmark evaluations on conversation logs, and a post-course student survey.
  • AI tools: LLMBridge, GPT-4o, GPT4o-mini, Claude Haiku, Phi-3, LLaMA-3
Key findings
  • In classroom deployments across three courses, student project inference costs were kept under $10 while handling roughly 500 requests per day.
  • Intelligent model selection reduced costs by 40% compared to using an expensive model alone while preserving response quality.
  • Smart context management reduced costs by 30% to 50% compared to providing five previous messages as context.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Today's Internet infrastructure is centered around content retrieval over HTTP, with middleboxes (e.g., HTTP proxies) playing a crucial role in performance, security, and cost-effectiveness. We envision a future where Internet communication will be dominated by "prompts" sent to generative AI models. For this, we will need proxies that provide similar functions to HTTP proxies (e.g., caching, routing, compression) while dealing with unique challenges and opportunities of prompt-based communication. As a first step toward supporting prompt-based communication, we present LLMBridge, an LLM proxy designed for cost-conscious users, such as those in developing regions and education (e.g., students, instructors). LLMBridge supports three key optimizations: model selection (routing prompts to the most suitable model), context management (intelligently reducing the amount of context), and semantic caching (serving prompts using local models and vector databases). These optimizations introduce trade-offs between cost and quality, which applications navigate through a high-level, bidirectional interface. As case studies, we deploy LLMBridge in two cost-sensitive settings: a WhatsApp-based Q&A service and a university classroom environment. The WhatsApp service has been live for over twelve months, serving 100+ users and handling more than 14.7K requests. In parallel, we exposed LLMBridge to students across three computer science courses over a semester, where it supported diverse LLM-powered applications - such as reasoning agents and chatbots - and handled an average of 500 requests per day. We report on deployment experiences across both settings and use the collected workloads to benchmark the effectiveness of various cost-optimization strategies, analyzing their trade-offs in cost, latency, and response quality.