How to Integrate LLMs Into Your Existing Product

Most AI projects we work on aren't greenfield. They're integrations into existing systems — adding a chatbot to a customer portal, building intelligent search for a knowledge base, or automating document classification in a workflow that's been running for years.

Adding LLMs to an existing product is different from building an AI-first product. You have constraints: existing databases, established APIs, users who expect things to keep working. Here's the practical guide we wish someone had written for us.

Step 1: Choose the Right Integration Pattern

Before picking a model, decide what the AI actually does.

Pattern 1: Conversational (Chatbot)

User sends a message, AI responds. Optionally grounded in your data (RAG).

Best for: Customer support, internal Q&A, documentation assistants.

// Simplest chatbot integration
async function chat(userMessage: string, conversationHistory: Message[]) {
  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [
      { role: "system", content: SYSTEM_PROMPT },
      ...conversationHistory,
      { role: "user", content: userMessage },
    ],
    stream: true,
  });
  return response;
}

Pattern 2: Classification

AI categorizes input into predefined buckets. No free-form generation.

Best for: Support ticket routing, content moderation, lead scoring, sentiment analysis.

async def classify_ticket(ticket_text: str) -> str:
    response = await client.chat.completions.create(
        model="gpt-4o-mini",  # Cheaper model is fine for classification
        messages=[{
            "role": "system",
            "content": "Classify this support ticket. Return ONLY one of: billing, technical, feature_request, bug_report"
        }, {
            "role": "user",
            "content": ticket_text
        }],
        temperature=0,  # Deterministic output
    )
    return response.choices[0].message.content.strip()

Pattern 3: Content Generation

AI creates content based on structured input. Emails, reports, summaries.

Best for: Marketing copy, report generation, email drafting, data summarization.

Pattern 4: Semantic Search

Replace keyword search with meaning-based search using embeddings.

Best for: Product catalogs, documentation, knowledge bases, FAQ systems.

# Embedding-based semantic search
async def search(query: str, top_k: int = 5):
    query_embedding = await embed(query)
    results = await vector_db.query(
        vector=query_embedding,
        top_k=top_k,
        include_metadata=True
    )
    return results

Step 2: Pick the Right Model

Step 3: Architecture Decisions

Sync vs. Async

Synchronous (streaming): User sends a message, sees the response stream in real-time. Good for chatbots and interactive features.

Asynchronous (background): User triggers a task, gets notified when it's done. Good for document processing, batch classification, report generation.

// Async pattern with job queue
async function processDocument(documentId: string) {
  // Enqueue the job — don't block the request
  await queue.add("process-document", { documentId });
  return { status: "processing", jobId: generateId() };
}

// Worker picks it up
queue.process("process-document", async (job) => {
  const doc = await db.document.findUnique({ where: { id: job.data.documentId } });
  const summary = await llm.summarize(doc.content);
  await db.document.update({
    where: { id: doc.id },
    data: { summary, status: "processed" },
  });
  await notify(doc.userId, "Document processed");
});

Where to Put the AI Layer

Don't bolt the LLM directly into your existing API routes. Create a separate AI service layer:

Existing App → AI Service Layer → LLM Provider
                    ↓
              Cache / Vector DB

This separation gives you:

• Easy model swapping (switch from OpenAI to Mistral without touching your app)
• Centralized prompt management
• Cost tracking per feature
• Circuit breakers for LLM outages

Step 4: Cost Management

LLM costs can spiral fast. Here's how to keep them under control:

1. Use the cheapest model that works. GPT-4o-mini handles 80% of use cases.
2. Cache aggressively. Same question = same answer. Cache embeddings and responses.
3. Set per-user rate limits. 50 messages/day is generous for most features.
4. Truncate context. Don't send 100 messages of history. Send the last 10.
5. Monitor daily. Set up alerts for spend thresholds.

Real cost example: A customer support chatbot handling 1,000 conversations/day with GPT-4o-mini costs approximately €150-300/month. The same volume with GPT-4o costs €1,500-3,000/month.

Step 5: The RAG Decision

If your AI needs to answer questions about your data, you need RAG (Retrieval-Augmented Generation). If it just needs to follow instructions and generate content, you don't.

You need RAG when:

• The AI should know about your documentation, products, or internal knowledge
• Answers need to be grounded in facts (not hallucinated)
• The knowledge base changes frequently

You don't need RAG when:

• The AI follows fixed instructions (classification, formatting)
• The context fits in the system prompt (< 100K tokens)
• Accuracy isn't critical (brainstorming, creative writing)

We wrote a detailed comparison in our RAG vs. Fine-Tuning post.

The Integration Timeline

Most integrations ship in 3-5 weeks if the existing product has a clean API.

Need help integrating AI into your product? See our AI & LLM Integration services.

Sources

• OpenAI API Pricing
• Anthropic API Pricing
• Mistral AI Platform
• LangChain — RAG Documentation