Choosing how to build an AI feature is one of the first architectural decisions you'll need to make. It affects more than how quickly you can launch. Your choice will shape data privacy, operating costs as usage grows, response latency, model update cycles, and the expertise your team needs to maintain the feature.
There are three main approaches. You can connect to a third-party model through an API, run an open-source model using your own infrastructure, or develop a custom model tailored to your use case. Each comes with different trade-offs.
So, which one makes sense for your product? The answer depends on five factors: your data requirements, expected usage, performance needs, available team expertise, and long-term product plans. Let's break down how each option compares.
Option 1: API (OpenAI, Anthropic, Gemini, Cohere)
Using a third-party API is usually the fastest way to add an AI feature to an existing product. Instead of hosting a model, managing GPUs, or building an ML infrastructure, you send requests to a provider such as OpenAI, Anthropic, Google, or Cohere and receive generated responses. The provider handles model serving, infrastructure maintenance, and updates. Your team can focus on integrating the API into the product and building the logic around it.
When an API is the right choice
An API works particularly well when you want to launch an AI feature without a long development cycle or a dedicated machine learning team. You can start with an existing model, test it against your use case, and adjust the integration as you learn what users need.
This approach makes sense when:
- You need to launch quickly. You don't have to source model weights, configure inference infrastructure, or set up GPU servers. A developer can connect an API, build the necessary backend logic, and get an initial version ready for testing.
- You want access to advanced models. Providers maintain and improve their models, so your product can benefit from new capabilities without your team having to train or update the model itself.
- Your usage is still unpredictable. With pay-as-you-go pricing, you can start small and pay for actual API consumption rather than investing in dedicated infrastructure upfront.
- Your team doesn't have ML infrastructure expertise. You still need developers who understand API integration, data handling, and application architecture, but you don't need to operate the model-serving stack.
The main advantage is a lower operational burden. You can spend more time improving the feature itself and less time keeping the underlying model running.
AI API cost comparison: What happens at scale?
Most AI APIs use token-based pricing. You pay separately for input tokens (the prompt, instructions, conversation history, and other context sent to the model) and output tokens (the response it generates). Output tokens usually cost more than input tokens, although the exact rates depend on the provider and model.
For example, as of November 2026, Anthropic lists Claude Sonnet 5.5 at $2 per million input tokens and $10 per million output tokens. If your application processes one million input tokens and generates one million output tokens, the model usage costs $12.
That may sound inexpensive, but the total changes with usage. Imagine your feature handles 100,000 requests per month, with an average of 2,000 input tokens and 500 output tokens per request. That's 200 million input tokens and 50 million output tokens. At the same rates, the monthly API bill would be around $900.
And this is only an example based on one model's standard token rates. Actual costs depend on the model you choose, how much context you send with each request, response length, caching, and any additional API services you use. A long conversation history or a large knowledge base retrieved for every query can increase input token consumption significantly.
When comparing OpenAI API alternatives, look beyond the headline price per million tokens. OpenAI, Anthropic, Google Gemini, and Cohere offer different models, pricing tiers, and capabilities. A cheaper model may be sufficient for classification or simple support questions, while a more capable model may be worth the higher cost for complex reasoning or multi-step tasks. Test candidates with representative inputs from your product before choosing one.
Data privacy: Your model runs elsewhere
When you use a third-party API, your application sends the relevant input data to the provider's infrastructure for processing. Depending on the feature, this might include user prompts, conversation history, uploaded documents, or information retrieved from your database.
This doesn't automatically mean the provider uses your data to train its models. For example, OpenAI states that API data isn't used for training by default, while Anthropic's commercial API terms also exclude customer inputs and outputs from training unless the customer explicitly permits it. However, data retention policies and available privacy controls differ between providers and API features.
Your team needs to check where data is processed, how long prompts and responses are retained, whether zero-data-retention options are available, and what contractual or regulatory requirements apply. You should also decide what information the application actually needs to send. Removing unnecessary personal or confidential data before an API request can reduce exposure.
If your product handles sensitive information or operates under strict data residency requirements, these considerations may influence your provider choice or make a self-hosted model worth exploring.
Latency: The extra network step
An API call adds a network round trip to the model's processing time. Your application sends a request to an external server, waits for the provider to process it, and receives the response. Network conditions, the distance to the provider's infrastructure, request size, model choice, and provider load all affect the total time.
This extra step can matter in real-time features, such as voice assistants, live chat, or interactive recommendations. However, the network round trip is only one part of the total response time. The model's own inference speed and the amount of text it generates can have an even greater impact.
You can reduce the perceived wait by streaming responses as they're generated rather than waiting for the complete answer. Choosing a faster model, keeping prompts concise, and avoiding unnecessary API calls can also help. Still, you have less control over infrastructure-level latency than you would when running a model yourself.
Option 2: Open-Source Models (Llama, Mistral, Phi, Qwen)
Open-source models offer more control over how an AI feature works and where it runs. Instead of relying on a proprietary model hosted by OpenAI or Anthropic, you can use publicly available model weights from projects such as Meta's Llama, Mistral, Microsoft's Phi, or Alibaba's Qwen. You can run them on your own infrastructure or use a third-party hosting service.
One distinction worth knowing: many popular models are technically open-weight rather than fully open-source. Their weights are available, but training data, code, and licensing terms vary. Always check the license before using a model commercially.
When to use an open-source LLM in a startup
Open-weight models make sense when you need more control over data, customization, or infrastructure costs. They're worth considering when:
- Data privacy matters. Self-hosting lets you process prompts and responses within your own infrastructure, which can be useful for products handling sensitive business or customer data.
- Usage is high and predictable. Running a model on your own GPUs can become cheaper than paying per token through a commercial API. However, infrastructure costs continue even when usage drops.
- You need customization. You can fine-tune models, optimize them through quantization, and adapt them to domain-specific tasks, depending on the license.
- You need deployment flexibility. Models can run in your cloud environment, on dedicated servers, or, in some cases, directly on supported devices.
The trade-off is that your team takes on more responsibility for infrastructure, performance, and maintenance.
Infrastructure requirements: Where and how to host an open-source model
Self-hosting requires computing resources, usually GPUs with enough video memory (VRAM) to load the model and handle requests. Requirements depend on model size, quantization, context length, and expected traffic.
You don't have to buy hardware. Providers such as Runpod and Lambda let you rent GPU instances. For example, an NVIDIA H100 PCIe with 80 GB VRAM on Runpod's Secure Cloud is listed at $2.89 per hour, or roughly $2,110 per month if kept running around the clock. This excludes storage, networking, redundancy, and engineering costs.
There's also a middle option: managed hosting. Services such as Hugging Face Inference Endpoints and Together AI let you run open-weight models without managing the underlying GPU infrastructure yourself. You still get model choice, but rely on an external provider for hosting.
Open-source vs. frontier models: How big is the capability gap?
Open-weight models have improved significantly and can handle many practical tasks, including summarization, classification, and information extraction. However, performance still varies by model and use case.
A Llama vs GPT-4 comparison, for example, won't have one universal winner. A smaller open-weight model may be sufficient for a focused feature, while a proprietary frontier model may perform better on complex reasoning or coding tasks.
Epoch AI's 2026 research found that leading open-weight models lagged behind the best closed models by an average of about four months on its capabilities index during January–May 2026. The gap is narrowing, but that doesn't mean the models are interchangeable.
Test candidates using your own product scenarios. Compare answer quality, latency, reliability, and inference costs rather than relying on general benchmarks.
Customization and maintenance: More control, more responsibility
Open-weight models can be adapted through fine-tuning, quantization, or retrieval-augmented generation (RAG). However, not every feature needs fine-tuning. For example, if an AI assistant needs to answer questions using company documentation, RAG may be enough.
Self-hosting also means maintaining the deployment. Your team must monitor GPU usage, manage scaling, handle failures, apply security updates, and evaluate new model versions.
For startups with unpredictable usage and limited infrastructure expertise, a managed API may be easier to maintain. But if your product has steady demand, strict data requirements, or specialized workloads, self-hosting may justify the additional effort. Perpetio's AI integration services cover different approaches to integrating AI into software products.
Option 3: Fine-Tuned or Custom Model
Sometimes, an existing model doesn't quite fit your product. It may struggle with specialized terminology, produce inconsistent outputs, or fail to handle a task that requires specific domain expertise. In these cases, you can adapt an existing model through fine-tuning or, in more demanding scenarios, train a model using your own data.
When to fine-tune an LLM
Fine-tuning makes sense when you need to change how a model performs a task rather than simply give it more information. It can be useful when:
- Your product serves a highly specialized domain. You need consistent responses using industry-specific language, formats, or workflows.
- You have valuable proprietary data. Your company has a large collection of high-quality examples that can help the model perform a specialized task.
- Your usage is very high. A fine-tuned, smaller model may reduce per-request costs or latency enough to justify the initial training investment.
Training a model from scratch is a much bigger undertaking. For most startups, fine-tuning an existing open-weight model is a more realistic starting point than developing an entirely new foundation model.
Fine-tune vs. RAG: Which approach do you need?
Fine-tuning and retrieval-augmented generation (RAG) solve different problems. Fine-tuning changes how a model behaves, while RAG gives it access to external information when generating an answer.
For example, if you're building an AI assistant that answers questions about company policies, product documentation, or customer records, RAG is usually the better starting point. You can update the knowledge base without retraining the model. If the assistant needs to follow a specific response format, use specialized terminology, or perform a task consistently, fine-tuning may help.
As explained in Perpetio's guide to LLM and RAG development, the two approaches can also work together: RAG provides current information, while fine-tuning helps the model follow specific behaviors and formats.
Research published in the Findings of EMNLP 2025 also explores how fine-tuning can improve RAG pipelines, showing that the approaches aren't mutually exclusive.
Custom AI model training cost: What are you paying for?
The cost depends on the model, training method, dataset, and deployment requirements. A small LoRA fine-tuning experiment may cost only a few dollars in GPU time, but preparing training data, evaluating results, and integrating the model into a production environment require additional work.
For a broader estimate, Cadence's 2026 fine-tuning cost breakdown puts real business projects at roughly $5,000–$50,000 or more. Much of that budget goes toward data preparation, engineering, evaluation, and deployment rather than the training run itself.
Training a model from scratch costs considerably more and requires substantial datasets, computing resources, and machine learning expertise. It's rarely justified for an early-stage product.
Start with an API and RAG, then scale
For most products, the practical approach is to start with a commercial API and RAG. This lets you validate the feature, understand real user requests, and collect examples of where the model falls short.
Once you have enough usage data, you can identify specific problems that prompting or retrieval can't solve. If fine-tuning can measurably improve quality, reduce inference costs, or meet performance requirements, you can introduce it later.
You don't need to invest in a custom model before you know what your product actually needs.
The Decision Framework: 5 Questions That Determine Your AI Architecture
Still unsure which approach to choose? These five questions can help you decide between an API, an open-weight model, or a custom solution. The goal is to find the simplest architecture that meets your product's requirements without paying for infrastructure or expertise you don't need yet.
1. Is your data sensitive enough to require on-premise deployment?
If your product handles confidential information, regulated data, or sensitive customer records, check whether a third-party API meets your security, privacy, and data residency requirements. If it doesn't, self-hosting an open-weight model or building a custom solution may give you more control over where data is processed and stored.
However, using an API doesn't automatically mean compromising privacy. Some providers offer enterprise data controls and zero-data-retention options.
2. Will your product exceed 10 million tokens per month?
At higher usage, API bills can become substantial, making self-hosting worth investigating. But 10 million tokens isn't a universal break-even point. The actual cost depends on model pricing, GPU requirements, utilization, and maintenance.
For example, a 2026 OECD analysis of AI openness found that private hosting wasn't economically attractive for small workloads below 100 million tokens per month under its assumptions. At one billion tokens monthly, the estimated break-even period was around 30 months.
Calculate your expected API spending and compare it with the full cost of running your own infrastructure before deciding.
3. Does your domain require knowledge the model doesn't have?
If your AI feature needs access to company documents, product information, or frequently updated data, start by considering retrieval-augmented generation (RAG). Fine-tuning is more appropriate when you need to change how the model responds, follows instructions, or performs specialized tasks.
This is the central distinction in the fine-tune vs RAG vs API AI decision: an API provides access to a model, RAG supplies relevant external knowledge, and fine-tuning adapts model behavior. You can combine all three.
4. Does your feature require a response in under 200 ms?
If low latency is essential, consider smaller models, on-device inference, caching, or a hybrid architecture. An API request introduces network overhead, but model processing time, prompt length, and response generation also affect latency.
Be specific about what needs to happen within 200 ms. Receiving the first token is different from generating a complete answer. NVIDIA's inference metrics documentation explains how to measure time to first token and end-to-end response latency.
5. Does your team have machine learning engineers?
If not, start with a commercial API. Your developers can integrate an existing model without taking responsibility for GPU infrastructure, model serving, and training pipelines. You can also use RAG to give the model access to product-specific information.
As your product grows, you can revisit the architecture based on actual usage, costs, and performance requirements. You don't need to invest in a custom model before you have evidence that an API no longer meets your needs.
For a closer look at the implementation options, explore our guide to integrating ChatGPT or Claude into an existing mobile app.
What we've seen in real projects: How different AI approaches fit different products
The right AI architecture depends on what the product needs to do, how it handles data, and what users expect from it. These Perpetio projects show how those requirements shape implementation.
- JRNEY: API-based AI for travel planning. JRNEY is Perpetio’s own AI trip planner, built around OpenAI GPT-4. Users describe a destination, dates, and interests in plain language, and the app generates a day-by-day itinerary. They can then refine it through an AI chat. Using an existing model API made it possible to build these conversational planning features without developing a foundation model from scratch.
- Recordr: Local processing for privacy. Recordr is a macOS meeting recorder that uses AI to transcribe, analyze, and summarize conversations, including speaker identification and action items. Its key architectural requirement was privacy: all data processing happens locally on the user's Mac, so meeting information isn't sent to third-party services. This shows why deployment and data handling can matter as much as model capabilities when choosing an AI architecture.
These projects illustrate two different priorities: JRNEY uses a hosted model to power an interactive AI feature, while Recordr puts local processing and data privacy at the center of its design. Both required the AI implementation to serve the product's actual use case.
Build an AI feature with the right architecture
Choosing between an API, an open-weight model, and a custom solution affects your product's costs, performance, privacy, and maintenance requirements. Perpetio helps businesses plan and build AI-powered software, from selecting the right approach to integrating and deploying the finished feature.
Explore our AI integration services to see how we help turn product ideas into working AI features.
