Building With AI: Frequently Asked Questions

Building with AI got dramatically more accessible, but the gap between a demo that wows in a meeting and a feature you can trust in production is where most projects stall. I build AI systems for a living — Otto, the engine behind RunOctopus — so these are the questions builders and operators actually ask me, answered the way I'd answer a friend. If you want the long-form version, I wrote a fuller field guide to building with LLMs and a deeper piece on AI agents that work.

Which AI model should I use?

Match the model to the job, not to whatever's topping a leaderboard this week. Reach for a frontier model like Claude Opus 4.8 when the task needs hard reasoning, long context, or agentic tool use; a balanced model like Sonnet 4.6 for the bulk of production traffic; and a fast, inexpensive model like Haiku 4.5 for high-volume, simpler calls. The move I use every time: start on the strongest model to prove the task is even possible, then step down to the cheapest model that still passes your evals. That sequence saves you from both under-powering the build and overpaying for it.

What's the difference between prompting, RAG, and fine-tuning?

These are three different levers, and you should try them in order. Prompting is just instructing the model in context — your first and often only tool. RAG (retrieval-augmented generation) fetches relevant documents at runtime and drops them into the prompt, which is how you give a model your own current, private knowledge without retraining anything. Fine-tuning actually adjusts the model's weights to bake in a consistent style or narrow behavior. Most problems are solved by better prompting plus RAG long before fine-tuning earns its cost and complexity, so reach for it last.

Do I actually need AI agents?

Usually less than you think. An agent is a model that plans, calls tools, and loops until a task is done — the right tool when the work genuinely needs multiple steps and decisions you can't script ahead of time. But if your task is a single input-to-output transform, a plain prompt with structured output is simpler, cheaper, and far easier to trust. Add agentic complexity only when a straight-line call provably can't do the job. I go deep on where agents earn their keep in build AI agents that work.

How do I stop my AI from hallucinating?

You reduce it, you don't fully eliminate it — and designing as if you could is the mistake. The biggest lever is grounding: give the model the real source material through RAG and instruct it to answer only from what's provided and to say plainly when it doesn't know. Ask for citations back to the source, lower the temperature on factual tasks, and verify high-stakes outputs with a second check or a human. Build assuming the model can be confidently wrong, because occasionally it will be, and your guardrails are what keep that from reaching the user.

How do I know my AI feature actually works (evals)?

Build evals — a fixed set of real inputs paired with known-good outputs that you run every single time you change a prompt or swap a model. Without them you're tuning by vibes, and every change becomes a coin flip you can't see the result of. Start with a handful of cases that capture what "good" and "broken" actually look like for your task, score each new version against them, and grow the set every time something breaks in production. Evals are the thing that turns AI work from guessing into engineering.

How much does it cost to run an LLM feature?

It's usage-based, priced per token of input and output, and it varies a lot by model tier — a frontier model costs meaningfully more per call than a small fast one. So the cost question is really a routing question: send the genuinely hard calls to the strong model and the high-volume easy ones to the cheap model. Prompt caching for repeated context and trimming bloated prompts cut real money over time. Before you ship, estimate tokens-per-call times calls-per-day so the bill isn't a surprise.

What is a system prompt?

The system prompt is the standing instruction that frames every conversation — the model's role, its rules, its tone, and its hard constraints — held separate from each individual user message. It's where you set the behavior you want to persist across every call: who the assistant is, what it must never do, and how to format its output. A clear, specific system prompt is one of the highest-leverage things you can write in the whole build, and it's worth iterating on directly against your eval set rather than tweaking by feel.

Is my data used to train the model?

It depends on the provider and the specific tier, so always check the actual terms rather than assuming. Anthropic's API and commercial offerings are not trained on your business data by default, and most enterprise API providers draw the same line — though consumer chat apps sometimes don't. The rule I follow: read the data-usage policy for the exact product you're using, prefer the API or business tier for anything sensitive, and never assume a consumer-app default carries over to your build. When in doubt, treat the policy as the source of truth, not your memory of it.

How do I handle latency so my AI feature feels fast?

Stream the response so users watch tokens appear as they're generated instead of staring at a spinner — perceived speed matters as much as raw speed. Route simpler steps to a faster, smaller model, keep prompts and context lean, and cache the parts of your prompt that repeat across calls. For multi-step or agentic work, show progress as it goes so a slow task feels like it's working rather than frozen. Most "this AI feature feels slow" complaints are really "this AI feature feels unresponsive," and that's a UX fix as much as a speed one.

Can I trust AI-generated content for SEO and AEO?

Only if you hold it to a real quality bar — how the page was created matters far less than whether it genuinely answers the question. AI is excellent for drafting and structure, but unedited generic output is exactly the thin content answer engines skip and that quietly erodes your trust signals. Use AI to draft, then ground it in real facts, edit hard for accuracy and your own voice, and make every claim specific enough to be quotable. Do that and AI-assisted content competes fine; skip it and you've built a content farm. The AEO mechanics are in get cited by AI search.

What are structured output and tool use?

Structured output is forcing the model to return data in a strict shape — like JSON matching a schema — so your code can rely on it instead of parsing free-form prose. Tool use is letting the model call functions you define (search a database, hit an API, run a calculation) and fold the results into its answer. Together they're how you turn a chat model into a dependable component inside real software rather than a clever text box, and they're the foundation everything agentic is built on. If you're building anything beyond a single text response, you'll want both.

How do I get started building with an LLM?

Pick one narrow, real task, write a clear prompt for it, and call the API directly — you can genuinely have something working in an afternoon. Get a single end-to-end call working before you add RAG, tools, or agents; every bit of complexity should be earned by a problem you actually hit, not added preemptively. Build a tiny eval set early so you can tell whether your changes are helping or hurting. Start small, ship the working core, then expand outward from it.

Should I build on one model provider or stay flexible?

Start with one strong provider so you can move fast, but keep the model call tucked behind a thin abstraction layer so swapping later is cheap. Lock-in risk is real, but wiring up multi-provider plumbing before you've even proven the feature works is premature optimization that just slows you down. Get it working on one good model, prove the value, then add flexibility if and when cost, availability, or a specific capability gives you a concrete reason to. Flexibility you never use is just complexity you pay for.