01 · Frame the problem
- What business problem is the chatbot actually solving?
- Who are the users, and what do they need to walk away with?
- What would cause this project to be considered a failure?
How I think
The questions come before the architecture
Anyone can list technologies. What actually matters in a customer conversation is judgment — knowing which questions to ask, and being willing to recommend the simpler answer. Here's how I reason through a few situations that come up constantly.
Worked scenario
A customer asks for “an enterprise chatbot”
Before designing anything, I want to understand the problem well enough to know whether they even need what they asked for. These are the questions I'd work through — roughly in this order.
02 · Understand the data
- What information sources are involved?
- Is the data structured, unstructured, or both?
- What are the security and privacy constraints on that data?
03 · Choose the approach
- Is retrieval (RAG) necessary, or is the knowledge small and static?
- Does the workflow actually require an agent, or is a fixed flow enough?
- What actions, if any, should the AI be permitted to take?
04 · Define quality & risk
- What level of accuracy is required, and who decides?
- How will responses be evaluated — by what and against what?
- Where should a human stay in the loop?
05 · Plan for operations
- What's the acceptable latency and expected usage volume?
- What is the operating budget?
- How will the solution be monitored once it's live?
Notice how few of these are about technology. By the time I've worked through them, the architecture usually designs itself — and sometimes the answer is that they don't need a chatbot at all.
More situations
Judgment calls I make often
Expand each one to see how I'd think it through. There's a consistent bias here: toward the simpler solution, clear success criteria, and keeping humans in control of consequential decisions.
“Agent” is the exciting word, but most requests describe a known sequence of steps — which is a workflow, not an agent. I'll ask them to walk me through the task concretely. If we can enumerate the steps in advance, a workflow is cheaper, faster, easier to test, and easier to trust.
An agent earns its cost only when the path can't be known ahead of time — when a later step genuinely depends on what an earlier step discovers. I'd rather ship a reliable workflow now and add agentic behavior where a specific decision actually needs it.
I start at retrieval, not the model. The most common cause of weak RAG isn't the LLM — it's that the right passage never gets retrieved. I'd check chunking, embeddings, and whether the query and documents are even in the same semantic space.
Then I'd build a small labeled evaluation set so “weak” becomes measurable. Only after retrieval is solid would I look at the generation prompt, add re-ranking or hybrid search, or consider a different model. Guessing without an eval set just moves the problem around.
I'd name the gap plainly: a POC proves the idea can work; production proves it can be operated safely. The distance between them is usually evaluation, monitoring, security, governance, cost at scale, and human-in-the-loop design — not model quality.
I'd turn that into a concrete checklist with the customer so the “last 20%” is visible and planned, rather than a vague promise that it's “almost done.”
I'd ask how the decision is actually made. If someone looks at it once a day, real-time infrastructure is standing cost for latency nobody uses. Real-time is worth it when the value of the data decays in minutes. Often the honest answer is that a scheduled batch job is cheaper, simpler, and entirely sufficient — and I'd rather say so than sell complexity.
Bigger isn't automatically better, and it's definitely not cheaper. For grounded tasks, a smaller model with good retrieval usually beats a larger model without it. I'd match the model to the task: reserve the biggest models for genuinely hard reasoning, and use smaller, faster, cheaper ones where they meet the accuracy bar — measured, not assumed.
Good instinct, but every exposed tool is a new attack surface. I'd scope credentials tightly, make tools least-privilege, and put an approval gate in front of anything destructive or irreversible.
I'd also treat any content that comes back from a tool as data, not instructions — prompt injection through tool results is a real risk. The goal is a secure, standards-based path to enterprise data, not a model with unrestricted keys.
This is the highest-risk situation on the list, so I'd stop and fix it first. Without an agreed definition of success, every demo is a debate and the project never “lands.” I'd work with stakeholders to define what good looks like and how we'll measure it — before writing much code — so we're building toward a target instead of toward opinions.