We built our own chatbot that answers questions about Anthillo — who we are, what we do, how to reach us. It’s a chat widget embedded as a WordPress plugin on anthillo.com, backed by a Java service (Spring Boot, Spring AI). This isn’t a product for an external client — it’s dogfooding: a small team learning to build AI features on our own low-stakes case, a marketing FAQ bot. Along the way we made a handful of classic beginner AI-engineering mistakes, and they’re more instructive than most success stories.
Version 1: the whole company crammed into one system prompt
The first implementation (ChatbotGPTServiceImpl) was the simplest thing that could work: we called OpenAI directly through Spring AI’s ChatClient, appending a huge, hardcoded SYSTEM_PROMPT with the full company description — services, areas of expertise, contact details — to the user’s question. No retrieval, no knowledge base, just “stuff everything into the context and hope the model finds the right part.”
private static final String SYSTEM_PROMPT = """
You are an assistant representing Anthillo.
Anthillo provides Software development, Infrastructure...
Anthillo provides Software development, Infrastructure...
Anthillo provides Software development, Infrastructure...
Contact: office@anthillo.com
"""; // the same block pasted three times
public String askAnthillo(String question, String lang) {
String prompt = SYSTEM_PROMPT + languageInstruction(lang) + question;
return chatClient.call(prompt);
}When we looked closer at that string, it turned out the same company-description block was pasted into it three times — paragraphs about “Software development,” “Infrastructure,” “Security” repeated one after another. A classic effect of iterative prompt editing: someone added a section, someone else added a “similar” one further down the file, and nobody ever cleaned it up. The result: every request to the model, no matter what the user asked, carried the same redundant text three times over. Higher token cost, a real risk that the model would “get lost” in duplicated context, and zero ability to scale the knowledge base — adding a new page to the bot meant editing Java code and redeploying the whole service. Company knowledge lived inside application code instead of a separate, manageable layer, and nobody wanted to code-review a marketing paragraph.
Version 2: a dedicated RAG service
The second approach (ChatbotServiceImpl, the one currently wired into the controller) looks different: the Java service no longer knows anything about company content. Instead, it calls a separate RAG service over a plain HTTP GET, passing the question and language, and gets back a generated answer based on retrieval over a company knowledge base.
public String askAnthillo(String question, String lang) {
String url = ragServiceUrl + "/query/?question=" + encode(question)
+ "&generation_language=" + lang;
JSONObject response = restClient.get(url, JSONObject.class);
return response.getString("response");
}The gain is clear: company knowledge is decoupled from the service code and can be updated independently, without redeploying the Java app. Application code stopped being a storage bin for marketing copy and became a thin orchestration layer instead.
Takeaway. Prompt stuffing works fine as a prototype, but it doesn’t scale — once the knowledge grows, you need a separate retrieval layer, not an ever-larger string in your codebase.
What was missing: sessions, throttling, moderation
RAG solved one problem, not all of them. The backend is fully stateless — one question, one answer, no conversation memory, no database on the service side. Our widget’s own TODO list admits as much: “save user context,” “save user questions,” “user throttling,” “moderation API integration” — none of it implemented. For a public widget on a company website, that’s not a cosmetic gap. Without throttling, any visitor can generate unlimited requests against a paid API. Without moderation, we have no guarantee what someone types into the chat box or how the model responds. Without sessions, every answer is disconnected from conversation context, which hurts UX but at least adds no extra risk — the one gap we’re currently comfortable accepting as a trade-off for a simple FAQ bot.
Inconsistent JSON contracts between services
A separate lesson: the Java service parsed the RAG backend’s response as a content field, while the actual response (and the fixture used in the unit test) had the shape {"query": "...", "response": "..."}. Code and test had drifted apart, and nobody noticed, because the integration between the two services was tested by hand, not by contract. If the RAG backend had ever actually renamed that field, we’d have gotten a JSONException in production instead of a red test in CI. The lesson is simple: an HTTP contract between two services, even internal ones, deserves a contract test, not just “it works in my Postman.”
Secrets don’t belong in code or in logs
The last, most painful lesson is about secrets hygiene. An LLM provider’s API key (OpenAI, in our case) has to be treated like a production password — never log it, not even at DEBUG or INFO level, because logs end up in places you don’t control as tightly as your repository. Never commit IDE configuration files (.idea/) or keystores (*.p12) — that’s exactly the kind of file where a secret ends up by accident, since IDEs save runtime environment variables alongside the rest of the project settings. Passwords and keys belong in a secrets manager or in environment variables injected at runtime, not in an application.yml sitting in the repo. And one more thing: the moment you suspect a secret might have leaked — even “just locally” — rotate it immediately. Rotation is cheap; cleaning up after an incident is not.
Our chatbot works today, but the TODO list is still longer than the “done” list. Next up are short-lived conversation sessions, per-user throttling, and wiring up content moderation before we start treating this widget as more than an experiment. The real value of this project isn’t the chatbot itself — it’s the concrete checklist of mistakes we now run through on every subsequent AI project before we put anything in front of real users.