The demo works. The tenth user is where it breaks.
Standing up a chatbot against a hosted model is a five-minute job — until real traffic hits it. Then the external API's rate limits, cold responses, and occasional failures turn a slick demo into a spinner. The interesting engineering isn't the conversation; it's the layer that makes the conversation dependable.
AmazeBot is built around that layer: a Django REST backend that throttles, caches, and retries around Blenderbot so the React UI stays responsive.
A model integration is a reliability problem wearing an AI costume.
What the backend had to guarantee.
Protect the upstream model.
Request throttling at roughly 10 requests/second stops a burst of traffic — or a runaway client — from blowing the API budget.
Don't pay twice for the same answer.
Response caching returns repeated prompts instantly and cuts calls to the external model.
Fail softly.
Error handling and retry logic absorb transient upstream failures so a blip doesn't surface as a broken chat.
A thin UI over a careful API.
The React client posts a message to a single Django REST endpoint. The backend checks throttle and cache, calls Blenderbot with retries, and returns the reply over CORS to the frontend.
Keeping the model call behind the REST boundary means the API key never reaches the browser and every safeguard — rate limit, cache, retry — lives in one place the frontend doesn't have to know about.
The external API sets the real latency.
Early on, response time and reliability were entirely at the mercy of the hosted model — some calls were quick, some slow, some failed outright. The fix wasn't a faster model, it was caching and retries: repeated prompts stopped hitting the network at all, and transient errors got a second attempt instead of a red bubble.
That reframed the project. AmazeBot is less "a chatbot" and more "a well-behaved client for someone else's model" — which is the skill that actually transfers to production AI work.
A conversation, nothing more.
The interface is a single-column chat with Tailwind styling and honest loading states, so the retry-and-cache machinery underneath stays invisible. The user sees a smooth conversation; they never see the throttle decision or the cache hit that made it feel instant.
Three things, next time.
Stream tokens over WebSockets.
Streaming replies would replace the wait-then-appear pattern with real-time text and make the bot feel far more alive.
Persist conversation history.
Storing turns per user would give the bot context across a session instead of treating every message as isolated.
Add authentication.
JWT-based auth would tie throttling and history to real users rather than anonymous traffic.