Weekly knowledge base review cycle active · Monthly analytics reports running · Resolution accuracy improved from 74% (week 1) to 82% (week 8) and climbing
Who we worked with
An Australian online home goods retailer doing $6.8M in annual revenue, with a 4-person support team handling 1,400+ tickets per month — 78% of which were repetitive queries about order status, return policies, product availability, and shipping timelines.
Average first-response time was 11 hours on weekdays and 28+ hours over weekends. During semi-annual sale events, the queue spiked to 3,200+ tickets in a single week. Customer satisfaction had dropped to 71%, support costs were running $184K/year, and a team that was burning out had no visibility into which products or policies were driving the most confusion.
| Metric | Before | After | Change |
|---|---|---|---|
| First response time | 11 hrs (weekday) · 28+ hrs (weekend) | < 8 seconds (AI) · < 2 hrs (escalated) | 99% faster |
| Queries resolved without human agent | 0% (all manual) | 82% AI-handled | 82% automation rate |
| Customer satisfaction (CSAT) | 71% | 94% | +23 points |
| AOV (AI-assisted sessions) | $87 baseline | $104 | 20% increase |
| Monthly tickets requiring human | 1,400+/month | ~250/month | 82% reduction |
| Support cost per ticket | $11.20 | $3.40 (blended AI + human) | 70% reduction |
A support team surviving, not serving — copying and pasting 400 times a month
78% of 1,400 monthly tickets were identical questions that needed no human judgment to answer — but there was no system to answer them automatically, no visibility into what was being asked, and no support at all outside business hours.
Support Team Overwhelmed by Repetitive Queries
Of 1,400+ monthly tickets, 78% were repetitive: order status (31%), return policy (19%), product availability/specifications (16%), shipping timeline queries (12%). The team spent the majority of their time copy-pasting identical answers. During peak sale events, ticket volume tripled — stretching response times to 5+ days, at which point customers initiated chargebacks or left negative reviews instead of waiting.
78% repetitive tickets · 5-day queue during salesWeekend and After-Hours Black Hole
The team worked Monday–Friday, 9 AM–5 PM AEST. But 34% of orders were placed between 6 PM and midnight, and 22% on weekends — precisely when no support existed. Customers with urgent issues (wrong item shipped, delivery problem) had no way to get help until Monday. Weekend CSAT was 62%. Weekend-placed tickets averaged 28+ hours before a first response.
56% of orders placed outside support hours · 62% weekend CSATNo Product Discovery Support
A 2,800-SKU catalog with complex attributes — dimensions, materials, color variants, room compatibility — and a site search limited to basic keyword matching. Searching "grey couch for small living room" returned irrelevant results. Customers needing help choosing between products emailed support and waited hours, by which time many had left the site. The client estimated $320K/year lost to abandoned consideration-stage sessions.
$320K/year in abandoned consideration sessionsNo Data-Driven Insight from Support Interactions
The team used a shared Gmail inbox with canned responses — no ticketing system, no tagging, no analytics. The client had no idea which products generated the most questions, what information was missing from product pages, or which policies confused customers most. Every decision about product descriptions and FAQ content was a guess.
Shared Gmail inbox · zero support analyticsGPT-4 with RAG — resolving 82% of queries instantly, 24/7
A 16-week build — data audit through shadow-mode testing and live launch — using retrieval-augmented generation so every answer is grounded in real-time product, order, and policy data. No hallucinated tracking numbers. No made-up return policies.
Data Audit & Architecture
Wks 1–3Knowledge Base & Vector Store
Wks 3–6Integration & Assistant Build
Wks 5–12Shadow Testing & Launch
Wks 10–16- Analyzed 6 months of support emails (8,200 conversations) — categorized every query by type, complexity, and resolution path; identified 82% answerable from 3 existing data sources
- Designed RAG architecture: GPT-4 as the reasoning layer with real-time retrieval from the Shopify product catalog, ShipStation order data, and a vector-indexed knowledge base
- Ingested and indexed 340 help articles, 2,800 product SKUs (full attribute data: dimensions, weight, materials, color options, room recommendations, compatibility notes), shipping policies, and FAQs into a Pinecone vector database
- Built the retrieval pipeline: customer query → embedding → top 5 relevant documents retrieved → passed as context to GPT-4 alongside conversation history
- Built the conversational interface as a React chat widget embedded in the Shopify storefront and integrated into the client's email support channel
- Connected to ShipStation API for real-time order tracking — any order by number or customer email returns current status, carrier, tracking link, and estimated delivery, instantly, 24/7
- Built the natural language product recommendation flow: query → attribute extraction → catalog search → ranked results with images, prices, and live stock status
- Implemented smart escalation: sentiment analysis detects frustration (repeated questions, negative language, explicit requests) and routes to the human team with full conversation context — no customer repeats themselves
- Ran 3 weeks of shadow mode: the assistant processed all incoming queries in parallel with the human team, but responses were only shown internally — accuracy measured against human responses
- Matched or exceeded human accuracy on 74% of queries in week 1, improving to 82% by week 3 after prompt tuning and knowledge base enrichment
- Launched to customers with a "Chat with us" widget; human team shifted to monitoring escalations and reviewing flagged conversations — no longer buried in repetitive tickets
- Weekly review cycle: team reviews all conversations with negative feedback or escalations; knowledge base updated based on new products, policy changes, and recurring edge cases
- Monthly analytics report: top query categories, resolution rates, CSAT trends, and product pages generating the most questions — used by merchandising to improve descriptions
- Analytics revealed 3 categories drove 40% of all questions (outdoor furniture assembly, bedding size compatibility, lighting dimmer compatibility) — product pages updated, query volume in those categories dropped a further 15%
Grounded in real data — no hallucinated responses
Indexed from 8,200 analyzed support conversations
Resolution accuracy: 74% (wk 1) → 82% (wk 8)
Every choice made for accurate, real-time AI support at scale
| Technology | Role | Why This Choice |
|---|---|---|
| GPT-4 (OpenAI API) | Conversational AI core | Best-in-class natural language understanding; handles nuanced product and policy queries |
| Pinecone | Vector database | Fast similarity search for RAG retrieval across 2,800 SKUs + 340 help articles in real time |
| Python (FastAPI) | Backend API layer | Lightweight async framework for real-time chat processing with low latency |
| Node.js | Integration middleware | Connects Shopify, ShipStation, and AI services into a unified request pipeline |
| React | Chat widget frontend | Embedded in Shopify storefront; responsive on mobile and desktop; no page reload required |
| ShipStation API | Order tracking | Real-time order status, carrier info, and tracking links for 24/7 instant responses |
| Shopify Storefront API | Product catalog | Live inventory, pricing, and full attribute data for natural language product search |
| AWS (ECS, Lambda) | Cloud hosting | Auto-scaling for traffic spikes during semi-annual sale events (3,200+ tickets/week) |
| MongoDB | Conversation logging | Stores full conversation history for analytics, improvement cycles, and escalation context |
| Klaviyo | Post-chat email flows | Triggered follow-ups based on chat interactions — abandoned product discussions, return confirmations |
Results that transformed support into a revenue channel
82% query automation, a 23-point CSAT jump, $106K in annual support savings, and a product recommendation capability that outperforms the site's own search — all live within 16 weeks.
Without Human
Savings
(was 11 hours)
71% → 94%
4 engineers, full AI support stack in 16 weeks
A compact specialist team — AI/ML lead, backend integration engineer, frontend developer, and QA — covering everything from RAG architecture and GPT-4 prompt engineering to Shopify widget delivery and shadow-mode accuracy testing.
AI / ML Lead
GPT-4 RAG architecture design, Pinecone vector store setup and indexing, prompt engineering, sentiment detection, shadow-mode accuracy measurement and tuning
Backend Engineer
Python (FastAPI) backend API, ShipStation + Shopify Storefront API integrations, Node.js middleware, MongoDB conversation logging, AWS ECS + Lambda deployment
Frontend Developer
React chat widget embedded in Shopify storefront, product recommendation cards with live images and stock status, mobile-responsive chat UI, Klaviyo post-chat flow integration
QA Engineer
3-week shadow mode validation, accuracy benchmarking against human responses, edge case identification, escalation flow testing, post-launch monitoring and analytics setup
16-Week Full Delivery
Shadow-mode validated before public launch · 74% → 82% accuracy improvement · Weekly improvement cycle live from day one post-launch
Ongoing Optimization
Weekly flagged-conversation reviews, monthly analytics reports, knowledge base updates for new products and policy changes, CSAT trend monitoring
What the client said
"Before this, our support team was just surviving — not serving. They were copying and pasting the same 'your order is on its way' response 400 times a month. Now the AI handles all of that instantly, and my team actually has time to help customers who need real human attention. The product recommendation feature was a surprise bonus — customers are telling us they prefer chatting with the AI over using our site search. That says something."
Let’s bring your idea to life
Your innovative idea deserves a team that can bring it to life. Reach out to us today to discuss your project, and we’ll work with you every step of the way.