The AI Feedback loop - Learning from Tesla and Shopify
A simple framework for turning one-off prompts into a reusable system that improves across your business
Last week we explored how to setup feedback systems to help improve your AI prompting. You can view the article here.
This week, I want to show how the operational reality of keeping a feedback loop alive. I thought I would share different case studies of businesses to show what they are doing, starting with Tesla.
This diagram shows how when a Tesla drives, if they detect an inaccuracy, it gets written as a unit test and then passed into their feedback loop.
The key thing here is that this system is improving itself. That’s the brilliant thing about AI systems!
This image makes it a bit easier to see.
This is a fascinating talk that shows how they do this at scale.
The brilliant thing about this process is that it starts to build a data flywheel. The more data you get, the faster it improves.
This article deep dives on the AI driving data flywheel.
Shopify
I think Shopify is fascinating and that most people reading this can learn from their business. They are quite open about their AI process.
https://www.bvp.com/atlas/inside-shopifys-ai-first-engineering-playbook
The main thing that they talk about is how engineers are now shifting from writing code to acting as orchestrators, guiding AI systems and evaluating their output, with a central internal gateway routing all AI requests.
To do so, they do this using an agent swarm.
If you prefer a podcast then here is the CTO speaking about their transition.
Interestingly, for them pull request volume, test failures and deployment rollback are becoming the real bottlenecks in the agent era. Hint hint, if those aren’t your bottlenecks right now, then you probably aren’t maximising your AI capability.
Listen here.
One last article to share, but my most favourite of the bunch, this is about their engineering practices and their RFP agent that they made.
“I’m seeing a concerning trend in talking to CTOs and CEOs about the cost of tokens,” says Thawar. “They’re thinking, ‘Can I afford an extra $1,000 - $10,000 per month per engineer as they use tools like Cursor, Windsurf, GitHub Copilot, and more?’ So they clamp down on spending.”
This mindset is at odds with the goal of driving AI adoption. “If your engineers are spending $1,000 per month more because of LLMs and they are 10% more productive, that’s too cheap. Anyone would kill for a 10% increase in productivity for only $1,000 per month.” (In fact, if your engineers are spending $10,000 per month and getting value, Thawar wants you to DM him so he can learn what you’re doing.)
https://www.firstround.com/ai/shopify
They talk specifically about how their RFP agent learns from previous answers that led to RFP wins, with the resulting RFP going back into the repository to make future responses better. This is as close as you can get to a textbook business-side feedback loop.
Let’s recap
The problem currently with most organisations is this. Let’s think about it at a prompt level as it is easier to visualise.
What we are trying to do is to add in a feedback loop.
An AI system as a whole follows this process, as we have seen with Tesla and Shopify. We need to start somewhere small when thinking about this and so I thought I’d make a flowchart to help you think about tasks.
But how do we make this work in practice? I’ll show you what it looks like in SharePoint. This is the simplest way to get started.
Your system will look like this, let’s step through it.
AI Prompt Library/
│
├── 00 - Read me first/
│ ├── How the library works.docx
│ └── Prompt template.dotx
│
├── 01 - Client delivery/
│ ├── Weekly client reporting/
│ ├── Proposal drafting/
│ └── Meeting notes to actions/
│
├── 02 - Hiring/
├── 03 - Marketing/
└── 04 - Internal operations/This structures allows you to make a start really easily and gives you a clear plan and a across the business for all your prompts that you use. No matter what business you are in, you can benefit from a system like this.
You make a folder for each area of the business.
And then in each folder you have the different tasks that you are going to perform.
For example in client delivery, you have weekly client reporting, proposal drafting, or meeting notes to action.
I’ve built my version of this as a standalone website, which I find helpful. It mirrors the same structure, but allows me to interface with this more easily through code.
Now what do we store in each of these folders?
If you go back to last week’s article, you put in different prompts for the different tasks you perform. So for example the below is a weekly client status update that your might store in your prompt library.
System prompt: Weekly client status update
You are a project manager writing a weekly status update for [CLIENT NAME].
Format:
Lead with progress and wins from this week
Then cover what's coming next week
Flag risks or blockers at the end, framed as "things we're watching" rather than alarms
Keep it under 400 words
Use bullet points sparingly. Short paragraphs work better for this client
Tone:
Professional but warm. This client values directness and doesn't like corporate fluff
Write like you're updating a trusted colleague, not presenting to a board
Avoid jargon unless it's terminology the client uses themselves
What the client cares about:
Timeline. Are we on track?
Budget. Any surprises coming?
Decisions. Do they need to do anything before next week?
What the client doesn't care about:
Internal team logistics
Technical implementation details unless they directly affect timeline or cost
Examples of updates they responded well to:
[Paste 2-3 previous updates that got a good reaction. Even a "thanks, this is great" email is a signal worth capturing.]
Feedback log:
[This is the part that matters. Every time you edit the AI's output, write down what you changed and why. This section grows over time.]
[DATE]: Client didn't like when we led with a delay. Moved progress to the top, delay to the risks section. Much better response.
[DATE]: Cut the technical paragraph about API migration. Client replied asking what half of it meant. Keep it outcome-focused.
[DATE]: Added a "decisions needed" section. Client said this was really helpful and started replying faster.Remember at the bottom to add feedback on the prompt and how they have performed.
If your team is more advanced, you can build this structure out in Github, I have an example repo here that I use for coding.
It’s the same concept, you have different prompts in each folder.
While we aren’t building self driving AI cars with this system, we are practising the process of centralising knowledge and improving what we do have.
The benefits of this approach
You don’t have to reinvent the wheel every time you prompt
You can build up your own IP
You can build your feedback loops and improve the performance of everything. Again you can’t do this when you build every prompt from scratch every time.
If you have multiple people working, it really helps to all get on the same page.
Your average prompt is much better when you have pre-thought about everything as opposed to relying on your to prompt in the moment when you are rushing.
That’s it. Your homework this week is to make a start with this! Even if you add 3 prompts then you are starting the process of actually keeping a feedback loop alive with AI.
Knowledge sharing like this is a really powerful way to get better as a team and you will find that you make rapid improvement when you are all engaged and trying to improve the prompts and the process together.
The cool thing is that no matter what AI system you have, this is the foundational layer that you need to get used to. Using feedback to improve the system. So even though it might feel like a baby step relative to self-driving cars, you are still learning and leveraging these new AI systems for what they are good for.
Book Club
This book is an amazing insight into the origin story of Tesla. The amount of hurdles they have had to overcome is incredible and should give you motivation if you are struggling with your own business or company.
I’m an Amazon affiliate, I get a small commission if you make a purchase.
