Data Flywheel in agents

UpdatedMay 2, 2025

Hi, I'm Vikrant, a passionate software developer with a strong belief in the power of teamwork, empathy, and getting things done. With a background in building scalable and efficient backend systems, I've had the privilege of working with a range of technologies that excite me - from Express.js, Flask, and Django to React, PostGres, and MongoDB Atlas. My experience with Azure has given me a solid understanding of cloud infrastructure, and I've had a blast building and deploying applications that make a real impact. But what really gets me going is exploring the frontiers of AI and machine learning. I've had the opportunity to work on some amazing projects, including building advanced RAG applications, fine-tuning models like Phi2 on custom data, and even dabbling in web3 and Ethereum. For me, it's not just about writing code - it's about understanding the people and problems I'm trying to solve. I believe that empathy is the unsung hero of software development, and I strive to bring a human touch to everything I do. Whether it's collaborating with colleagues, communicating with clients, or simply trying to make sense of complex technical concepts, I'm always looking for ways to make technology more accessible and more meaningful. If you're looking for a team player who is passionate about building innovative solutions, let's connect! I'm always up for a chat about the latest tech trends, or just about life in general.

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

Spending money on GPU's was my best investment

The best investment I made wasn't in stocks, crypto, or day trading.It was spending money on GPUs to learn how LLMs work Have you ever tried deploying an model yourself?Not just used an API. I mea

Jun 8, 20263 min read174

Spending money on GPU's was my best investment

AI has agency where is yours?

Claude code is going of the roofs with opus 4.6 AI coding has reached a stage at which where you can just prompt away what you want and it does build a really good solution for you, all you need is a

Mar 11, 20265 min read23

AI IDEs and agents I used this year

I started with Cursor as my first AI IDE ,used it for a year For the last 3-4 months switched to Claude Code but I am back to Cursor now. Why I switched to Claude Code?Cursor had problems working with large react components it failed miserably when...

Nov 10, 20252 min read9

Improving LLM Workflow Evaluation: How to Sidestep Common Mistakes

Improving AI Workflow Precision Using Tailored Validations and Scoring Methods

Jun 22, 20255 min read99

Improving LLM Workflow Evaluation: How to Sidestep Common Mistakes

2 Simple tips for making your Langgraph agent Production ready

LangGraph is a great framework that allows you to develop agents while keeping you in control of prompts and routing. If you are done building your agent and running it in Python Notebooks and want to put in production there are somethings that you n...

Jun 10, 20253 min read57

2 Simple tips for making your Langgraph agent Production ready

GuruGen

23 posts

GuruGen: Your Personal AI & Tech Advantage.

Data flywheels are very important if you are building agents in production. If you have live users using your agent having a proper data flywheel in place is really crucial.

What is a data flywheel?

Data flywheel is a self-reinforcing loop that uses the data that you have gathered via logs or user feedback to continuously improve and adapt the product for better usability and accuracy.

Its basically a feedback loop that you implement in your product development lifecycle. It is mostly used in live products. Its a important facet that you have to consider to keep the product error free and easy to use for the users.

image credits - Jason Liu

Data Flywheels in AI Agents

AI agent is just classical piece of software paired with token predicting language models called LLM’s.
Though mostly the ideas and concepts used while building the agents are similar to the traditional softwares, there are some major differences, the classical software is mostly Deterministic on the other hand software powered by AI is non-deterministic.
We need data flywheels for these agents too to keep them error free, and as easy as possible to use for the users with lesser friction.
For any data flywheel to provide us value, the most important thing we need is data.

Data points to use

In agents there few sources that you can utilize to collect this data for your flywheel, here some examples:

Tracing Logs:

If you are using something like langsmith or langfuse to trace your llm calls and chain , data at each step is captured on these platforms. You can use this data to better understand whats going under the hood and how do different models perform for different nodes or use cases. It also allows you to look at the time it took for each call.
Failed Chats:

Sometimes the agent might fail at serving some user query and might cause your agent to break , in such case that request is failed. You can look at the traces and logs to understand what happened at each step , each llm call and each parsing that happened.
User feedback in the chatbot:

Its always better to have some kind of a feedback mechanism inside the UI of your agent to capture if the response by the agent was good or not. This is a valuable marker for use that is organic and true with the user.
User patterns:

You might observer some patterns that user are following with your agent , maybe you need to see what tasks users are performing frequently and see how we can improve that task or that capability of the agent.
Direct user feedback:

The most valuable data point that you would ever get is live feedback from the user, be it on online meetings or some meetups. You can have a genuine conversation with user and get important feedback.

How do they help you

By observing the tracing logs you can save the cost of each interaction with the agent by choosing the right LLM to use at each step. You can also see what parsing techniques are performing better with XML or JSON, also seeing the tool calls traces you will be able to understand if you have written the tools in way that is comprehensible to the smart LLM’s.
Failed chats allows you to find the loop holes in the agent, maybe some queries are causing recursion and eventually crash the agent. Also by referencing and analyzing the chats where users gave good feedback , you can extract good prompts for the LLM’s and some user approved response formats with relevant examples in the prompt.

Better Agents

After seeing all of this data , your beliefs might shatter , the feature that you thought was so important maybe isn’t, the users want something else, maybe the core reason why people are using the agent is totally different from what you intended it to. Facing the harsh truth and seeing the loopholes and failed chats you will be informed with the holes that need cementing. By prioritizing these issues and fixing them will make your agents flow smoothly through each node and solve real user problems.

P.S

This was a very high level overview and short article on the topic I’am thinking of writing on this topic in more detail. I hope you enjoyed it , if you have any feedback let me know ,Peace, GG!

Command Palette