Andreas Welsch: 0:30

Today we'll talk about getting AI ready with reliable data, and who better to talk about it than someone who's actively working on that bar. Moses, Hey Barr. Thank you so much for joining.

Barr Moses: 0:40

Thanks for having me.

Andreas Welsch: 0:42

Wonderful. Why don't you tell our audience a little bit about yourself, who you are and what you do.

Barr Moses: 0:46

Yeah, happy to. As mentioned, my name is Barr Moses. I'm the CEO and co-founder of a company called Monte Carlo. Monte Carlo is the leader and creator of a category called Data and AI Observability. And what we do is we work with hundreds of enterprises ranging from companies like Cisco and Intuit to leading. Airlines, like American Airlines to leading retail and CPG like PepsiCo and many other organizations. And the thing that's common to all of them is they use data and AI to power their business to build better experiences for their customers to build better. Solutions for their customers, all powered by data, and AI. And what Monte Carlo does is partner with these companies to help them make sure that their data and AI products are actually reliable and can be trusted. The worst thing is when they're, when you look at you steer at a, an agent gives you a totally wrong answer that you totally did not expect. Or maybe you're looking at a report that's based totally on the wrong numbers. In those instances, you gotta. Lose trust and the, data and AI team's credibility is on the line. Their brand, their reputation is on the line. And so we help data and AI teams know about issues before they happen and be able to catch them before they have catastrophic impact on businesses.

Andreas Welsch: 2:10

That's wonderful. Thank you so much for, sharing that. And I think, when, we're looking at leaders being asked to go figure out AI, go explore agents. I think a lot of times that's just a disaster waiting to happen because you are not as informed maybe, or you are just doing this for the first time. So you think Yeah it's maybe as easy as some of the vendors advertising these platforms make it sound. But at the end of the day, it does come back to your own data and to your company's data. So really excited about the conversation with you today. Should we play a little game to kick things off? What do you think?

Barr Moses: 2:49

Sure. I'm not a very fun person. I have to warn you of that, but happy to see. I'm happy to participate. That's fine.

Andreas Welsch: 2:57

We'll see. So this one is called In your own Words, I'd like for you to answer with the first thing that comes to mind and why. And for you in the audience, I'd love to get your answer as well to make it a little more interesting. You only have 60 seconds for your answer. So are you ready for, What's the BUZZ?

Barr Moses: 3:15

Born ready.

Andreas Welsch: 3:16

Okay, good. So let's see. If AI were a book, what would it be?

Barr Moses: 3:24

Oh, great question.

Andreas Welsch: 3:25

60 seconds on the clock.

Barr Moses: 3:27

That is a great question. If AI were a book, what would it be? Trying to think of a good blockbuster. One of the books that I've read recently that somewhat related, but not entirely is a book by former Snowflake CEO. His name is Frank Slootman and the book is called Amp It Up. I was just remembered of that book. And the whole sort of premise of the book is around how can you amp it up and by that means drive up the intensity and the urgency as a culture. And I think if AI has done something, it's definitely amped it up for everyone. Everyone is there's an increased urgency, increased intensity, increased innovation cycles, increased products have, are changing on a weekly and monthly basis. New foundation models are being released left and right. And so everyone really has to amp it up. Everyone has a harder job. That's what come, that's what comes to mind.

Andreas Welsch: 4:34

I love that. Great answer. And indeed it's a lot about amping amping it up. I saw a poll yesterday by Axios where they have Americans across all different demographics and the sentiment was yeah, yes, it's great to have so much AI, but actually can we slow it down a little bit? Can we actually tone it down and be more deliberate? So it's interesting to see how these different demand dynamics play out. And I think a lot of people have also have have had access to generative AI tools and have been able to see this. And there's obviously been a lot of discourse in the media and so on. So interesting to see how that perception shifts as well. But definitely right, we need to act with urgency, but we also need to make sure that we act on, the right things. A couple weeks ago I was meeting with the C-suite leaders on an engineering company and, they shared their vision for how AI should help them operate the business better. And throughout that conversation we actually realized that they first need to have a data foundation in place which they don't have yet. It's actually not so much about an AI transformation, but more about a business transformation. We'd love to do win-loss analysis. Great. How are you capturing the data? In people's minds or on spreadsheets, maybe if we're lucky. So we need to ask people. Okay. So there are some fundamental things and I'm wondering what are you seeing in your discussions? Certainly the companies you mentioned are well underway on their AI and machine learning and Gen AI journey. But I'm sure you're meeting with others as well. So are engineering companies like the one that I met with, the only ones that are facing this or are others are going through similar challenges too?

Barr Moses: 6:19

Yeah, I think it's a great question. Maybe a reaction to that and also what you said prior. I think let's just start by facing, I think AI facing reality. I think the, pressure around AI is real. I. And that's happening everywhere in boardrooms and in the media and with your peer groups. We actually just did a, survey with a couple hundred data and AI leaders and asked them a number of questions about their approach to data foundations, AI, and the results were pretty striking. A hundred percent of leaders. Are currently have AI in production or are planning on having AI in production this year? So clearly everyone is feeling the hype and everyone is acting on it, right? I don't think that's new at all. However, only two, only one out of three respondents actually think their data is ready for AI. Now, in a world where we live, where a hundred percent of people are working on AI, but the large majority of people think that their data is not ready in AI, we're obviously faced with. A problem, if you will. Now I think what does being AI ready mean for your data? We can unpack what that means, but maybe just to double click into the implications here for many organizations, I was just talking to CTO of a Fortune 100 company, and basically he told me like, look, by the end of this year, I expect us to have over 500 agents. That out in the wild that we've built. And agents are not deterministic systems. And so I cannot for certainty predict what the output of a model can be. And so I might have 500 agents out in the wild sharing unpredictable outputs that I have no oversight on. And so that reality is a scary reality for people. Yes. And the the implications on revenue and brand and risk are real. Just to give you a couple examples, a couple years ago Citibank was actually hit with a several hundred million dollars for data quality issues. And so you can actually, there, there's severe regulatory risks for. For practices that are not strong enough in data quality that's been here for, a while. And that's not new in the last five to 10 years has been multiple recent issues describing catastrophic results and impact of the data being data AI platform being wrong. Fast forward to today. Just a couple of months ago a user actually was able to convince a chatbot, a Chevy Tahoe car to to, purchase that car for$1. An agent, basically, again, an agent sold a Chevy Tahoe car for$1 because the user was able to convince the chat bot to do that. And obviously you can imagine the repercussions. For, in that instance. And so organizations far and wide, on the one hand, need to invest a ton in AI or being asked, being tasked. And AI, on the other hand, a hundred percent with Trinity, I can tell you the foundations are not there. And the ability to deliver reliable data and AI products is becoming more important than ever in this world.

Andreas Welsch: 9:44

So I, was just at Data Summit in Boston about two weeks ago where lots of data leaders met from data management, data governance, many different roles, heads of analytics and so on. And they shared a similar thing, but they also shared the concern that I'm not getting the budget for it. My boss says, we have lots of budget that we can assign to AI projects. Can you come up with an idea for an AI project? But I know that the data isn't what it needs to be. It's not complete, it's not accurate, it's not fresh and what have you. What are you seeing there? How are leaders coping with that? That yes, on one hand there, there is budget, but management ask me to invest it in shiny things. That we can, again, report up and look at and say, we're doing AI, but we actually know that we need to fix the foundation before we can do something that is reliable, that will not expose us to a significant risk.

Barr Moses: 10:54

Yeah, it's a good question. I I think by and large from what we can see, most companies have budgets and are willing to invest in AI. A lot of it is experimental budgets or sort of innovation budgets. And so I think there's a question of what will be sticky and what will be here in 18 to 24 months. Sounds like that resonates. I think that's certainly time will tell, but I do I think it's hard. It's very rare that I come across companies that don't invest in AI. Now when I think about data and AI teams, there's primarily three core problems that I hear data and AI teams are faced with, and I think that influences their budget decision making. So I'll just walk through what these three core problems are. The first core problem is that they are being tasked just like every other team, just like every other function, they're being tasked with finding ways to accelerate the output and the productivity. Of their team with AI and automation. So every single team, engineering support, customer success data and AI teams, they're being asked to do more with AI. And so that means things that were workflows that we're already doing, things that we're already spending time on doing them faster and better using AI. That's like the first kind of core problem that they have. The second core problem that data and AI leaders face is. They are, they own a data platform and they own a lot of enterprise data, a lot of proprietary data. And that data is feeding AI products. Maybe it could be a chat bot, it could be an agent, whatever it is. I. And the data that they're providing to these AI solutions needs to be reliable. Now, why does this data even matter? Because when I'm building an AI product, everyone has access to the latest and greatest model. I can always switch between Anthropic and open AI and something else. We all have access to that with just a few clicks. And I have an API and I'm done. But the thing that I have that my competitor or another, company doesn't have is I have proprietary data. I have enterprise data, so I know I have more information and can actually tailor these products to offer a better customer experience. I'll give you an example. If you work with an airline company for example, or if you work with a hotel chain I can offer a recommendation product. For example, Hey maybe you wanna. Have this for lunch or have this experience at the hotel, I can offer a better solution or a more personalized concierge, if you will, because I know your preferences and I have prior information about what's your lunch preferences or whatnot. And so I can actually, I. Make more intelligence recommendations for you. If that data is unreliable, if I'm using the wrong data to feed those AI products, then obviously everything crumbles, right? So that's the second kind of core issue. Make my data AI ready. And a lot goes into that. We can go into more detail. There's you have to make sure that you have structured data is reliable, your unstructured data is reliable. That this whole thing is like a big thing, right? But, that's the, second core issue. Then the third core issue that I really see data and AI leaders, struggle with or find the need to divert attention and resources to is now that they've released AI products, how do we make sure that those are reliable? So how do we make sure that the agent is not selling the Chevy Tahoe car for$1? Or because you can have the perfect trainee data the perfect prompt, the perfect context, but the output of the model will still not be fit for pur for purpose. And again. Can go into more detail and so that, but at a very high level when I think about decision making for a company and for, budgets, these three things are honestly table stakes. I would say the first one, certainly like making data teams more productive. Yes. With AI and automation. Really table stakes. I think the second more and more table stakes, like getting the foundation ready, making, having AI ready data. I think the third category, building reliable AI solutions is probably an area that people are still in the early days of. We're still many companies are just moving to the cloud, right? They're still in that, on that journey and there's a lot that goes into actually delivering AI solutions. But at a high level, these are like the three, three big problems that the data and AI leaders have to tackle.

Andreas Welsch: 15:30

I think that makes it very tangible, right? And also shows this progression of how, should you think about this? Also in, in three simple steps now. You mentioned yes we, need to put more emphasis on, data. We need to empower data teams more. What are you seeing, how are leaders getting buy-in along the chain of command to do these data projects? Again, when everybody's asking what's our AI strategy?

Barr Moses: 15:53

Yeah, it's a good question. Look I think one of the things that is really important and obvious, but like easier said than done is tying what this is to. Tying what you're building to, to real value and to real impact and maybe even just saving, co finding cost savings along the way using AI. And by that lemme just give you like a practical example of how you can use AI to save time. At Monte Carlo, what I mentioned we offer observability for companies or help companies make sure that their data and AI products or their data and AI estate is reliable. And we also use AI ourselves. And so we actually built or we are building a number of agents. We're building an observa suite of observability agents. And the goal of this agent is basically to deliver real hard ROI for data and AI teams. And, I'll give you an example or kind of walk you through how these works both'cause I think it's super cool. But also because I think, I actually think it's valuable for data and AI team. One of the things that data teams spend a lot of time on, mostly data analysts spend a ton of time on, is trying to figure out what in their data needs to be monitored. So I have a lot of tables sometimes thousands, tens of thousands, hundreds of thousands of tables in my. Data lakehouse into my data warehouse. That could be at your GCP platform, Azure, Databricks, Snowflake, AWS, what have you. And if I'm a data analyst or a data engineer, I need to make sure that data is accurate, but I have no idea. I. What are their requirements? It's hard for me to be able to specify at the table or even feel level to know exactly what needs to be accurate, what the, definition of accuracy even means. And so oftentimes what data teams do is they spend time rigorously and tediously, first of all, profiling the data. So like understanding the structure of the data and understanding the data itself. Digging through that, looking through metadata, making connections to understand the semantic meaning of fields. And then coming up with specific monitors that might say, oh, you know this, I'm just making this up. Again. If this is like a, an airline the, and I have a column that has number of every row is a flight. And so I need to make sure that the flight numbers have a certain sort of they need to look a certain way, they need to have a certain number of characters, et cetera. So I need to have a monitor to make sure that that data is always accurate.'cause I can't mess up flight numbers, right? Yes. Like flight numbers, data need to be accurate. And so that process is really, hard and tedious and manual to go through all of that and come up with monitors. What we have done is actually released a monitoring agent. It's in production already. It's used by hundreds of customers, so it's been out live and for, a while. We actually have a 60% acceptance rate, which means that 60% of the monitors that we recommend are being used and get accepted. And what we do is actually, this agent actually weird to say this, but it mimics the human behavior. It goes through, it profiles the data, it tries to make this the understand the meaning of different connections between the data and then make recommendations for what you need to monitor. And so that cuts down for a data team time from like weeks to minutes if you will. And so it's really cool to start seeing what you can actually start offering to data and AI teams to make them more productive. And so what we're building and we're about to release in the next few weeks. And this is what I think is a really cool application of LLMs has to do with troubleshooting. So the first thing that data and AI teams do is they monitor data, right? They need to know when the data's wrong. The second thing that they do is when the data is wrong, they need to start. They have a workflow for understanding why it's wrong, starting to triage and troubleshoot and try to understand what was the root cause. And so what our, troubleshooting agent does is again, mimics what a data gov data steward would do. What a data steward data analyst would typically do is they would start with coming up with hypothesis. For what might be wrong. So let's say I get a notification that there's a particular issue with a particular report. And this report is being used every single day by our field operations team. It's high visibility. It's really important that I. Look into this issue. It's, data. It's a report that's being used all the time, and then I start coming up with hypothesis for what could go wrong. And I start going upstream table by table and saying, okay, let me check if the data arrived on time. Okay, let me check if this upstream data arrived on time. Okay, now let me check that anyone make a change to the code. Somewhere that anyone break something. Okay. Maybe the ETL system, maybe the the job failed and then incomplete. I basically had to come up with a list of dozens of hypothesis and start to check them and test and cover what happened. What we've done in, this troubleshooting agent is we basically have an ensemble of LLMs. There's an l there's this master LLM that comes up with the list of hypothesis, and then it spawns off a new agent for every single hypothesis. And so every agent then basically recursively looks into a particular hypothesis, and so you can have up to, I think around a hundred or so agents running in less than two minutes. All looking into different hypothesis for what could go wrong at the same time. And so something that could have taken me years frankly, to go through all of this hypothesis and try to understand what goes wrong with a troubleshooting agent, again, with a combination of. Breaking down the problem into different tasks. Every task is a hypothesis to look into. And then using a lot of LLMs to look into each of this hypothesis and then basically come back and report back on what they found. And then this sort of master, there's a, there's sort of a. Bigger LLM that kind of summarizes all that and gives you like A-T-L-D-R, here's the root cause what happened this is the reason for this incident is X, Y, Z. And so that is really powerful. So you're basically taking, again, things that data and AI teams are already doing, like they're already spending time on that basically cutting the amount of time significantly. And so the ROI in those instances is very clear, if that makes sense.

Andreas Welsch: 22:35

Great example. And very tangible too. I love these examples where it's about, hey, it would have taken one person or a team of people months or years to do this. Now we can do this in minutes or even less than a couple minutes. So super powerful, and I think we need more of these examples that show what can this actually do. Beyond the hype talk and totally these kind of things. So sounds like you are already working on some exciting things and are testing this with customers or rolling this out to customers. Super, super cool. Now, it's been a few weeks. I think I would say it's probably even been a few months since we've started talking and setting up today's session. And I remember probably earlier in the year I came across an article from you where it said Agentic AI might actually be doomed to fail. And we, somehow landed on this topic of getting data AI ready. But I'm wondering why do you think AI agents might be doomed to fail? When are they doomed to fail? It's probably the better question to ask. When are they doomed

Barr Moses: 23:42

to fail? Yeah, good question. Look we talked about this earlier on this call when folks are releasing hundreds of agents into the wild and like waiting to see what happens. I think we know what happens. I think we know what the story looks like here. And I love this example of sort of the troubleshooting agent because I just wanna stay, I just wanna say when obviously when our team presented to me, shared we were working on the troubleshooting agent. I think that was like a turning point for me because I think, there's a lot of AI skepticism out there and there's a lot of question of what does it actually mean? And just to be clear I don't think AI will replace people or replace data and AI teams. If anything if I look at a corollary like engineering, as engineering as a space, the more advancements we've had with engineers the higher there was a demand for engineering teams. And I think the same will be for data and AI teams. And the more we use AI, the more we need data and data, there's gonna be an increase in demand. It doesn't mean that we don't need to change how we work and, what we do. And I'm, I very much believe in that. And also I think this application of the troubleshooting agent really brought to life for me What. How powerful if you put LLMs to use, how powerful that could be. And so you can imagine a lot of complex problems that you might be working on today that if we can intelligently break them into smaller tasks and use different, smaller and larger LLMs in different instances, you could actually teach LLMs to do really, clever things. And so I wanna start I, think it's important me to say that because I think. I think figuring out what agents can do for your business is important. And I do believe, I am convinced today that there is value in that for your company, for your business. And I think you have to see it and to experience it yourself, to believe it. It's very different when it's in theory. So first of all I and, there's a lot of talk about like, how in, in a couple years, software engineers will really just be managing a fleet of agents. Like you'll have one agent to build features, one agent to fix your bugs, one agent to do PRDs with you basically have a fleet of agents. And so similarly you can envision that in the world, data and AI teams have like a fleet of agents one agent to monitor their data, one agent to troubleshoot their, and we'll all just sit here in our cushy chair and monitor all the agents. I can't wait for that. I obviously like that means that our world will, be very different and how we work will be very different, but, in that world, or the reason why I wrote that, that I think there's a high chance that AgTech solutions will fail is if we, are not thoughtful about what it means to build high trust agents. And I'll give you an example, the the most like popular example, if you will think this went viral on XA couple years ago. Or, maybe a couple last year, I think someone wrote on, on one of sort of Google's LLM solutions. What should I do if cheese is slipping off my pizza? Yes, you may have seen this and the answer was, oh, you should just use organic super glue. No problem. To like glue the cheese back onto your pizza. Now, just to be clear, like the training data was good. The like prompt was good. The con, everything was good, right? I. But the output was totally inappropriate and made no sense for the context. Now I will continue to use Google or maybe use perplexity or something like, but Google can get away with it. We can't get away with that. Most organizations in the world can't get away with building agents that will have, that will share responses like that. That are really inappropriate. And, continue to have trust. And so I think the way that we need to manage our agents and the way that we need to manage our data and AI platform needs to change drastically. And one of the things that we've done at Monte Carlo over the last several years is work with thousands of enterprise and help figure out. What is the way to build reliable data and AI products and based on, again, thousands of, customers and conversations, we've found that problems with data and AI products can really be reduced to four core problems. The first is data and AI solutions can be running on bad data, meaning the data that you're feeding, it might be inaccurate or late or just wrong. The second core reason for why data and AI systems might be wrong is because there's a code change. Maybe it's a bad join maybe it's a schema change. Maybe it's a code change in the agent itself that's actually changed something dramatically. So the third reason for why things can go wrong is if there's a problem with the system. A system that could be like your ETL jobs, like airflow or DBT, or it could be orchestrators, it could be things like land chain or land graph. Basically anything along your data and AI say can go wrong in your system.'cause the one thing for sure, a hundred percent of systems fail at some point. And then the fourth thing that can go wrong, and I alluded to this earlier, is. You can have all those things work perfectly. You can have the perfect context, the perfect prompt, but still the map model output will be not fit for purpose. And so I think for people who are building agents who are building AI solutions, unless we start thinking about the holistic health of those systems overall as a function of those four things, data code systems and model output, we will fail.

Andreas Welsch: 30:00

Wow, that's that really hits home and I think comes down to the core of it really. Barr, thank you so much for sharing this. We've covered a lot of ground in the last 30 minutes. From helping data teams empower them and understand that they have big responsibility in what they can do to observability, making sure that our data is actually accurate and correct. And then talk about some of the risks, what happens when you don't do that, and where the failure points in, Agentic AI. Also, great to hear what you're doing at Monte Carlo and how you're bringing AI and agents to data and AI teams. Sounds like a really exciting opportunity and really exciting space too.

Barr Moses: 30:39

So yeah. Thanks.

Andreas Welsch: 30:42

Yeah, wonderful. Thank you so much for joining us and for sharing your experience with us today.

Barr Moses: 30:46

Absolutely. Thank you for having me. It's an exciting time. There's a lot of exciting stuff happening, I think the future is bright. I'm excited to be part of it. And thank you to everyone joining.