Data-Driven Leader Series - Episode 2 - Digital Turbine

Welcome to our Data-Driven Leaders Series. Rill's customers are data driven leaders coming from a variety of industries from ecommerce to technology. One thing they all have in common is a passion for understanding data and gaining insights. 

Episode 1 featured Kat Tomlin, Director of Business Operations at Disco.

In our second episode, we interview Glenn Goldman, Director of Product Management at Digital Turbine. Digital Turbine is the leading independent mobile growth platform – leveling up the landscape for advertisers, publishers & OEMs.

Glenn and I dive deep into the following topics.

• Glenn’s journey from music to marketing and now leading all of the data product initiatives 
• The need for frameworks in data-driven organizations - CAMA (collect, aggregate, model, action)
• Old world vs new world analytics with specific examples of how Rill helps Digital Turbine achieve speed and efficiency

I’ve noted some of my favorite highlights below:

We are part of the mobile ad ecosystem. So a lot of people think, “well, you're just putting ads, you're putting apps on phones, you're putting ads into the ecosystem”. But I think it's so much more than that because you have to understand the intent behind what a person wants when they engage with a mobile application. (13:40)

We had lots of different tools that allowed us to build dashboards with colors and different pretty graphs and configurations and all that other stuff. What those tools did is they allowed us to answer questions that we asked, which is important. What Rill allows us to do is to answer the questions we haven't asked yet. It does that by surfacing information in a way that is immediately digestible and actionable. (32:45)

I can't tell you how many times that would have been a week, 2 weeks, maybe even 3 weeks of an investigation. We were able to resolve it in 1 hour and that is time, that is money, that is speed. So just one example of how Rill has really changed the way we operate. (35:55)

The immediacy and access to data across a number of different dimensions and measures allows you to really understand what's happening in your system. I almost imagine being in a nuclear facility with all the buttons and levers. You need to be able to see all of the buttons and levers to be able to know if something is off or something is wrong. (39:15)

The most important thing is that not everybody wants to look at a dashboard. Not everybody wants to have to sit there and click, click, click, click, click. Ultimately, people know what they're after and anything that gets in the way of getting to that answer often interrupts the entire creative process of solving that problem, which is the beauty of chatbots and LLMs. (48:22)

Check out the video interview and full transcript below.

Michael Driscoll: 

Good afternoon and good morning everyone. I am delighted to introduce our next guest on the Data-Driven Leader Series. I am joined by Glenn Goldman, Director of Product Management at Digital Turbine, and one of the world's most active users of our product. Glenn, welcome to Data-Driven Leaders. 

Glenn Goldman:

Thank you, Mike. It's a pleasure to be with you today. 

Overview of Digital Turbine (00:25)

Michael Driscoll:

Before we get into your journey and what got you into media in the first place, I thought we would start by talking about Digital Turbine, its business, the types of advertisers you work with, the platform, and then we can talk a little bit about your remit inside the publicly traded Digital Turbine empire. 

Glenn Goldman:

I'm happy to give an overview of the company. For folks that don't know Digital Turbine, we work with a variety of mobile players, basically anybody in the mobile ecosystem. From the telecommunications companies, the mobile carriers like Verizon, to the folks that are making handsets, OEMs, hardware manufacturers like Samsung and Motorola, to the folks that are making apps themselves, the app developers, agencies and brands. So basically anybody who wants access to the mobile ecosystem, we work with them. And what we do is we connect them directly to the device end user. That is our remit. 

Our goal in life is to bring everybody immediacy and access to devices and to end users, and we do that through a number of different solutions in the entire growth life cycle. I would start with the user acquisition side, and on the UA side, we have our own way of connecting users through in-app advertising, using a really compelling technology, proprietary technology that we call SingleTap, which allows folks to access and interact with users without having to go through the traditional storefronts. So we do direct delivery of mobile applications to users bypassing the kind of chaotic and noisy storefronts where people often get sucked into what Google or Apple want them to do. We engage users with really compelling creative, video, playables, etc. We engage people with native tools on the device itself, like stock notifications where we can provide compelling promotional content, both for brands and for mobile advertisers. Then on the monetization side through our BrandSafe SDK, we can allow app developers to monetize their programmatic supply. So really if you're in the mobile space, we're working with you. We're looking to either get you access to customers or to help you monetize. 

Glenn’s career journey - music -> marketing -> product (03:29)

Michael Driscoll: 

I know you've been at a Digital Turbine for almost a decade. I think you've got your 10-year milestone coming up, which, I think we all sometimes find it hard to believe that we've been anywhere for a full decade. But I know that you were in the music business before joining Digital Turbine. Maybe just share a little bit for our audience about how you went from someone who had studied music and music production, once a musician, always a musician. Tell us a bit about your journey and how you came to be leading a product management team at Digital Turbine. 

Glenn Goldman: 

Yeah I'd love to get into it. I would say that my journey has always been about connecting the creative and technology. That's really where I've always lived is between those two spaces. So I was born and bred a musician, play a number of different instruments, and I've always had a love of all the different styles of music, and I really wanted to be a part of that for a long time. So whether it was playing music or curating musical experiences for people where I was a DJ for many years both in college and then out in California. Then starting to work through how I can activate and monetize that in a way that it was scalable because as most people know, DJ’ing house parties is not a career, although it is a favorite pastime of mine. So really what I started to do was to find ways to take that love of the creative process and to scale it up for other people. I started working in the music industry out in California in San Francisco in the digital music distribution space and the marketing space. This is right around the time where social media was starting to take off. So it offered a new channel for us to work with independent artists and labels to help them promote their content on these really organic channels that were just coming up. This was before sponsored paid ads and we were doing a lot of growth hacking and trying to get into that space. 

Michael Driscoll:

Was MySpace a part of things at that point? 

Glenn Goldman:

Oh yeah, absolutely. It was an interesting time, it was an interesting space, but one of the things that I loved about being in San Francisco at that time was getting access to technology in a way that was going to accelerate music's growth. So we all know that with iTunes and digital distribution, things were starting to explode, starting with Napster, of course. And then, the tools that allowed people to pirate music, which wasn't my favorite part. But still being able to connect into those technologies for promotion and for distribution. That's kind of where I realized that there was this synergy between the creative process and the kind of technologist side of things. 

So I actually built my first product out there, a licensing platform that would allow content creators, whether they were artists or in some cases we were working with people that would do parody songs or even sound effects. We had all sorts of different indie labels that we worked with to monetize their content through a self-service licensing platform. If you're at all familiar with…you hear a song in a movie or in a game, you're like how did that song get there, right? I wonder who did that. Well, typically it's a set of gatekeepers, like highly paid music supervisors, and they have access to the best music in the world, but they are coveted for their ear and for their access. And if you're an indie artist starting to come up, you don't really have access to those music shops because it's a really high barrier of entry, you have to get all this validation to be able to break into that. We wanted to cut that down, especially in San Francisco, where you had this burgeoning app developer space, you had people building all these games in basements or gaming studios starting to come out. We just saw this need and this opportunity to connect hundreds of indie artists with tens of thousands, or I should say indie labels with tens of thousands of different artists and hundreds of thousands if not millions of songs to this new marketplace of app developers. So we had everything precleared, we built out a platform. 

That's kind of where I learned about the marketing funnel and the checkout flows and how to tap into customer experience and all those other things. It was a really eye opening experience for me, but the thing also to this conversation that I took away from it was really understanding the data behind the experience. I would say that that's the first opportunity that I had to really understand that every transaction is not a data point, it's a person. And I've carried that with me throughout my career, especially as I've moved on to other companies working with millions of artists and SaaS products and now at Digital Turbine. It's really been critical for me to always evaluate the world in terms of people and what they're trying to do, not just the data point, because when you lose sight of the user and the customer experience and their intents and their motives and their personas, you effectively just start to optimize to a number. That's where your KPIs go down and business goes down. So, that was an important lesson for me. 

Day in a life of a data-driven leader (10:00)

Michael Driscoll: 

In the world of data analytics, I think one of the numbers that everyone cares about in advertising is things reach, unique reach and incremental reach. We have so many proxies for people, but, of course many of these can be manipulated and sometimes inaccurate, and so the goal is to reach people, not to reach devices, not to reach pages or sites. Ultimately there's a person that we're trying to connect with in some way and expose a new technology, new applications to, but I strongly agree it can get lost sometimes. We can forget about the people in this process and suddenly we're just looking at data points, not recognizing they're really a means to an end, not an end in themselves. Maybe just drill down and get very specific. Tell us a little bit about your day to day role and we'll talk a bit about some of the data tools that you're using in that role. Tell us a bit about a day in the life of Glenn. When you wake up in the morning, what sort of data tools are you working with? What sort of analytics, insights, questions are you asking in your day to day operational activities with your team? 

Glenn Goldman:

So first, I'll give everybody a little bit of an understanding of my purview where I focus, and then we can talk a little bit about the day to day. I lead all of our data product initiatives and what that essentially means is if it has to do with device, customer identity, attribution, if it has to do with experimentation, audience extrapolation, audience targeting, and audience optimization, all of those areas kind of fit into my world. It's a really interesting space, especially at a company like Digital Turbine. Every company has to solve these problems, but one of the things that makes Digital Turbine unique is that we have really assembled key strategic pieces over the course of a number of years to build this platform, this network. We've acquired a number of companies and it's become really important for us to find the right ways of connecting all of the data points to extract value through all of those channels. So it's a really tremendous opportunity for me and something that I love and I'm honored to be a part of. 

So day to day. I mean after my second pour over because I'm a big coffee drinker in the morning, that's how I kind of get things going. I wake up every morning thinking to myself, how can I connect a real person to an experience that's going to be valuable? Again, I go back to that because I think I mentioned earlier Digital Turbine is this platform where we work with all these different players. We are part of the mobile ad ecosystem. So a lot of people think, “well, you're just putting ads, you're putting apps on phones, you're putting ads into the ecosystem”. But I think it's so much more than that because you have to understand the intent behind what a person wants when they engage with a mobile application. 

The traditional way of thinking about it would be to say that when I'm playing Candy Crush or whatever the game may be some Match3 game that I'm going to go in there, and maybe an ad will float by me and I will hate it, and then I will try and X out of it as quickly as possible. Our view of the world is that if you're in this space, what is the context you are in that can allow you to enrich your experience without detracting from your current experience? That's one of the reasons why we built our SingleTap technology. When you engage with an ad within a mobile application, and you click on that ad, if you happen to find it interesting, you're off to the Google Play Store where you're overwhelmed with all of these other recommendations and things that take you out of that experience. But with SingleTap, we allow a user to stay in their context, to continue to engage with that game and deliver the application in the background. Then we provide a helpful notification that sits in their tray so that they can come back and activate that game when they're ready to. And that's part of the thinking that goes into all the products that we develop and the experiences that we develop. So I think that's a really important part of how I view the world. 

Identifying key success metrics (15:19)

Michael Driscoll:

When you evaluate the efficacy and the performance of SingleTap it sounds like a big differentiator for Digital Turbine. I think a lot of folks in the world of digital advertising have an always-on sort of global digital nervous system with signals flying everywhere, and truly the world never sleeps. I'm sure when you wake up in the morning, there's already been hours of activity that you're gonna look and see how the platform is performing, how are our partners doing. What are some of the metrics that you look at to evaluate, and maybe we can talk broadly and then even specifically with SingleTap? What are some of those metrics that you look at as signs that things are going well or warnings that maybe something needs to be adjusted? Give us a sense of how you evaluate in a data-driven way and successfully monitor the performance of the platform.

Glenn Goldman: 

Probably similar to many other product managers, I always break down my success metrics into three layers. I've got my business success metrics, product success metrics, and observability success metrics. I have to make sure that I'm looking at all three of those at different layers, and we use different tools to look at all those different ones. 

On the business side, it's probably a couple of the more traditional metrics like we are looking at, things like gross revenue, interactions, and the revenue that we can generate per user. But for me, more important is actually going back to what I mentioned earlier, those user experience and user engagement metrics. Because at the end of the day what we need to do is ensure that when people interact with the experiences, that they walk away with a positive experience, not a negative experience. Otherwise, they're not gonna come back. At the end of the day, we know that we have a massive footprint and an opportunity to engage with customers in lots of different areas. Yes, it's an app, but it's also through these native experiences on their device, whether it's through those notifications, or what we call our Beachfront property, which is our alternative storefronts, or media applications. So all of these different opportunities are areas for us to demonstrate value. 

We look at monthly active users, which is more a more traditional MAUs, DAUs, and WAUs. We look at those metrics because we want to ensure that when people engage, they're coming back and that they have a positive experience. We look at the traditional funnel metrics, like the number of people that start in those ads that ultimately click through and your ad tech KPIs of impressions down to clicks, down to installs. Then I would also say that a big part of what we care about is making sure that there's recurring retention and engagement for our advertisers. We all want to speak the language of our mobile advertisers and what they care about at the end of the day is that customers are engaging with their products on a recurring basis. LTV, ROAS, retention, and so any of our customers' metrics are our metrics. 

Lastly I would say the third layer of that is those observability metrics. Where we're looking at API response time and throughput, just to make sure that we have a really healthy system that's able to keep up with all the demand and the traffic that we're running. So we have lots of different tools that we use at different layers of that. Obviously Rill is an important part of this on our business metrics and our product health metrics. Especially on our product health metrics where we have access to really an unlimited number of measures and dimensions that they give us that kind of slice and dice capability. 

CAMA Framework (19:52)

Michael Driscoll: 

We’ll definitely drill into how Digital Turbine has been using Rill, but maybe before we do that, I'd love to hear what are some of the other tools that you use across in this trifecta of business, product and observability metrics. Just for folks who are listening, what are some other great data tools that you find to be valuable as you look at these different kinds of metrics for both internal and customer facing?

Glenn Goldman: 

Before I talk about tools, I always want to talk about framework because for me the tools are a means to an end. What matters is how we think about the problem. When we started to frame up how we wanted our data platform to work, I wanted to ensure that we had a consistent way of communicating that to everybody in the organization. So we came up with a framework, which I call the CAMA Framework and it stands for collect, aggregate, model, and then of course action. Actually, in deciding on this framework, I was really inspired by something I read. I think it might have been a Gladwell book if I'm not mistaken, but it was a US Air Force pilot named John Boyd. The OODA loop which stands for observe, orient, decide, and act. So the context for folks that may not know about the OODA loop is, Boyd came in as a tactician. He was this celebrated fighter pilot, and they wanted to extract his experience and his learnings to these fleets of newcomers. What he was able to do is to break down why he was successful in a way that was tangible and practical for people, which was to understand that every decision he made didn't come out of nowhere. It was a series of steps done in almost milliseconds that led to the next good decision. And so as you said, observe, orient, decide, act, and it was kind of a way to think about and process how to make complex decisions in a short amount of time. For me that really became an anchor of what I wanted to do as well, cause it wasn't so much about the immediacy, but it was about the loop. Every piece of data that we collect, every interaction that a customer has with our platform should provide value into the next interaction. It's the core 10 of advertising. 

For us we have this different loop, it's called the CAMA Loop. It starts with Collect, where we have access to billions of mobile devices around the world globally. We do that through both our APK and our BrandSafe SDKs that are integrated into app developers' phones. We capture data through those technologies, into our data lake using all sorts of streaming technologies, like Kafka and Spark to move data from from point A to point B. 

We use this next stage of the loop, we have Aggregate. As I mentioned earlier, when you are a massive company where you have these different assets that weren't necessarily built at the same time or by the same leadership group we've acquired, really some strategic parts. Being able to connect all those pipes in a way that was efficient, fast, and not duplicative, we needed to find the right technologies to do that, and that's where Databricks came in and really powered this kind of single pane of glass with what they call Unity Catalog. It's a wonderful tool and it gives us a single queryable layer across all of our data sets, regardless of whether they're in the Databricks or Google BigQuery or somewhere else, or Firebase, so very powerful. We use a medallion data architecture to then process raw silver, you know, bronze, silver, gold, raw, and then enrich our kind of aggregation tables. 

Moving to the Modeling side of things. That's where we have both some proprietary tools that we've built ourselves and then also some that we license. So one of them, the proprietary tools that we built is something that we call DT Sage. For us it's all about being able to get access to real-time data and to understand where those trends are moving, and provide learnings around it. So anomaly detection is so critical to our business, understanding data quality and where we're seeing things move around. Are these just the normal patterns of our business and new device launching, or did something go wrong? So we built out tooling, using our own kind of coding and some AI models like the Facebook profit model, and then integrating those into other tools like Slack and email so that we can give people direct access to things that are going on around the business that that may be problematic, but they can be investigated. Interestingly, on the modeling side, one of the things that Rill’s been great for is actually connecting to those alerts. An alert is fired through Slack, we now can click on a Rill link and it'll take you directly to the KPI that you need to investigate and allow you to deep dive that to understand why that may have gone wrong, which is already just immeasurable in terms of the value that we can get out of something like that. Also for modeling, we're big, heavy, Vertex AI users on the Google side. They power a lot of our model registry and our feature registry and where we do a lot of that work, and that really unified our data science team, which was incredibly important. 

Then on the Act side, we've also started to incorporate some new technologies that are available to everybody again on the Google Suite. We're using Notebook LLM to be able to take some of our own extrapolations of those data, whether they're documents or reports that we've built out, feed them in into a system that allows us to then interact with it creatively, collaboratively, in order to advance any idea. Whether it's our internal manuals that we use for how to operationalize a specific thing or just in terms of how we build the PRD, that's a really great tool for us as well. So lots of different tools in the space that help power our CAMA loop. 

Michael Driscoll:

Then on the action side, you talked about, collect, aggregate, model, action, I presume that's primarily Digital Turbine proprietary technologies. Maybe say a little bit about that last piece of the framework. 

Glenn Goldman: 

So action is where we take these new insightful connections that we are able to make through our data through our modeling, and then make them available to our products and our channels. One of the things that Digital Turbine recently announced was these new formations, these new products that we're calling our Ignite graph and DTiQ. Our Ignite graph is really our view of users and audiences and all the entities in our space. It's a series of user graphs, app graphs, content graphs, etc. that help us to understand who a user is and what they've done. We have a ton of modeling that goes into that which allows us to really understand user behavior, affinity, the makeup, the lifestyle traits, psychographic traits of a particular person that we can then use through all of our different channels and so we take that and we plug it into our DTiQ, which is our ML framework, in order to action those through. The recommendations that we make, whether their app or a recommendations to customers, to action them through setting our bid floor pricing on our monetization SDK, to action them through insights that we can present back to our customers and dashboards that allow them to better understand and connect that the performance goals that we're able to hit are actually in their target audience as well. That's all part of this Ignite graph and this DTiQ which make up our ability to provide real performance and benefit for our advertisers. 

Old world vs new world analytics (30:12)

Michael Driscoll: 

Where in that framework have you found the Rill analytics tool and business intelligence tool fit in? If we think about this again, collect, aggregate, model, action, maybe give us a sense of where you found Rill analytics to be valuable in that framework of CAMA. You mentioned one example of firing alerts out of your Facebook profit designed alerting system and sending alerts to Slack. 

Glenn Goldman: 

So it's toward the tail end, which is probably one of the most important parts or the bookend I should say of action. Rill for us was a way to empower people. If I look back a couple years ago, I think that when DT looked at where we were in this space, we said, hey, we have the right strategy, we know we're in the right place, we have the right assets. And we're there at the right time, by the way, whether it's with the changing regulatory landscape and how we can support mobile publishers and carriers with alternative storefronts to the traditional storefronts, as we all know that's starting to change with the epic Apple legislation and new legislation that's getting passed in in Europe. So we have all these things, right? But the way we win isn't just strategy or product or timing. It's about the people in the company. It's about their ability to execute on the information that they have to do their job better than others. You asked "what do I think about when I wake up in the morning?” Democratizing access to data is one of the most important things for a company. As soon as you empower people with insights into what's going on for their role, relevant to their role, and allow them to shorten that OODA loop, that's where we can start to win because it's all about speed. That's where Rill came in. 

In the old world, prior to starting to work with Rill, we had lots of different tools and I bet you could guess their names, but they were tools that allowed us to build dashboards with colors and different pretty graphs and configurations and all that other stuff. What those tools did is they allowed us to answer questions that we asked, which is important. What Rill allows us to do is to answer the questions we haven't asked yet. It does that by surfacing information in a way that is immediately digestible and actionable. So when I think about the old world, it was a ticket. Try to communicate what it is you're looking for to a person who doesn't maybe always have all the business context. Get a response, get a result that's curated in a specific way. You go in there, you see something interesting, and immediately you're back to the ticket. You can't deep dive, you can't go any further, you reach a dead end on your journey. As I mentioned earlier our ability to win is about speed and efficiency. And so what Rill’s done is it's allowed us to take data and from that find those insights in a self-service way. 

I can think of countless examples. We launched a new experiment and in this case we were launching a new decisioning algorithm to help us make better ad recommendations. We launched this new algorithm and we started looking at our business KPIs within the first hour on the Rill dashboard. Which, by the way, is another thing that is really important - old way was a daily refresh, new way we've got hourly data coming in, real time data effectively. We launched this experiment and within the first couple hours we can see that our KPIs don't look exactly right. So it's drill down, drill down, drill down, drill down. We start looking across all the standard metrics and we're still not able to see it because we're in our kind of curated view of the world. Within the course of the same hour meeting where we're debugging this live, we hit the back button, we go to the Explore view, and we're we're just scrolling down and somebody points out, “hey, what about that metric”, and there was one metric that that was not in our purview, wasn't something we had planned to look at. Suddenly this aha moment of a dimension that you wouldn't have even expected and it wound up being a configuration issue, but it was a completely different style of experiment that we had run before, so we didn't think about that. I can't tell you how many times that would have been a week, 2 weeks, maybe even 3 weeks of an investigation. We were able to resolve it in 1 hour and that is time, that is money, that is speed. So just one example of how Rill has really changed the way we operate. 

Michael Driscoll: 

Thank you for these compliments. You were talking a little bit about the difference between the old world and the new world. Tools that allow you to build dashboards that allow you to answer the questions you kind of already knew or wanted to ask versus answering questions you didn't know you had. I think we've all had that experience in data analysis for better or for worse. Behind every question, the answer to every question often leads to another question, right? The speed of those question and answer loops of iterative investigations, if it takes hours between each question and answer cycle, it really can be limiting to the ability, the agility of businesses to move fast. 

We call Rill a dashboard, it's an exploratory, interactive dashboard, but how do you, if someone were to come to you and say, “Hey Glenn, I'm starting a new role at an e-commerce, we're looking for tools that will help us make sense of some of our KPIs that are always changing”, how would you explain the difference between a tool like Rill and sort of traditional dashboard tools like Tableau and Looker and others of that nature? Candidly, I think we ask ourselves, how should we describe this tool to folks who haven't used it before. 

Glenn Goldman:

It goes back to a little bit of what I was describing earlier, which is the immediacy and access to data across a number of different dimensions and measures allows you to really understand what's happening in your system, and gives you this kind of holistic view. I almost imagine being in a nuclear facility with all the buttons and levers. You need to be able to see all of the buttons and levers to be able to know if something is off or something is wrong. Or being in an airplane or something like that with all the controls and knobs, even if your expectation is that there's 3 knobs that you're gonna be looking at on a regular basis. There's a couple things, but when something goes wrong, you need to be able to look across all of these other tools. Is this light on? Is that light supposed to be on at the same time as this other light example, right? Seeing the comparison between two dimensions and how those are growing. 

We use it both on the day to day product health side, but then also on the insight side. Actually this is a use case for Rill that I didn't even anticipate we would have, but quickly found out that it was driving a lot of value. We wanted to be able to provide the rest of our organization with an understanding of audiences in a rich way. And we didn't want to go and build another dashboard, another tool to do it, right? A lot of people would immediately get to work in view.js or whatever it was building from the frontend, their web application, and I guess with Replit maybe that would be faster now. But no, it was really about harnessing that kind of slice and dice, that immediacy and that kind of holistic view of all your different dimensions together, being able to play with that at scale. Because what we're talking about is being able to see the overlap in audiences at a user level between the folks that are engaging on our ads products, or exchange product, or on-device products, because as I mentioned earlier, unifying that data and modeling that data in order to really build insights through DTiQ is for us so important. One of the dashboards that we built is what we call our Audience Insights Dashboard. It allows us to view the growth of our audience across all of our different products, broken down by dimensions like geo and model and OS version and whether they have certain signals on those users, we've captured those signals deterministically or we've modeled those signals. So from that we've been able to make so many product level decisions about where we should play and when we should play. This is a space where, hey, we're seeing growth in this area, we're seeing kind of a 1 + 1 equals 3 scenario by connecting these two different product experiences. We're able to outperform for device end users. If they're users of our on device experiences, and they're users of apps where our exchange SDK is embedded, we can take those learnings and provide something really compelling for them, that allows them to to get ads that are really just more useful for them. Apps that are more useful for them. So that has been incredibly important to us and the reason why Rill was so important is because we actually tried to build this using Tableau. It wasn't performing there and the page would load for a minute and a half. 

Michael Driscoll: 

You have billions, I mean, the audiences that you work with at Digital Turbine scale to the billions, right? I'm sure one of the challenges you face is scale. You are operating at a scale that is not something you can put in Excel or something like more traditional tools. 

Glenn Goldman: 

That’s right. Modeling the data and then accessing the data are two things that Rill has brought to the table. Being able to model it using the technologies that you do, like Druid and what have you, but then also to access the data, both are equally important for us.

Current and future ways to use AI (43:51)

Michael Driscoll:

I know we're almost at the top of the hour, and I so appreciate making the time to chat. I want to talk a little bit about looking forward. We've talked a lot about the use cases, strategies, and frameworks that you use in your day to day work at Digital Turbine leading data initiatives on the product side. I would be remiss if we didn't talk about AI. This is what's top of mind for all of us. This is the next great revolution that we're living through here. We talked a few moments ago about the early days of social media and the emergence of smartphones in the mobile web. I was just wondering a little bit about how you're thinking about the future of data analytics at Digital Turbine. As you're looking at AI and you mentioned you're using Google's Vertex offering, what do you see as some of these future use cases bringing AI powered tools into your day to day workflow? And I'll add one final piece, which is how do you think that we should be thinking about bringing some of these AI capabilities into a tool like Rill?

Glenn Goldman:

100%. AI is a funny word, funny two letters I would say, in that for those of us that were in ad tech for the last 10 years, we've been using machine learning techniques forever. So what they used to call AI was just ML, of course it's now advanced beyond that. The tooling has gotten better and better and better. If I step back from the data analytics side and take my data hat off and just think about it from a product perspective, we use it in so many different ways today to drive better decisioning, better business outcomes. 

Every PRD that we write goes through an AI process. It started originally using the native tools like Chat GPT, Gemini, Claude, Perplexity, but now we built our own AI tool, which is what we call PRD bot. This was built by one of our lead architects and PRD bot takes some of the traditional mechanisms of natural language, chatbot interfacing, but then it also overlays these personas that actually influence the generation of the PRD. This was kind of the unique aspect of it, building this agentic solution where you're not only getting what the model was trained on, but you're getting an influenced version of how you should think about creating a PRD by running it through the ops persona, by running it through the revenue leader persona. How would our revenue leader, our chief revenue officer, look at this PRD, what questions would they ask to make sure that their concerns are being addressed? We created all these different personas to inform how you go about creating that PRD. It's fun, first of all, but it's also saved just so much time in terms of the cycles of the back and forth you go through. When you're in product, it's all about stakeholders' conversations and discourse and management and you have countless conversations where you're going to a stakeholder and walking them through what you're trying to build, and then they give you that feedback. Now we've kind of captured those key questions in these personas to accelerate the cycle. That's one of the unique things that we're doing over at DT and, and I mentioned Notebook LM which we're heavy users of as well and a lot of the different operations teams.

But when it comes to analytics, I think the most important thing is that not everybody wants to look at a dashboard. Not everybody wants to have to sit there and click, click, click, click, click. Ultimately, people know what they're after and anything that gets in the way of getting to that answer often interrupts the entire creative process of solving that problem, which is the beauty of chatbots and LLMs. To be able to interface with something natively and then work your way through it. I know that’s something the Rill team has brought to life as well with a recent feature that you guys have produced where you can kind of natively communicate with a chatbot to explore your data. I think that is one of the key, let's call it revolutions that we'll see now coming in the next couple years. There have been some evolutions, but when we think about revolution, which is happening now, it's about being able to communicate with something that allows you to explore your data in such a way that really shortens that feedback loop. I'm curious to see what my children will be doing, well, whether they'll even be able to type on a keyboard when they are teens.

Michael Driscoll: 

I'll add one more layer as a final thought for you to comment on, which is when you talked about the last word in both the CAMA framework and in the OODA loop framework is action. And I think that one area that certainly holds promise is that for so many analytics tools we have sensors, we have insights. But everyone always talks about, ultimately, an insight without action it's not very valuable. Companies have to act on what they perceive. 

In that example you gave, when you observed that there was a reason why something was not working with that new launch, you had to take action on it and correct it. It seems like there's an opportunity for interfaces, and let's give the example of the world of advertising platforms where if you want to change something like a bid floor, or you want to change a targeting parameter, is there an opportunity for those actions to also be mediated through a natural language interface? How is Digital Turbine thinking about that today, where again, you talked about analytics, not everyone wants to click through dashboards, they just want to ask, why is revenue down for this advertiser in the last 6 hours and get an insight, but if they want to take action on that, how might you think about that possibility in the future? 

Glenn Goldman: 

I think we're already seeing it. We've been seeing it when it comes to customer support agents and all these other things that allow you to simplify the process of getting to where you're trying to go. Just yesterday, I was trying to debug my internet service. I just wanted to talk to a person, but I was willing to put up with the chatbot to see how close it got. And the truth is, it got me there in about 45 seconds, because I was able to ask a couple standard questions and it pointed me exactly to where I needed to go and I was off to the races. That was something that two years ago, maybe even one year ago, I would not have put up with because the technology wasn't good enough to be able to actually interpret from what you were saying the correct appropriate action to take. So I think we're already seeing that quite a bit. 

Now, on the adtech side, I think audience targeting is changing very quickly. The old world of being able to select a group of people based on the fact that their IAB classifies them as sports lovers or casual gamers, I think what we're quickly realizing is there's such a deeper, more expressive way for us to kind of capture those insights and find those people than what we've been using historically. We're actually moving away from your traditional click button content management system audience targeting to more like we were talking about with natural language expression. You can imagine a world where when an agency comes to you as an ad platform, they're not walking through, here's my brief describing my target audience. If this is McDonald's, for instance, and they're saying, what we're really after is people in the Midwest, 35 to 44, who are hamburger lovers. That's kind of a very simplistic view of the world. Now we can take that, we can plug that in directly as an input into our audience building service, and we can use our ability to extrapolate with, I mean LLMs have completely changed the game, we can use our ability to extrapolate from all of the interactions that we have on the platform. With the apps that we have with content because we have really rich content that people can engage with through our moment app as individual signals, individual nodes that describe their experience and connect that in a really compelling way through the Ignite graph. The access to this technology has just accelerated. The ability for companies like Digital Turbine to drive performance for advertisers, to drive those really compelling experiences for customers, without having to go through the rigmarole of clicking buttons and making decisions. As everybody knows, it's all driven through modeling techniques that are doing, what I would call automated audience discovery and automated audience optimization. So we firmly believe that in our data exists an advertiser’s target audience, and that we want to know how they think about their audience, but that we have a deeper insight into our audience that we can provide on our platform.

Michael Driscoll:

It certainly feels that these deterministic paths we've historically used to define audiences are kind of overly rigid.  We've all seen that where we say, I want to construct an audience and you end up finding three people that match this kind of overly rigid definition of soccer mom, that lives in Rochester, you know, you have 3 people. And certainly the ability for large language models to understand that maybe soccer and football are synonyms, and not everyone uses the word mom. You could define a mom as, even though it may not be out there, that someone who's buying diapers and baby formula is very likely to be a mom, even though it's never been labeled that way. That sort of fuzzy logic that large language models are really, really good at, it certainly feels like a resonant match for what you just described as creating audiences and Digital Turbine has an extraordinary amount of signal to help inform that. 

Glenn, I'm gonna end on that. A great place to end is reminding our audience that you've got some really differentiating offerings here that have been launched recently and are coming down the pipe. Thank you so much for taking time out on your Friday to speak with us and share some of your wisdom, your learnings, your experiences. We look forward to the next conversation and, again, thanks for making time to talk with all of us. 

Glenn Goldman: 

Thanks, Mike. This has been a pleasure. It's been a real treat for me too. I really enjoyed the conversation. I love what you guys are doing here, which is really bringing just conversation to life, and so that people understand all the different ways that data and that technology, and then I can go back to creativity, can come together to make things that are valuable for people.