Using AI to tell a story with data
Unstructured data presents a massive opportunity for unlocking new customer insights. If organizations can translate that data into easily understood stories about their customers, they can take actions that help increase customer satisfaction and reduce churn.
There is no shortage of unstructured data available. In fact, by some estimates, 90% of all data is unstructured, and it is growing significantly faster than structured data. The text, images, audio, and other types of content that constitute unstructured customer data are multiplying rapidly across a range of platforms, including Discord, Reddit, Slack, and X. Collecting all this data and transforming it into actionable insights can be challenging, but the potential benefits are worth the effort.
When I was a UX engineer at Google, I saw how combining generative AI capabilities with customer data could enhance digital experiences. I was part of an amazing team that worked on marketing web pages, including the cloud.google.com homepage. Some of our projects involved building conversational UIs for sales and support that partially automated Salesforce tickets. With semi-automated support responses, we were able to rapidly address customer issues and keep customers happy.
For most organizations, though, the customer feedback chain is broken — and at Sushidata, we’re working to fix it. As a co-founder and CEO of the company, I’m working with our talented team to help organizations deepen customer insights and improve customer experiences by applying AI to unstructured data. Through our company’s journey so far, we’ve learned some important lessons about where to focus efforts and how to make the most of data.
Maximizing the value of existing customers
Focusing on the experience of existing customers can often deliver the greatest impact. It takes by far more resources to acquire new customers than to retain the ones you already have. While you’ll never stop working to bring in new prospects, addressing the needs of current customers should be among your highest priorities.
By uncovering insights from unstructured data, you can spot potential problems early. For example, some of the data we gather for our clients includes complaints from their customers who are unhappy with how long support takes them to resolve their problems. We are now working on an alerting mechanism that leverages natural language processing to fire off an alert to the right team in any medium necessary — Slack, email, or even text. With this mechanism, our customers will be able to address issues before their customers decide to explore competing solutions.
Meanwhile, understanding customer needs in real time can create new upsell opportunities. If a customer describes a particular business challenge on Slack or posts a feature request on Discord, you might be able to offer solutions that meet their requirements. And here’s some food for thought: When someone asks for enterprise pricing, what should you do? To me, the answer is to engage that customer and be as helpful as possible toward that potential buyer.
Tapping into the power of AI
How do you find the gold in vast volumes of customer data? Fully capitalizing on unstructured data requires AI. Employing large language models (LLMs), you can efficiently collect customer data from multiple sources, connect pieces of data acquired from distinct places, unify the user across platforms, analyze data to understand sentiment, spot real-time trends, and present new insights in a visual format.
This work would be nearly impossible without AI. Let’s say you wanted to organize conversational data from multiple online forums into a coherent customer narrative about your product. More than just finding every mention of your product, you need a context-aware system that can discern relationships between numerous messages, connect particular pieces of information, and generate a single story that makes sense. Given the nuanced, often ambiguous nature of human communication, this process requires sophisticated AI capabilities and context-keeping even when conversations are intertwined and fluid in nature.
Addressing key strategic decisions
How do you implement an AI-powered approach to generating customer insights from unstructured data? There are a few essential decisions you need to make, and challenges you have to overcome, before you can start capitalizing on this wealth of data.
Collection
No matter your organization’s size, community platforms such as Discord, Slack, X, and Reddit are often the best places to mine for customer sentiment and feedback. In addition to the conversations your team may be having directly with customers on these platforms, customers are also talking with their peers (i.e., potential customers) about your company and your products, providing potentially critical information.
You need to decide which technology will allow you to collect all relevant data quickly and efficiently, while adhering to platform rules and data privacy regulations. At Sushidata, we use OAuth to facilitate data collection as opposed to using Zapier, because the friction of connecting to Zapier is not something we want for our users. We place special attention to each source and make sure connecting to that source is as fast and efficient as possible. OAuth is an open standard that enables us — and our clients — to connect to an API from each platform. With OAuth, organizations can easily access customer information without having to get into the ethical gray area of scraping data from public forums.
Unification
Unifying data from multiple sources is one of the greatest challenges of analyzing customer data. On one hand, you want to bring all of this siloed data together. But you also want to understand which information came from which platform so you can take action in the right place.
At Sushidata, we assign IDs per source platform. If someone wants to drill down on a particular product issue, bug, or feature request, they can go directly to the source with a click.
Storage
If you are mostly collecting text data, you might use a traditional database. At Sushidata, we use Cloudflare’s serverless database with a separate database instance for each tenant to make sure each organization’s data is separate from everyone else’s.
If you are including other types of data, such as images, a vector database (which keeps related data in close proximity) can help speed up performance. Cloudflare’s developer platform enables us to determine which data should be included in the vector database.
In addition, choosing object storage, such as Cloudflare R2, can help you store a large amount and variety of data, from text, images, and video to log and event data.
Analysis
Organizations today have access to huge amounts of data, but using that data to make informed decisions requires analysis. AI is critical for tagging and analyzing all that data, and then generating actionable insights.
Finding or building the right AI models is key. Sushidata offers access to multiple AI models so you have the flexibility to easily explore new models as they become available. We use Cloudflare Workers AI for embeddings and text-generation models, which are run at the edge, close to users.
With the right models, you can analyze the context of the unstructured data you’ve collected and then perform multi-dimensional sentiment analysis. When my Sushidata co-founders — Victor Sanchez and Victor Ilisei — and I set out to measure sentiments, we wanted to do more than evaluating whether customers were happy or sad. There are so many more emotions you can explore.
We decided to use AI to perform five-dimensional sentiment analysis. That helps our clients better understand whether their customers are expressing confidence or fear, confusion or clarity, and more. Homing in on the right sentiment enables you to better determine the best action to take.
Visualization
In most cases, the people who use customer insights are not data scientists — they are members of a customer experience or community management team. You need to find a solution for presenting insights to them in a visual format that conveys information quickly and easily.
With the right visualization capabilities, that team can immediately see whether your company is receiving more feedback, feature requests, bug reports, or mentions of other issues. The customer experience team can map out customer journeys and then work to optimize those journeys. And they can use visualizations to share insights with company leaders.
Security
Securing customer data and complying with data privacy regulations are crucial. To protect the privacy of individual customers, you need ways to de-identify data, removing personally identifiable information (PII), as data is collected. You also need to comply with platform rules on data collection. And as I mentioned earlier, multi-tenancy is important for safeguarding data.
If you are training your own AI model, you also need to ensure that the data you are feeding into the model is not compromised or corrupted. For example, we have seen companies using Reddit data — including conversations between company employees and customers — to train generative AI models. They plan to deploy those models in forums to answer customer questions on their behalf. But they need to be sure they have clean, accurate data. If someone goes into a forum and impersonates users, the models based on that data will not deliver accurate, constructive responses.
Envisioning the future of AI-powered storytelling
Applying AI to unstructured data has tremendous potential for better knowing your customers — how they feel about your products, what problems they are experiencing, and more. With that knowledge, you can take the actions that improve customer satisfaction, reduce churn, and ultimately boost revenues.
We envision a future where you can go to Sushidata and ask, “Why are my users leaving?” and have AI tell you a story from your own data. That story can use an evolving graphic or a dashboard that — with the click of a “play” button — will help you understand your data in a way you’ve never experienced before.
Yes, there are some challenges to realizing that vision. Bringing together data from multiple sources, analyzing that data, securing data, and creating a compelling story machine are complicated tasks. But we founded Sushidata to turn that vision into a reality.
Cloudflare has played a key role in helping us build and manage our platform. Using Cloudflare products has enabled us to successfully analyze and categorize tens of thousands of conversations from online forums, transforming data into coherent customer stories. Organizations are using those stories to address customer needs and build long-term strategies that produce the greatest value from their customers.
This article is part of a series on the latest trends and topics impacting today’s technology decision-makers.
Dive deeper into this topic.
Learn more about the three roadblocks to modern app development and find out how to maximize developer productivity in the Increase developer velocity ebook.
Author
George Portillo — @georgeportillo
CEO and Co-Founder, Sushidata
Key takeaways
After reading this article you will be able to understand:
What benefits you can achieve by analyzing existing customer data
Why AI is essential for unlocking the value of unstructured data
Key challenges and strategies to turn unstructured data into actionable insights