Just a few years ago, every conversation about artificial intelligence (AI) seemed to end with an apocalyptic prediction. In 2014, Elon Musk said that, with AI, we are summoning the demon, while Stephen Hawking said that AI could spell the end of the human race. More recently, however, things have begun to change. AI has gone from being a scary black box to something people can use for a variety of use cases.

This shift is because these technologies are finally being explored at scale, including by product teams for market opportunity identification. AI hasn't always been used in the industry. In fact, it started out as a scientific curiosity. In the 1950s, computer scientist John McCarthy wanted to see whether it was possible to build machines that could learn how to do tasks such as play chess themselves. Today, AI is everywhere.

In this chapter, we'll explore how market opportunity identification can be improved with big data and AI, covering the following topics:

• Identifying market opportunities the traditional way
• Big data challenges in market opportunity identification
• Using AI for market opportunity identification
• Exploring AI-powered market reports

Market opportunity identification is important for a product team to identify an unmet need. It helps them find out how their product will stand out in the market and what they need to do in order to grow. They need to identify the competitive landscape, define the market opportunity, and use this to create a value proposition.

Furthermore, market opportunity identification sets the groundwork for later chapters, including creating product ideas and predicting future market trends.

When it comes to commerce, venturing blindly into the unknown is a recipe for failure. Belief in a market opportunity is not enough – there needs to be hard data. Market opportunity identification is the process of acquiring and analyzing data, from any source, to understand the potential size of a market and the potential share of that market that you can capture.

Broadly speaking, the market should be considered in three segments:

• External: External markets are those that are already established and they may be served by other companies.
• Internal: Internal markets are those that exist within the company but they may not be recognized as such yet.
• Potential: Potential markets are those that have not been identified yet.

Traditional market opportunity identification can be done in a number of ways:

• Firstly, it can be done by surveying the consumers. Market surveys are one of the most effective ways to find out what customers want and how much they are willing to spend for it. This not only gives a company an idea of what to produce but also helps figure out how much money they will make from their products.
• Secondly, market opportunity identification can be done through brainstorming techniques such as the 5 Whys technique. The 5 Whys technique is simple and can be used as a brainstorming exercise by asking why? five times in response to a topic, problem, or issue.
• Thirdly, market opportunity identification can be done by analyzing internal data such as surveys and interviews that have already been done before. Often, large companies will have a lot of unorganized, unstructured information that is divided across departments and projects, from Google Surveys to JotForm to SurveyMonkey. When this data is analyzed, they may find opportunities they weren't aware of before.
• Finally, traditional market opportunity identification can be done by examining external data and data from social media platforms or competitors in the same industry. This includes sites such as LinkedIn or Facebook where both companies' and individuals' data can prove valuable.

One tool that is becoming very popular for market research is Google Trends, which allows people to examine the search volume in a particular area of interest over a certain period of time. For example, if you were interested in finding out about the popularity of rooftop gardens in Los Angeles, you could type into Google Trends Los Angeles Rooftop Gardens, and analyze search trends over time. Additionally, Google Trends lets you easily export this data for further analysis, whether it's to create visualizations or merge with additional data.

Once potential market opportunities have been identified, then the company should consider the decision to either pursue or ignore them. If the company chooses to pursue an external, internal, or potential market, then it needs to consider which approach is best.

For example, if an internal market is being considered for pursuit, it may be best for the company to create an incentive and offer it to its employees in order to get them involved with the new product.

If an external market is being considered for pursuit, it may be best to invest in advertising campaigns to make customers aware of the new product.

If a potential market is being considered for pursuit, it may be best for the company to invest in research and development in order to create a product that will appeal to this new market segment.

There is also a difference between customer-driven identification and opportunity-driven
identification. Customer-driven identification is where an organization determines a marketing need based on what customers want or need, whereas opportunity-driven identification is where an organization identifies areas of potential value based on strengths and weaknesses.

For example, if an organization is strong in manufacturing but weak in marketing, then opportunities for enhancement may be found in the marketing area that would not have been found had they done customer-driven identification.

While there are a number of benefits to traditional market opportunity identification, these methods fall short in the modern world, which is faced with big data challenges.

Big data has become a buzzword in the product community.

Big data involves the speed at which data is generated, the amount of data that's generated, the types of questions that can be answered with it, and the number of sources it's coming from. In short, big data is about more than just size.

Big data describes the veritable explosion of data we're seeing from billions of people accessing the internet.

Product teams want to use big data to identify new market opportunities and new ways to target their customers. Yet, many companies are struggling with how to collect and analyze big data, particularly when the data is scattered across different sources.

Business executives are asking questions such as: How can we get useful information from all of this disparate data? How can we make better strategic decisions with our big data? How can we address the challenges of storing, organizing, and managing big data? And how can we increase the value of our big data?

Commonly, there are a few major challenges associated with managing big data for market opportunity identification:

• The first challenge is that it's difficult to find relevant predictive patterns in a sea of unrelated variables. For example, using traditional business intelligence (BI) techniques such as data munging and manual data mining, it might take weeks or even months to discover a predictive pattern in a large database of customer survey results. Imagine that your dataset includes demographics, firmographics, psychographics, purchase data, reviews, and more. That's a lot of information to look through, and a lot of variables that might not seem relevant at first glance.
• A second challenge is that traditional BI tools are not designed for efficient discovery of predictive patterns when analyzing big data, which is increasing in volume at a faster pace than traditional databases are being updated. Not only does this make it difficult to keep up with the latest insights, but it also becomes more costly and time-consuming to build insights into existing systems.
• Another major challenge with big data is simply getting it in the first place.

Large companies such as Google and Amazon have access to tremendous amounts of computing power and virtually unlimited storage thanks to their substantial investments in hardware and software services, but for smaller organizations – even those that are eager to harness the potential of big data – the story is different.

Data is coming in faster than ever before. The explosion of data being generated has outpaced the capabilities of traditional database systems to keep up with growth, but setting up and maintaining systems such as AWS or GCP requires dedicated engineers, given the requirements for technical know-how on scaling, security, data pipelines, and more.

Therefore, the problem is often that businesses lack the necessary resources that would allow them to collect, aggregate, analyze, and interpret such volumes of information without investing large sums of money in server clusters or other specialized infrastructure.

With AI, companies can solve these challenges more easily, and analyze big data to improve market opportunity identification.

The most important thing to remember about market opportunity identification with AI is that it is not about creating a magic wand that will instantly identify all of the major new market opportunities. That would be a data scientist's or software developer's dream. But in the world of marketing, where success depends on being fast, efficient, and smart, those are the characteristics of an outright nightmare.

Marketing and product development professionals are constantly under pressure to deliver new products and services to market – and be first to market with them. They need to do so in a cost-effective manner, without sacrificing quality, and often with limited resources.

AI offers speed, efficiency, and smartness. With models deployed on scalable servers, AI can scan a huge amount of data and identify patterns that humans wouldn't see. This means that millions of data points can be analyzed within hours or even minutes.

AI is being used in a number of verticals, such as autonomous vehicles, facial recognition, and fraud detection.

By using AI for market opportunity identification, marketers can free themselves from the burdens of manually reviewing and analyzing information on countless possible markets or new product categories. They can offload tasks such as data collection and the creation of new product ideas to machines.

The best way to use AI for market opportunity identification is by focusing on real problems that need real solutions. For example, marketers may want to analyze their existing customer base for potential new product categories they could enter into – or specific products they could create for those existing customers.

Another example is using AI to identify new geographic markets for existing brands – or even entirely new brands – by looking at various combinations of customer demographics, buying habits, lifestyle choices, and other criteria in each region of the world.

In either case, AI can assist marketers by helping them use data already in their possession more effectively than they could otherwise (by automating some data collection tasks and streamlining others). It can also help overcome some key barriers, including assessing whether any given potential market or product category is truly scalable.

An additional benefit of using AI for market opportunity identification is that you can also use it as a tool for demonstrating your commitment to innovation and differentiating your brand(s) from competitors' offerings – which can also boost your brand awareness, reputation, and overall value.

As we've explored, AI has a number of broad benefits for product teams. Let's dive into a specific use case: AI-powered market reports.

AI has transformed machine learning, a branch of artificial intelligence that automates the identification of patterns in data. The big data analytics industry is abuzz with the possibilities of AI-generated market reports (www.commerce.ai/reports) in identifying opportunities and trends.

An AI engine can be used for a variety of purposes, including generating market reports to identify potential opportunities for improvement in marketing campaigns. The technology is ripe for marketing professionals to take advantage of, making it easier than ever to generate local or global reports using existing data.

Here are some of the benefits of using AI-generated market reports:

• Automation: Automating repetitive tasks such as generating reports can help marketing professionals save valuable time and energy.
• Cost-effectiveness: Generating report templates could cost less, with AI used in the background to quickly adapt the template for local or global needs.
• Overall quality improvement: By using AI, marketers could have more accurate, precise, and complete information to use in their campaign planning efforts.
• Data tracking: Using machine learning algorithms to analyze large amounts of data could help marketers gain valuable insights into consumer preferences and buying trends.

Highlighting these benefits is not meant to imply that there are no drawbacks to using AI-generated market reports. Instead, it is meant to highlight several key benefits that can help you make a final decision about whether or not you should use this technology in your business operations moving forward.

With a new AI-generated market reports feature, Commerce.AI now delivers high-quality insights straight to the public.

Previously available exclusively to those with access to Commerce.AI's data engine, market reports are now available to all.

Market reports analysis is based on consumer feedback and offers valuable insight into how people consume products and services. With the press of a button or a simple search, anyone can now access market reports across 10,000 categories.

Commerce.AI's AI-generated reports are a concrete way to find your next product idea or seek a new customer base:

Figure 1.1 – A sample of Commerce.AI's AI-generated market reports

For example, searching for gaming keyboards will show us a high opportunity score, as this niche market category has grown tremendously during pandemic-fueled lockdowns. As Commerce.AI continues to grow, the quality of its insights will only improve and become more accurate as more product data related to new trends in the marketplace is analyzed.

Commerce AI's data engine has delivered over 140 million dollars in revenue through insights across market categories. The market reports are an incredible resource built around billions of unstructured product data points, from sources such as forums and blogs, surveys, videos, customer support tickets, CRMs, and more. They provide a glimpse into how people consume products and services, with the potential to help you find your next big idea.

When it comes to innovation and product development, market research is invaluable. It provides insights into how people consume products and services, which can be used to inform the next idea that will take on the world. Without using consumer feedback as a starting point for conceptualizing original ideas, you're limiting possible endeavors.

Commerce.AI's market research reports include a summary for each category, the fastest-growing brands, the bestseller, top-rated products, the number of products and reviews, and more. All this data is consolidated into a single value – the opportunity meter – showing the size of the market opportunity at hand.

AI calculates this by taking into account the average market size, the number of competitors in each category, and the potential for growth, creating a line of best fit between the data points:

Figure 1.2 – A sample AI-generated market report

Traditional market research is expensive, inaccessible, and confusing. Commerce.AI's new AI-generated market reports feature solves these problems as well as the issue of a limited time frame in exploring market opportunities.

Market reports are available without having to pay an exorbitant fee for access to data-driven consumer insights.

Commerce.AI's system relies on a variety of different types of product data points, including unstructured data, which makes up 95% of the web. These sources include Amazon, Walmart, Target, and other commerce sources that are crucial for product teams.

All of this, compiled together and analyzed with AI, produces marketplace insights providing valuable knowledge into how people consume products and services.

With Commerce.AI's new marketplace insights, you can be confident in the next product idea you're considering as well as your current markets. With data from 10,000 market categories scanned and reports available to all, there has never been this scale of access to high-quality insights.

Skip the expensive market research firms that don't have a clue about how people are spending money these days, and take advantage of the power of billions of data points analyzed for you with AI-generated market reports.

Of course, there are other outlets for market reports, from Gartner to Nielsen. However, without the power of AI, these traditional market reports can be subjected to biases, while missing out on analyzing billions of data points. Moreover, AI-generated reports are far more cost-effective, costing a fraction of their traditional counterparts.

For instance, Gartner's research subscription costs around $30,000 a year, since the overhead that goes into traditional reports is high. This is because it takes highly paid teams of analysts many hours to produce a report.

In contrast, AI-generated reports compile billions of data points across the web using natural language processing (NLP) software; as such, they take only minutes to generate and cost far less. NLP works by computationally analyzing patterns in language, which gets encoded as numerical token values. It's based on the idea that all language is structured around a core set of elements, such as words and phrases, and these can be combined in many different ways to express an enormous range of ideas in a vector space.

Billions of data points are boiled down into a single opportunity score for any given market, providing unparalleled accuracy and scale, without sacrificing clarity and ease of understanding. These data points provide organizations with unequaled insights on how to advance their strategy and prepare for expansion around the globe.

Clearly, AI has the potential to radically improve the quality and speed of market opportunity identification.

But it's not just about finding better ways to do what researchers have always done or even automating tasks. Instead, AI can help us see new opportunities that we never could have uncovered before.

In a traditional BI environment, analysts must sift through hours of data on thousands of companies in order to identify promising new targets.

With an AI solution built around unstructured data sources (such as text documents or images), this task becomes straightforward; an AI system can simply scan millions of documents for identifying keywords or patterns and then provide recommended matches with commercial significance.

Commerce.AI is one of many advanced technologies available to product teams. It uses a range of machine learning techniques to find insights based on structured and unstructured data.

Understanding market opportunity identification is important for product teams to lay the groundwork for the product development process. Traditional methods have been in place for generations, and are still relevant today, while AI-based methods can give teams a competitive advantage. This understanding is crucial for the upcoming chapters, as product development should be based on a deep understanding of the market. Moving on from this understanding, we'll dive deeper into topics such as building, selecting, and iterating product ideas.

In the next chapter, we'll explore how the product creation process can be improved with AI.