AI or Authentic Insights

As a market researcher, one of my roles has been to train colleagues and clients in how to generate deep, useful insights. Anyone who has worked in research knows that getting beyond the data and identifying meaning is a critical skill and what sets insight apart from simple information. It involves curiosity, creativity, an understanding of context, and the ability to draw together knowledge and understanding from a range of sources.

The trick to doing this is to understand that consumer insights come from identifying the connection between two or more different pieces of information to explain why people behave the way they do. For example, the classic insight behind Nike's "Just Do It" is that people really want to get out and get fit (piece of information #1), but they also need some motivation to get started (piece of information #2).

To help people understand how to generate insights, I have often used a training exercise where people are given 2 random pieces of information and asked to come up with an explanation of how they relate to each other. It's a fun exercise that drives the point home and gets people thinking creatively to come up with an interesting story.

Which brings us to AI, or more specifically, the use of AI in market research. It's the latest hot topic. It's also something that we have been working on for a while at Edentify. We have developed a range of sophisticated APIs linking to various versions of AI and Large Language Models (LLMs) into our research platform so are deeply invested in this kind of technology. We use LLMs to help identify themes in survey replies, populate code frameworks, theme qualitative transcripts and validate respondent replies.

However, the more we have studied and implemented AI into our workflow, the clearer it has become that, while it is a helpful tool, it is not the whole answer. LLMs are great for understanding and summarising information quickly. They can allow you to get a feel for the main issues and sentiment coming through from large written data sources. They can save an enormous amount of time.

But what AI and LLMs cannot do is make creative connections that build insights, nor can they come close to integrating diverse knowledge and experience like a human.

And AI certainly cannot replace talking to real people: run far away from anyone trying to sell you "synthetic" research respondents.

Instead, the somewhat generic outputs that these AI tools provide are useful starting points. They allow us to streamline the repetitive and time-consuming aspects of the job and get to the real craft of thinking and analysing.

At Edentify we often talk about the power of research technology only being realised when it is combined with real people and real thought. It's the curiosity and creativity of researchers that takes research from information to insight: the ability to look at data, draw on experience, make connections, and think, "Ooh, that's interesting. I wonder why..."

We also wrote about this back in 2018, and while our technology has advanced a lot since then, our conclusions were similar: AI in Research

If you need help or advise on how best to use AI and LLMs then please contact us.

Note: An actual human being from Edentify called David wrote this article but chatGPT was used to come up with the title.