On Sunday afternoon, I sent Francois Ajenstat a draft of my November EMA Vendor to Watch on Golden Analytics (spoiler alert). It came out of several weeks inside the software: an onboarding session, a trial run against my own benchmark data, a roadmap briefing, and more hours than I’d budgeted (that last part is a compliment).
About an hour later, I was scrolling LinkedIn and came across a post from Francois. He’d asked AI to compare Golden with Tableau and Power BI. His finding: “Some of it is surprisingly good. Some of it is outdated. And some of it is just plain hard to decipher.”
So I replied, "Humbly making the case for independent analysis."
I went on to say that AI produces plausible, confident information that, at best, misses nuances and recent developments. At worst...it hallucinates features that don't exist. I use AI extensively in my own research, but there's no substitute for talking to product people and engineers, getting into the software, and seeing how it works.
Well, here's my less humble case.
I've Been on Both Sides
I've spent more than 20 years working with data and analytics technology, mostly as a practitioner. I've bought software, implemented it, lived with its limitations, and spent hours upon hours trying to figure out why something that looked straightforward in a demo turned out to be unusable in production.
Now I own an analyst firm, and I've been building out EMA's Data & Analytics practice. As I navigate that path, one thing is obvious: I want to evaluate technology the way I would have wanted someone to evaluate it when I was the buyer.
This takes time. A lot of time, sometimes. It would be considerably easier to read a briefing deck, ask a few questions, and write up the company's positioning. I could get an AI assistant to do most of that work for me.
And that's the problem.
The Demo Is the Beginning
My recent experience with Golden Analytics is a good example.
I loaded data I knew. I explored how the product prepared data, handled semantic definitions, suggested questions, and let me build and modify my analysis.
Some things worked particularly well. Golden's Key Questions feature suggested avenues of exploration I hadn't started with. Quick Insights found patterns worth pursuing. I could move between asking the AI for help and manipulating the analysis myself, without having to start over. Before every full-table query against my BigQuery bench, it stopped and asked me to confirm (a small design choice with a real effect on someone's compute bill).
That experience shaped my assessment. Golden’s real distinction was the way it encouraged users to explore the data and discover questions worth asking.
I probably wouldn't have understood that from a feature list.
And I certainly wouldn't have appreciated the company's operating model from a comparison table. During my trial, Golden's automated monitoring caught an issue in my environment, and the team followed up within hours. My feedback generated conversations and quick fixes. Talking with Francois helped explain the thinking behind the product and the company he wants to build. Buying software means evaluating the vendor behind it, not just the features.
Almost nothing that made Golden worth writing about came through in the AI assessment.
Of course, none of this makes Golden perfect. I found issues, made suggestions, and asked questions about things I couldn't explain. That's part of the exercise. If I spend several weeks evaluating software and come away with nothing but praise, I probably haven't done my job.
AI Is a Great Research Assistant. It Makes a Lousy Witness.
I'm not arguing against AI research. Far from it. I use AI throughout my own process and increasingly couldn’t do this job without it. The best results come from the pairing: AI covers ground quicker than I ever could, but I need to navigate and decide what holds up.
No question, AI is getting better at telling you what software does. It still struggles with what makes a product different, how it behaves, and whether it delivers on its promise. Part of the problem: vendors release features, change their positioning, fix problems, and occasionally abandon things. Published documentation doesn't keep up, and marketing descriptions don't necessarily explain how a capability works in practice.
Then there's the confidence issue.
An AI model can produce a beautifully organized comparison table, complete with categories, checkmarks, and concise descriptions. It can make a capability that doesn't exist sound like an established feature. Or miss the thing that distinguishes a new product because it doesn't fit an established category.
The formatting makes it easier to believe.
And even when the features are right, how useful is the comparison? As Francois put it, an iPhone and an Android phone check most of the same boxes, but the experience can be very different.
It’s the same problem I’ve written about with AI-generated dashboards. I give an AI tool clean data and watch it build an impressive dashboard. Then I give it dirty data and watch it build another impressive, entirely incorrect dashboard.
Making analysis easier doesn't guarantee better answers. Software evaluation has the same problem. Making comparisons easier doesn't make them more reliable.
If AI becomes the buyer's first impression of software (and it is), the quality of that impression depends on what the model learns from. Vendor copy and recycled reviews produce confident tables. Firsthand testing produces evidence.
What Are We Actually Paying Analysts For?
If all an analyst does is summarize vendor claims, package them into categories, and produce a nice-looking report, AI can already do a convincing imitation. And it's getting better. If a model can generate it from public material, it's a summary, and summaries are now free.
I don't see much value in defending that work.
The value comes from experience and judgment. Knowing which questions to ask because you've encountered the problem before. Recognizing when two vendors use the same words to describe very different things. Understanding why a seemingly minor implementation detail matters to an enterprise buyer.
And doing the work to find out whether the claims hold up.
That means talking to people. Product leaders, engineers, customers, and practitioners. It means getting access to the software and working with data and scenarios that you understand. It means separating what a product can do today from what the vendor hopes it will do tomorrow.
It also means independence.
A vendor can give an analyst access, answer difficult questions, and help resolve issues. None of that should buy a favorable conclusion. The analyst's obligation is to explain what they found, including the weaknesses.
That's what I want EMA's research to deliver. I'd rather publish fewer pieces with firsthand evidence than more pieces that merely repackage what vendors already say about themselves.
I'm under no illusion that every product warrants weeks of testing. Sometimes the right approach is a briefing, a focused demonstration, or an examination of the architecture. But the analyst should know the difference between what they've heard, what they've verified, and what they've actually experienced.
The reader deserves to know it, too.
Vendors have a stake here. Francois made the point himself: the AI's answer might be a buyer's first impression of your product, right or wrong. More content won't fix a stale answer. An honest opinion from an analyst who has actually investigated the product, saying what they found, will. That goes double for products that break the mold. The further a product departs from convention, the less a conventional comparison can tell you about it.
AI will make it easier than ever to produce an answer about which software to buy.
That makes independent analysis more valuable, provided we’re willing to do the work that makes it independent.
