With Technology, there’s no such thing as “Magic"’
TL;DR / Summary at the end of the post.
Disclaimer: The thoughts and opinions shared in this post are mine alone, and do not reflect those of my employer - or any other organization that I am affiliated with.
Why write this post?
When I sat down to start writing the “Geriatric Technologies” post for my security maturity blog series I felt that, invariably, companies listed in the “top right” regions of market analysis publications qualified as being “geriatric” based on my maturity criteria. As such, I set out to discover why I felt this way - and more importantly, to better understand the process of market analysis in order to to see if my gut feelings on this were perhaps misplaced.
And, as I generally find myself doing when I research this kind-of thing, I approached it from the lens of Psychology. What influences an analyst to rate certain companies above others? Especially when I know from firsthand experience that the ratings were often wrong about the way’s modern security technologies integrate into modern development practices. And more importantly - what are the financial incentives that lead to such company ratings?
This post is an aggregation of what I’ve learned from my research into the market analysis process, and is an expression of my thoughts about how to best utilize market analysis materials. I hope you find this post useful - if not a vindication of your own thoughts and feelings about this topic.
Market Analysis and the Mere Exposure effect
While I consider myself a fairly successful information security professional, it is not what I pursued a formal education in. As I have shared in previous blog posts, my collegiate education was in the field of Psychology - which I hold a Bachelor’s Degree in. Moreover, I graduated Summa cum Laude when earning my degree, which seemingly indicates that I held a solid grasp of the subject when I studied it over a decade ago 🤷♂️
But this is not a post for “humble bragging” about my past accomplishments. No, this post is about the practice of market analysis: how it is performed and marketed, as well as how decisions are made by consumers of the materials generated from these companies.
More importantly, this post is about how cognitive biases affect the analysis process, how companies can influence the outcomes of this process, and how technology buyers should (and shouldn’t) use the materials generated from market analyst firms to make informed purchasing decisions. I will also suggest how market analyst firms might help consumers further de-bias the rating materials they produce, but I highly doubt any change will come from my suggestions.
Most importantly, analysts are human beings - so of course they fall victim to cognitive bias; we all do. As such, I hypothesize that they subconsciously give preferential treatment to things they are familiar with (also known as the the mere exposure effect) when rating a given company’s technologies; and that with more exposure comes greater preferential treatment in analyst ratings and marketing materials. This is not to say that analysts can’t be biased against things they are familiar with - which is much more likely to happen when a company poorly delivers their briefing.
All the same, while I think that Haroon Meer & Adrian Sanabria expressed it well in their 2019 virusBulletin talk, my hypothesis on how the “mere exposure effect” works in this situation can be summarized in the form of a scatter plot:
Interactions and the Ambiguity Effect
The ambiguity effect is a cognitive bias that describes how people tend to avoid options that might be considered ambiguous, or when they feel like they’re missing information about something. In the case of market analysis, briefings and other analyst interactions help companies reduce ambiguity about their products in order for analysts to feel more comfortable about assigning ratings.
Unfortunately for companies, the costs associated with reducing ambiguity are quite high. In talking with friends that have experience in the market analysis industry, there are generally four types of analyst interactions that companies can participate in: briefings, inquiry/advisory calls, strategy sessions, and sometimes webinars.
When it comes to briefings, any company can schedule one with an analyst. These sessions usually last thirty minutes, and are treated as one-way communications from the company to the analyst. Most analysts will accept up-to two briefings a year from a given company (if their schedule allows for it), and will provide criteria and/or scenarios up-front for the company to speak to when delivering a briefing. Long-story short, companies can get up-to 1 hour a year of unpaid time with an analyst to share details about their product; it usually takes several weeks to get on an analyst’s calendar.
So how do companies get more time with analysts in order to reduce ambiguity and receive analyst feedback? Paid engagements - either in the form of inquiry/advisory calls, or strategy sessions. To engage in these services, analyst firms generally require a paid subscription. These subscriptions can start at $60k to $70k U.S. Dollars, and go substantially higher based on the package selected.
Paying for Attention Bias
Additional strategy sessions can cost anywhere between $10k and $40k per day depending on the analyst and firm (in addition to the company covering travel and expenses if not held virtually). These sessions allow for a technology company to receive analyst feedback on their product’s features and capabilities. During these sessions the analyst is expected to provide insights into what customers are looking for, and areas for improvement within the company’s product based on what they’ve seen competitors doing.
And in reality, these sessions are probably the greatest driver of “top right” performance on market analysis materials. The technology companies spending money on these briefings receive more details about what analyst’s hear from their customers, and have a better understanding of what the analyst is looking to see from companies when generating their ratings. Companies without this kind of information are at a severe disadvantage as a result.
In many ways, this is sort of like “insider trading” for market analysis. Technology companies with a subscription and paid strategy sessions get to tailor their product to the criteria of the “ratings test”, where as companies who don’t subscribe need spend time and money to navigate the public criteria everyone receives. Only the companies participating in strategy session have a head start on modifying their product in order to meet the “top right” criteria they are anticipate.
Finally, joint webinars between the company and an analyst also generally cost between $10k and $40k, again depending on the analyst and firm. These webinars are usually pitched as an opportunity for the analyst to share their insights into a given technology space, and for the company to highlight their own thought leadership in the research the analyst has identified.
Marketing and the Framing Effect
The framing effect is when our decisions are influenced by the way information is presented. Sort-of like how my scatter plot above framed your thoughts about this article faster and more succinctly then these words. And let’s face it - marketing works to shed a positive light on the product by framing its benefits in a way that makes the product desirable.
So how does this play into market analysis? Well, per the book “Up and to the Right” written by former market analyst Richard Stiennon, companies are advised to buy ad space in analyst’s local airports (and - I assume - other mass transit spaces). The goal here is to frame the company’s products in a positive light as early and as often as possible with an analyst while they travel to strategy sessions with their customers. Generating simple, elegant, and convincing marketing that analysts will appreciate is challenging (and a considerable investment).
All that being said, I want to be be abundantly clear that marketing isn’t a bad thing for companies to invest in - after all, how else else are customers going to learn about a product? But those excessive marketing campaigns in train or subway stations, airports, and on billboards along major highways? They are not for technology buyers or consumers. They are framing and exposure opportunities targeting the handful of market analysts still traveling to strategy sessions - and they are expensive.
Technology buyers & Bounded Rationality
Bounded rationality is the decision making process where we attempt to accept the available options as satisfactory - rather than optimize for greatest effect. With that in mind, it’s not far-fetched to say that the way people utilize information coming out of market analyst firms says a lot more about their team’s maturity than it does about the technologies they are buying.
Using analyst materials to discover some of the players that exist in a given space, and then testing a diverse set of available solutions - found both in and outside such materials - is a decent approach to being informed about technologies. Buyers should always do their own research, as it’s important to to see how well a technology fits into a company’s culture - and more importantly, how well the technology meets their needs.
As a friend of mine in the analyst industry put it: companies should be asking themselves if the criteria that leads to a product being “top right” material is the same criteria that should be used when making a purchasing decision.
The somewhat ironic answer here is that for mature teams , it’s not.
And if a business is making purchasing decisions based on a vendor’s “top right quadrant” position, then they are spending money with a vendor who invests considerable resources to perform well in market analysis - often at the expense of investments in their product. I don’t know about you - but when I’ve made purchasing decisions in the past, a company’s investments in their product are what mattered.
But then again, nobody ever got fired for buying IBM 🤷♂️
Reducing bias in Market Analysis materials
There are a few ways that I believe market analyst firms could reduce bias in their marketing materials - both in how they generate ratings, and in how they present their findings.
When it comes to generating their ratings, I believe it is important to baseline the customers that serve as a basis for the features and capabilities that analysts use to rate companies. As I’ve shared in my security maturity series posts about teams, the measures of success can vary wildly between different maturity levels. It would be a positive step if analyst firms made such a baseline transparently available to technology companies. Some products just aren’t as accessible as others.
Likewise when presenting their findings, a few ways that market analyst firms might provide additional transparency would be to display statistics about the vendors being highlighted. For example - how many hours did the ratings analyst spend with the product company in meetings, strategy sessions, or other events in the last 12-18 months? How much money has the vendor spent with the analyst firm over the same period of time? How many scheduled interactions did the analyst have with the product company?
Companies that perform well on market analyst ratings in spite of significant investments made by competitors would certainly stand out as a meaningful data point for buyers and consumers. It would also highlight the integrity of analysts (some of whom I am friends with and have great respect for) in a way that is very clear and unimpeachable. Lastly, it would shine a spotlight on how effective other market analysis de-biasing efforts are, like “interaction blackout” periods between analysts and the vendors they are rating.
Honestly, any of the data points I’ve suggested would add incredible transparency to the process, and would equally provide a defensible position from companies who appear lower (or don’t appear at all) on the marketing materials being generated. But I see this as wishful thinking, as the financial incentives for market analyst companies to add this level of transparency simply don’t exist.
How to support this content: If you find this post useful, enjoyable, or influential (and have the coin to spare) - you can support the content I create via Patreon.️ Thank you to those who already support this blog! 😊 And now, back to that content you were enjoying! 🎉
In Defense of Analysts
Good analysts will challenge companies on their marketing and product capabilities. They will make companies prove what they say is true, and will provide companies with challenges that their customers are asking about. This is a good thing, in that “top right quadrant” companies that I consider “geriatric technologies” at-least have a product that is minimally useful, and I’ll talk more about this in my next blog post.
That being said, analysts can’t very well talk to the capabilities of a product they know nothing about - and it’s unreasonable to expect them to fairly rate such products and companies. With the ~42 weeks a year that analysts perform research being chalk full of briefings, strategy sessions, and other engagements (plus the data gathered during the ratings process), they’ve got a lot of data at their fingertips. If analysts don’t have enough reliable data about a vendor, then you can’t reasonably expect that analyst to provide a good rating when they probably have a lot more data about the competition.
So how do analysts get to know about these unknown products and companies? Through briefings, strategy sessions, and other paid engagements. The financial incentives of market analyst firms to improve transparency and de-bias this process do not exist in the current economic model they have created. The materials these firms generate should be handled with a clear understanding of the financial incentives and psychological influences that play into how they are created and marketed.
So is Market Analysis “pay to play” ?
No; and almost certainly yes.
There are a number of mature technologies providing novel solutions that work well and solve problems at scale. Many of these companies choose to invest in their product(s) over spending money on briefings and outlandish marketing campaigns, which serves to benefit their customers. Unfortunately this also means that such companies are more likely to perform worse (or won’t appear at all) on the “analyst quadrants” of the world. Again, this is where interaction data would prove insightful for buyers and consumers.
A great example of this is Thinkst Canary, which does not show up on Gartner’s “deception” Magic Quadrant - and is, according to many people I have spoken with, the de facto market leader in that space. How is that even possible? And before you ask - no, Thinkst didn’t pay me to write this. Like many people, I’m just a fan of the work Haroon and the Thinkst team have put into their products.
Anyway - for every ten technology companies invested in their product, there’s at-least one company pouring money into marketing briefings, and subscriptions with analyst firms. These companies are often found in the “top right quadrant” due to the significant investments they make in time, attention, and subscriptions in order tailor their product so that they perform well. The companies that tailor their product closest to what the analysts are looking for are often times the winner.
Will some analysts recognize bias at work and put effort into de-biasing ratings before publishing their analysis? Of course. Will they still fall prey to some form of cognitive bias due to repeated exposure to a company and its product(s)? Also yes. Remember: they are human beings just like you and I - and we are all subject to biases.
TL;DR / Summary
Human beings are subject to all manner of cognitive biases, and market analysis firms have created an economic model which capitalizes on these biases to the benefit of their customers. For example, market analysts will naturally generate subconscious preferences for things they are familiar with through a process called the “mere exposure effect”. Meaning the more they are exposed to something, the stronger a preference they generate.
When this is paired with the “ambiguity effect” - our natural tendency to avoid options we consider more ambiguous - it’s hardly any wonder that analysts rate companies they have had more engagement with higher than those they’ve spoken with for thirty to sixty minutes a year. And if you’re scheduling free briefings with analysts - that’s just about all the time you’ll get.
To get more time on the calendar with analysts, companies need to spend money on subscription packages that start at $60k to $70k U.S. Dollars annually, and can go significantly higher. Additional time can also be purchased for $10k to $40k per day if companies want to hold strategy sessions with an analyst, which buys insights about customer needs - as well as product recommendations based on what competitors might be doing.
From the technology company’s side, you will find marketing campaigns that target analysts in places like airports, as well as train, subway, and bus stations. This is done to seed impressions about the company’s products with the analyst through the framing effect, which attempts to influence analyst decisions based on the way the information is presented. Much like how the scatter plot above which framed your thinking when reading this post.
Although none of this would matter if it weren’t for the customers subscribing to analyst firms for inquiries - which technology companies seek to win business from. We are just as guilty in all of this as many buyers fall prey to bounded rationality when they have the opportunity to optimize for making a greater impact in their security program, instead choosing “good enough” solutions (which often aren’t, but I digress).
Likewise, market analyst firms have opportunities to reduce bias in their publications by transparently sharing data like how many scheduled meetings were held with each technology company, how many hours were spent with them in the last 12-18 months, or how much money was spent by the technology company with the analyst firm. Honestly I suspect none of this will happen, as there’s no financial incentive to include this data.
That being said, good analysts will challenge companies on their marketing and product capabilities; they will make companies prove what they are claiming is true. This is a good thing in that “top right” products, which I consider “geriatric technologies”, will at-least be minimally useful. I’ll talk more about this in my next blog post.
So is market analysis pay-to-play? No; and almost certainly yes.
As always, thanks again for stopping by to read my pseudo-random musing 😊 While I work on the final blog post in my Security Maturity series where I talk about Geriatric technologies, you can git checkout other (usually off-topic) content I’m reading over at Instapaper, or read my free blog series on the DevSecOps Essentials.
Until next time, remember to git commit && stay classy!
Cheers,
Keith // securingdev
If you found this post useful or interesting, I invite you to support my content through Patreon 😊 and thanks once again to those who already support this content!