A manifesto for overhauling digital 'experimentation'
tl;dr
The Fundamental Problems with Experimentation
Experimentation isn't the 'proof' people think it is. Despite its perceived infallibility, digital A/B testing is often unreliable due to issues like generalizability, technical glitches, and statistical limitations. It provides valuable evidence, but not concrete proof.
Experimentation has become an end in itself, not a means to an end. The discipline has been commodified by software vendors and agencies, turning it into a standalone "job" rather than a strategic tool. This focus on the "hammering" ignores the larger goal of "building a house."
Experimentation equals optimization. Its scope is typically limited to minor UX tweaks, missing significant strategic opportunities. This narrow focus means big business decisions are still based on opinion, while small changes are "optimized" and disconnected from true innovation.
The myopia of ‘metrics’. Relying solely on simple metrics like conversion rate is incredibly short-sighted. These numbers don't reflect actual business value or long-term impacts like customer loyalty, and they fail to capture the complexity of a healthy business system.
From Ashes to Innovation: A Manifesto for Rethinking Experimentation
Resetting the vision: Critical thinking for business innovation. Businesses should emulate the scientific method: embracing bold ideas, skepticism, learning from failure, and having clear, aligned objectives across the entire organization, led by example.
From ‘metrics’ to systems thinking. Move beyond simple metrics to understand complex, interconnected business systems. Focus on ultimate, systemic outcomes and true customer experience, recognizing that the world isn't as simple as isolated numbers suggest.
Research + human decision-making. Experiments are merely one form of research, providing information to aid human decisions. The focus must shift to improving human decision-making processes, understanding biases, and integrating evidence from all sources.
The workflow and Operating System of de-risked innovation and learning. Innovation requires structured methods and processes. Build tailored "operating systems" to manage ideas from initial thought to validated outcomes, thereby systematizing critical thinking and reducing risk.
Conclusion
The landscape of digital experimentation needs a major overhaul. We must shift our focus from viewing experimentation as an end (and solely UX-focused) to recognizing it as just one piece of a broader innovation puzzle. Redefine it as research that supports human decisions to drive authentic growth.
Introduction
I have spent the last 16 years of my career in digital product experience roles, but always with experimentation at the heart. After all that time, one might expect a seasoned practitioner such as myself to boast true mastery over the discipline as well as absolute commitment and belief in its powers. Yet this is far from the place in which I find myself. Instead, my journey has led me gradually to a very firm realization:
"Experimentation" is universally employed by businesses in a way which achieves only a tiny fraction of its true potential.
'Nullius in verba' is latin for 'take nobody's word for it' and was the founding motto of The Royal Society, formed at the time of the scientific enlightenment. It represents the core of the critical attitude and the simple resolve to discover the world in a way not dictated by dogma, authority and opinion.
This is the driving principle behind any kind of experimentation, and yet, in my view, what gets called "experimentation" in business has been productised, commercialised, simplified and compartmentalised so as to have almost no connection to this principle at all. It is (mostly unintentionally) misused, misunderstood and, in many businesses, completely ineffective.
Experimentation is not as it seems. It is a bland imitation of what it should and could be!
In this article, I embark on a candid exploration of my experiences, aiming to distil the essence of what I've learned into a manifesto of sorts. My goal is to shed light on the shortcomings that plague current practices while charting a path forward toward a more enlightened approach. It's time to tear up all the rule books and carve out a new direction for this world grounded in real, meaningful business improvement.
One very important caveat: I am not for a second suggesting anyone doing experimentation right now needs to down tools, or that the whole endeavor is pointless. There are a great many things in business which everyone knows full well are not as effective as they appear on face value, but they are still valuable. This is long-range thinking about how we can refine and improve these concepts over time in order to realise enormous untapped potential.
For this reason, this is not intended to be a set of quick guidelines that can be implemented overnight. Likewise, it will be best read by the more senior members of businesses who are in the best position to drive transformation across broader areas of the business. The purpose is to inspire and guide towards a different future.
The fundamental problems with experimentation
"You must be ready to burn yourself in your own flame; how could you rise anew if you have not first become ashes?" - Friedrich Nietzsche
I first ran an A/B test around 2007 or 2008, and I was instantly hooked on the idea that you can get infallible evidence on the outcome of creative changes. For someone like myself, inclined towards data-driven decision-making, the notion that one could attain reliable proof regarding the outcome of creative changes is both immensely satisfying and instantly addictive. After all, what could be more rational, scientific, and objective than controlled experimentation?
But after all these years, I can very firmly confirm that the reality is completely different from the ideology. The fact that flippant, opinion-based decision-making is a terrible way to run a business is, and always will be, very true. The intention to use data to make good decisions and not be plagued by bias is, and always will be, entirely valid. The entire current method for actually trying to do it is, however, held together by pieces of string and sticky tape.
I believe the issues can be categorized as follows:
1. Experimentation is not the 'proof' that people think it is
Despite its widespread adoption and enthusiastic promotion, experimentation (A/B testing) mostly does not deliver quite what people believe it is delivering. The idea that a controlled experiment provides reliable ‘proof’ of how customers will respond to something is incredibly far from the truth for many reasons. These issues are largely swept under the rug because there is no viable solution to them. These are some of the main ones but there are more:
- Generalizability: One of the fundamental tenets of controlled experimentation is the idea that observations made under controlled conditions can be generalized to broader populations or contexts where the conditions are identical. In digital A/B testing, this means we assume that the outcome of an observation in a given period of time will be replicated at all times, i.e. when you push the thing live. However, there is absolutely no reason that this is necessarily the case. The composition of customers, their intentions and countless external factors in one week may be completely different the next week, let alone in 6 months' time i.e. the conditions are in no way identical. Your ‘observation’ may have no relevance at all outside that specific test.
- Digital technology issues: The implementation of experimentation initiatives is likely, for most practitioners, to be consistently marred by technical challenges related to data accuracy and integrity. Issues such as cookie tracking limitations, automatic cookie deletion, cross-device visits, data privacy regulations, ad blockers, and the general complexity of web technologies can and do very easily compromise the reliability of experimental data, leading to skewed results and erroneous conclusions. Nobody really knows how deep this rabbit hole goes, but one thing is certain: even subtle errors in data can create erroneous results.
- Statistical limitations: The reliance on statistical significance as a criterion for decision-making introduces inherent limitations to the experimentation outlook. The majority of experiments fail to achieve statistical significance due to sample size constraints, variability in user behavior, overly subtle effects or other factors, rendering their findings inconclusive. However, this does not mean that the idea is not a valuable idea, it simply means you cannot use statistical controlled experiments to determine whether it is a good idea. The output of experimentation is therefore entirely governed by the arbitrary limitations of its maths and nothing else! It’s like dipping a 30cm ruler into the ocean and thereby concluding that the ocean is 30cm deep!
- Statistical illiteracy: Experimentation statistics are incredibly complicated and very, very easy to get wrong. The people tasked with actually practising this discipline do not have the time, inclination or skillset to understand it. Even the ‘experts’ disagree on fundamental aspects of how to do it. The effect of this is that the vast majority of people running experiments are doing it wrong without ever realising it. Experimentation is meant to counter confirmation bias but, in reality, it just hides it in more ingenious ways!
To be clear, these things do not render experimentation useless, and it will still be far better than a mere guess, but the issue comes when experimentation is seen to be infallible proof and used as a final decision making tool, which it mostly is.
For example, most "CRO" largely runs on the basis of running tests and then using those results to project exact revenue forecasts for what that change will do. Do the people buying this truly understand how problematic that is? Fine when these issues are transparent and understood but they rarely are.
Even if none of these problems existed at all, the probabilistic nature of statistical testing means that there is a chance the result you think you are seeing is wrong. When accounting for these technical problems, that likelihood soars, so experimentation is and can never be the ‘reliable decision’ that people think it is.
It’s still an immensely useful tool, but it is simply a piece of research and evidence amongst others with its own inherent limitations and problems. At the moment, it is often seen as the the pinnacle of evidence that we strive towards. We run research in order to then run experiments and so we think an experiment is something bigger, better, and more definite than a piece of research. It is not.
2. Experimentation has become an end in itself, not a means to an end
If you were to ask a scientist what they do, they would not say "experimentation". Even within a complex laboratory setting, those conducting experiments would probably elaborate on the specific subject of their research and its underlying purpose. In scientific practice, controlled experimentation serves as a tool, but it is not necessarily the sole or even the most appropriate tool for the task at hand.
Picture asking a carpenter what they do and receiving the response, "I do hammering."
Yet, we find ourselves entrenched in this entity known as experimentation (or CRO, or whatever moniker it's given). It's become a job title, the focus of entire teams, and the bread and butter of dedicated agencies. However, in many cases where it's practiced, experimentation is a method and tool completely divorced from its intended broader purpose.
Several factors contribute to this disconnect:
- The need for simple solutions. The business world, for the most part, abjectly refuses to see the complexity in customer behavior and the innovation required to solve for the challenges it creates. We want quick wins and best practices. We want simple things we can buy that plug holes, move 'metrics' and solve issues without much effort. Similarly, we need to compartmentalize these different ‘things’ into easily manageable chunks and budget line items that can be dropped if they’re not delivering. This need is reinforced (unintentionally) by:
- Software vendors. From its inception, the discipline of experimentation has been driven by software vendors and their commercial agendas. What better way to elevate the importance of your product than to position it as an end in itself? The discipline becomes subordinate to the software tool, rather than the other way around. It's akin to purchasing a hammer without considering what needs to be built or why!
- Consultants and agencies. How does one market a service like experimentation? By packaging it as a neatly defined product that clients can readily purchase. It's streamlined for simplicity, boasting tangible outcomes like increased conversion rates and quantifiable ROI. Clients are promised a hassle-free experience, with consultants and agencies handling all the heavy lifting.
- The allure of emulating science. Who doesn't want to feel important, credible and academic in their career? Instead of just doing advertising for toilet paper, I can be a scientist! And so we appropriate the things of science and wear them as badges in order to look like scientists. These things of science (such as statistical significance and sample sizes) become far more important and interesting than the toilet paper advert, and so we forget why we are doing it in the first place and obsess over the discipline for its own sake.
The impact of this is that almost everyone involved is focused on hammering instead of on building a house!
This is the root cause of many of the complexities and issues in the industry today: the weird practices around forecasting ROI of AB testing; obsessing over winners and win rates; the unhelpful scientism behind most of our thought leadership. Most of all, the fact that experimentation is nowhere to be found unless there is a piece of 'software' involved that does it for you!
3. Experimentation = Optimisation
Businesses, eager to improve their online presence and drive conversions, often view experimentation as a means to fine-tune user experience (UX) and incrementally boost performance metrics. However, this narrow focus overlooks the broader opportunity experimentation presents: the chance to test, learn, and embrace failure as a catalyst for strategic innovation.
When Jeff Bezos talks about experimentation, learning and failure, he doesn’t mean testing CTA locations on pages; he’s talking about launching huge experiments like the Fire Phone, which might fundamentally change the nature of the whole business.
The word experimentation has come to mean ‘UX A/B testing for metric improvement’ when it ought to mean ‘the critical approach to business decisions and innovation.’
There are three core issues here:
- Limiting scope to UX: Experimentation in many organizations is confined to tweaking website elements, such as button colors, page layouts, or navigation paths. While these micro-level optimizations can yield incremental improvements in conversion rates or engagement metrics, they often fail to address larger strategic questions or drive significant business outcomes. In most businesses, bigger strategic projects are pushed through based on opinion whilst tiny UX changes are subject to strict controlled experimentation. Where is the logic in this? This, again, comes from our obsession with software. Experimentation is thought of only in terms of the software that enables it.
- Lack of strategic alignment: For a business to take an effective test and learn approach to strategy and innovation requires two fundamental things: a) clear vision, strategy, and objectives that are aligned and understood throughout the different levels of the business and b) the empowerment of people to own their objectives, make appropriate decisions and embrace failure and learning in their pursuit to deliver them. Whilst most businesses have some kind of vision and mission, in my experience the people on the front line are not connected to those things in any way and are certainly not confident enough to do anything they deem ‘strategic’.
- Ineffective or non-existent product processes: In product environments, experimentation ought to be embedded into the entire end-to-end development process such that ideas are tested and validated before anyone invests any serious time in building. However, even where product teams believe they are doing experimentation, this often turns out to be a superficial box-checking exercise once you lift the lid. In the worst-case-scenario, teams are driven by delivery objectives which means they need to keep busy. They come up with ideas without any serious thought, rush them into large projects and then ‘feature release test’ them on deployment, often cooking the numbers to make it appear successful. In this sense, optimization becomes a form of afterthought and politics exercise with no intrinsic value.
Optimisation is tweaking and finalizing something which is already finished. It's ironing out minor issues and polishing the edges. It's driving a little bit more out of a ‘metric’ when other things have done the serious leg work. It's a nice-to-have and good housekeeping, but when push comes to shove there are more important things to focus on.
And yet those 'more important things' are decided based purely on opinions and guesses (usually under the guise of expensive consultancy presentations), not with any sense of evidence-based learning and failure, simply because there is no 'software' that can run an 'experiment'.
4. The myopia of ‘metrics’
This one is, for me, perhaps the most fundamental and problematic of all these issues. Almost all digital experimentation is focused on the optimisation of simple metrics like conversion rate; however, this is an incredibly short-sighted way of viewing customer behaviour and business development.
Just because you can encourage someone to purchase does not mean they are a good customer for you. You can encourage more people to buy, but at the same time encourage them to buy cheaper, less profitable products which they are more likely to return and/or which erode customer loyalty & brand advocacy and increase pressure on acquisition media, all creating a downward spiral of unseen failure.
Everyone in the experimentation community understands that proving an impact on something like add-to-bag will not necessarily ever correlate to improvements in ‘downstream’ metrics like conversion, but nobody ever wants to admit that conversion and even revenue are nowhere near the bottom of the stream.
The world is unfortunately just not as simple as ‘metrics’, despite it being highly convenient to think that way. The effect of this is that your entire experimentation programme might look like it’s delivering results when the opposite is, in fact, true. Businesses have entire teams or agencies beavering away to drive improvements in conversion and diligently reporting their ‘ROI’ when they might be eroding the entire fabric of the business at the same time, and nobody would ever know because nobody is capable of seeing beyond these ‘metrics’ to what really matters.
It is simply impossible to run reliable controlled experiments on changes that measure the things that actually matter, such as customer experience or advocacy.
From ashes to innovation: A manifesto for rethinking experimentation
"The secret to change is to focus all of your energy not on fighting the old, but on building the new." - Socrates
The basic sentiment of using research to come up with ideas and validating those things with real customers is entirely valid, but the approach taken has become so mired with simplistic commerciality and business management requirements that it is often a mere facade of what it is meant to be with no real substance.
We need to remove our collective focus from the technical task of controlled experimentation and focus instead on the aspects that actually matter. We need to build a house instead of merely hammering!
There are 4 key aspects to this transformation:
1. Resetting the vision: Critical thinking for business innovation
Almost everything remarkable that has ever been achieved in modern civilization, from space flight to medicine and beyond, has come about due to the critical scientific attitude and its methods discovered during the rational enlightenment.
Why do businesses not try to emulate this? A business is a journey of invention, discovery, and innovation; why would you not want to try and learn from the guys who put someone on the moon?
How do great scientific discoveries and innovations happen? What was it that made them possible?
- Bold and visionary ideas that strive towards clear and common goals
- Scepticism and critical thinking that seeks to uncover the truth through evidence and shuns dogma, authority, and bias
- Determination and learning from failure
- Some kind of process that allows all this to unfold.
Not the "technical task of controlled experimentation."
Imagine if, instead of trying to get a person to the moon, NASA had instead decided to simply ‘do some experimentation to optimize rocket propulsion force metrics’ for its own sake and without looking beyond that simple task - this is essentially what the entire discipline of digital experimentation is today.
How does one create this culture?
- Clear vision and aligned objectives: Establish and agree a clear collective vision that inspires and aligns employees towards common goals. Implement objectives and key results (OKR)-like frameworks to provide clear alignment for achieving those goals, fostering accountability, and driving collective action. Most importantly, these kinds of objectives should tell people what they are trying to accomplish through the process of critical thinking and evidence-based decision making.
- Company-wide scope: Why innovate only in places that happen to have a tool or the ‘right kind’ of data? There are endless fields of science where data and methods are limited or unethical, but this does not stop the critical attitude and method. This thinking and approach can be applied to literally everything if it is framed in the right way.
- Leadership by example: Cultivate leadership qualities that embody humility, curiosity, scepticism, and critical thinking. Leaders should set the tone by actively seeking evidence-based thinking, encouraging open dialogue, and embracing vulnerability and failure as a natural part of the learning process. There is nothing more powerful than saying ‘I think this, but there’s a good chance I’m wrong; who else has a different view or different data?’
- Empowerment and trust: Empower employees to take ownership of their work, experiment with new ideas, and learn from both successes and failures. Create a culture of trust where employees feel supported in their efforts to innovate and contribute towards the organisation's vision and purpose.
2. From ‘metrics’ to systems thinking
‘Metrics’ are the perfect example of the human need to simplify what is complex. Businesses need to compartmentalise things into easily manageable chunks with easy-to-understand outcomes, and metrics seem like the perfect answer to this.
For an ecommerce website: traffic volumes are the responsibility of media or sales; conversion rate is the responsibility of the website; repeat purchases are the responsibility of the loyalty team; so on and so forth. Metrics allow management to create performance targets and to monitor teams, as well as understand the impact specific initiatives have on those metrics.
But this is all an enormous illusion!
Take ‘conversion rate’. This is nothing but a simple piece of maths which divides orders by traffic, as such it can be impacted by literally anything to do with the business, its market and competitors; consumer confidence, pricing, promotions and advertising, branding, etc. What happens to this number probably has very little to do with the website, and yet it is widely seen by senior people to be the sole indicator of performance in this area.
To be clear, there is nothing wrong with measuring metrics and even using them to evaluate changes, the problems arise when the metric is mistaken for a definite indicator of success and/or when it is assigned ownership and used as a performance indicator.
The world is just not as simple as ‘metrics’ - there are many complex systems at play which govern how actions impact other outcomes. These systems can be understood and used to create mental models of how things work and what is important.
This is not the place to go into this field in depth (there are a ton of valuable books and resources that can do that better - Google 'Systems Theory'), but the takeaway is that your people need to think in terms of ultimate, systemic outcomes which drive fundamental strategic behaviours and not metric improvements.
For example, if you make a change to your propositional statements, they might increase conversion, but how might that change someone’s expectations of the product or service? What does the change do to their relationship with you? How does that impact longevity? Will it change anything in other teams?
These things cannot easily be tracked, tested or measured, but then that’s the whole point of this article!
3. Research + human decision making
In the world of digital experimentation, the prevailing process follows a linear path of research, idea generation, and then experimentation. The experiment is meant to be the culmination of this process and is the device which makes the ultimate binary decision for you (“WIN” or “LOSE”). It is the thing which determines whether your idea was good or not and whether you should press on and do it. However, this positioning of the experiment as the decision maker is both a dangerous illusion and the root cause of virtually all the problems I outlined previously. It causes us to neglect the real thing that is actually happening: human decision making.
An experiment, in reality, cannot autonomously render decisions, nor does it ever provide unequivocal proof of a hypothesis due to all the problems aforementioned. Instead, it merely offers some information to aid in decision-making—an essential distinction that is often overlooked. In addition, when conducted properly, experiments serve as tests designed to enrich theories and contribute to broader lines of inquiry. They are but one facet of a multifaceted approach to research, alongside data analysis, customer research, and more.
At its core, decision-making remains a fundamentally human endeavour. Despite our best efforts to defer to experiments in the quest for objectivity, the responsibility ultimately falls on human shoulders. Whether through opinions and guesses or under the bad faith of "data-driven" approaches, decisions are still made by human individuals grappling with uncertainty and complexity.
The imperative, then, lies in honing our effective decision-making capabilities. We must shift the focus from the technical intricacies of controlled experimentation to the understanding and systematisation of human decision-making and its psychology. This entails:
- Re-classifying experiments as merely a form of research: Critically, recognising that controlled experiments are but one form of research, complementary but not necessarily superior to other methodologies such as data analysis or quantitative and qualitative customer research. Understanding the limitations and strengths of each approach empowers decision-makers to assess ambiguity with greater clarity. An experiment is just a piece of information, not a result or decision.
- Understanding effective decision-making: We need to invest our efforts in exploring the dynamics and psychology of effective decision-making processes, delving into topics such as bias identification and mitigation, evidence and confidence, the integration of intuition and rationality, the cultivation of confidence and consensus within teams, the logical process for progressing decisions and, most importantly, how decisions can be made against a systemic understanding of customer experience outcomes and not metric improvement. Instead of erroneously focusing on how data can give us the correct view of reality (which is never really possible), we need to focus on how the brain makes decisions based on knowledge inputs.
- Systematising this understanding into frameworks and processes: Whilst exploring the intricacies of decision making is an intellectual and theoretical endeavour, actual decision making needs to be made practical and simple for anyone in the business. We need to create frameworks and models that facilitate structured decision-making processes across the organisation. These frameworks serve as guideposts, enabling teams to synthesise research, ideas and evidence methodically, leading to well-informed and defensible decisions. Rather than just hoping people make the right decisions, we can create systems and processes that guide them.
In essence, the evolution of the ‘experimentation’ discipline should pivot towards becoming a centre of excellence for both research and decision-making support. By embracing research as a means to inform human decision-making and fostering a culture of collective critical thinking, organisations can navigate the complexities of the digital landscape with a more authentic confidence and clarity.
This strange world that we have ended up with, of experimentation programmes driving metric optimisation with their ROI forecasts instantly becomes a nonsense. Controlled experimentation is simply a type of research that supports all decisions, decisions which are made by humans in the pursuit of well-defined vision and goals.
4. The workflow of de-risked innovation and learning
Peter Drucker once (allegedly) said, "culture eats strategy for breakfast" - this has led many businesses down the path of believing that if you just get everyone to believe the same things and have the right values, then they will all automatically do the right things and you don't really need to worry about process or strategy.
Even if he did say it, then I'm pretty sure that's not what he meant and, anyway, you only have to think it through lightly to realise that it's nonsense. Culture is indeed very important, but all work needs methods and ways of doing things.
Innovation is one area that thrives on methods and processes. You could send a memo out to an entire company asking for ideas and receive perhaps thousands back in less than a day, but how do you sort through them and ensure you are surfacing the important ones? What is the process for dealing with them? Who does what? How do you manage the resource and workload efficiently? How do you manage the risk associated with bringing new ideas to market?
This all requires systems and processes, and it's one area which advanced experimentation programmes have done really well.
An 'operating system' is both a physical, software-based tool for programme management, but also a way of systematising and governing ways of working, methodologies and frameworks. It's a kind of sausage factory that enables the processing of ideas through progressive and risk-managed forms of development and validation.
This aspect of experimentation programmes is, for me, by far the most important, and yet it's rare to find anyone that has invested much time in the system. Again, this is due to the over-focus on the experimentation task itself.
Whilst there are software systems that can be bought that perform versions of this task, they generally always end up forcing you into a process which has been predetermined in generic ways. It's far better to build and evolve your own system according to your own bespoke requirements. In the experimentation world, no-code solutions like Airtable, Notion and others have become the gold standard here due to the simplicity and flexibility of being able to build such solutions.
Based on what I have already outlined here, such a system ought to be leveraged to enable the process of research, ideation, and development but, ultimately, to systematise the frameworks for effective human decision-making and critical thinking.
Conclusion
In conclusion, the landscape of digital experimentation is in serious need of reassessment and overhaul. Throughout my career, I have come to realise that current practices can be deeply flawed and often fail to deliver the results they promise, especially where unskilled resources are managing the process. The reliance on experimentation as the ultimate arbiter of decisions is misguided, as it overlooks the complexities and limitations inherent in the process.
It is imperative that we shift our focus from viewing experimentation as an end in itself, and focused only on website UX, to recognising it as just one piece of the puzzle in the broader quest for innovation and business improvement. This requires a fundamental shift in mindset, moving away from the narrow scope of optimisation towards embracing broader critical thinking as a means of strategic innovation.
To achieve this, we must reset our vision and adopt a broader, more critical and evidence-based approach to decision-making across the business. Clear objectives, company-wide alignment, and leadership by example are essential components of this transformation. We must also move beyond simplistic metrics and embrace systems thinking to understand the interconnectedness of various business outcomes.
Most importantly, we need to reframe and even demote experimentation itself to a form of research that supports human decision-making rather than relying on it as the sole determinant of success. By investing in understanding effective decision-making processes and systematising this understanding into frameworks and processes, organisations can navigate the complexities of the digital landscape with confidence and clarity.
Ultimately, the goal is to create a workflow of de-risked innovation and learning, guided by structured decision making processes and methodologies. This requires a concerted effort to build and evolve our own systems tailored to our specific needs and challenges.
In essence, by rethinking experimentation and embracing a more holistic approach to business improvement, we can pave the way for a future characterised by more authentic human innovation, growth, and sustainable success.
Building Thinking Bell | Simplifying Statistics for Experimenters | Built the Stats Engine at VWO
11moA great article Johny. I think you have articulated some very relevant points. I think scientific work has always been messy in history. It was never algorithmic like a flowchart. Science can be used to refute an idea with 100% certainty, but it can never prove anything with 100% certainty. (reference: Karl Popper, Nassim Nicholas Taleb) In the 21st century, I think it is the first time that engineers, marketers and product managers have started doing science. They are used to a much different mindset where things either work or they don't. I think it is a gradual filtering process in which the skeptics will survive and the determinists will probably give up. But experimentation will still continue to be the gold standard (as it has been in history), just that how experimenters interpret experimentation will evolve.
Data @ VML. Strategy // Marketing // Digital
1yGreat read - thanks for sharing. Ultimately as long as you are able to shift metric X by Y% with just one test - business stakeholders are going to throw money at that workstream. The key element lies in how (as you say) you use the output info and how it meshes with the rest of the efforts to improve digital properties.
Managing Director @ Kinsman & Co | Founder of Luxury Performance Agency l Scaling Brands With Digital, AI & Creative Media
1yJonny, great post, thanks for sharing!
Founder at Day1Data
1yThis makes sense. The most important feature of any model or decision is human experience. But experimentation can / should be used to help you validate these decisions & de-risk harmful strategies. (It also encourages a test & learn culture which is a very agile way of working!)
Creativity, Filmmaking, Strategy, Digital, UX, Design, Brand & Media
1yJonny- Feel free to message, I have an idea for this...