I Strongly Disagree with This Article

Hot 🔥

Anyone analyzing tests with GA4 needs to read this LinkedIn post from Lucia van der Brink and the comments. You shouldn’t be using GA4 as a primary data source. Tons of people unknowingly use GA4 data incorrectly. Rather, use it for directional insights as a secondary or tertiary data source.

This leads me to the topic that I’m obsessed with right now: testing tools vs. data warehouses. What should be your primary data source when analyzing tests? What do you do? Use a testing tool, a data warehouse, or a mix of both?

Hype ❌

I COMPLETELY DISAGREE WITH THIS ARTICLE. These types of posts actually make me upset. Yes, the debate continues. It probably will forever. Does that mean it’s not relevant or important because the debate isn’t settled? Absolutely not.

This article considers the debate from the wrong angle relative to the point they’re trying to make. Even if end-point stakeholders and clients don’t care about the nitty gritty details, that doesn’t mean that it’s unimportant and doesn’t matter.

Think about it this way instead. Different roles have different knowledge, different responsibilities, and different priorities. The CMO isn’t going to know nearly as much about statistics all the time as someone analyzing A/B tests. Just because the CMO doesn’t care about what stats model you’re using as much as the final TLDR, does that mean the road to get to the TLDR doesn’t matter? NO. That’s essentially what this article says to me (that it doesn’t matter). That’s ridiculous.

Whoever is analyzing the tests, i.e, the one responsible for understanding these concepts and applying them, should be the one who cares more and knows about the different stats models in depth. Related, they should know enough to have an opinion about it that can be supported and explained. They should understand their tools. They should make intentional choices. That means the final results are actually valid, accurate, and trustworthy with an understanding of how the results were achieved. Reiterating here, the proper stats model should still be applied intentionally. Then, when the results are presented to the CMO (for example) and action is taken based on those results, everyone has confidence to move forward and ultimately the bigger goals will be accomplished. Also, if there are questions or uncertainties, those can be answered and figured out. It’s not that the CMO must know and understand everything about the means to the end.

Faulty and/or data that is not understood will likely not accomplish the bigger goals consistently…it’s mostly plain luck when it happens.