JP Castlin posted an interview with marketing and media analyst Andrew Willshire titled Models & Mistakes (10 February 2023). Topics include forecasts, models, and econometrics.
On forecasts:
« four tightly linked stages – Measurement, Data, Models, Decisions. »
« The place to start (and where people seldom do) is to examine the decision that the forecast is trying to inform. What do you actually want to know and why? What decisions are even open to you, practically and politically within an organisation? Imagine the full range of forecasts that could be presented to you, and what you would actually do differently depending on what they say. Where are the breakpoints in your resulting decision tree? »
« It might not be possible to get exactly what you want, but by at least thinking about it, you can consider what the closest available proxy is, and also perhaps take steps to gathering certain data intentionally from this point forwards so that next year, you have something closer to the data that you actually want. »
« You don’t have to have the forecast first to figure out what to do. Russ Ackoff, in his book Management f-Laws states that “The future is better dealt with using assumptions rather than forecasts… Forecasts are about probabilities; assumptions are about possibilities. We carry a spare tyre in our cars not because we forecast that we will have a puncture on our next trip but because we assume a flat tyre is possible.” »
« So, if you want to “play it safe”, you create an organisation that knows how to respond to any forecast, and can swiftly adapt when the forecast inevitably turns out to be not quite right. »
On models:
« Don’t confuse the messy world of aggregate human behaviour with the tidy world of physics… It is extremely rare for models used in business and marketing to actually replicate the underlying physical processes, unlike, e.g., modelling fluid dynamics or electrical current flow. »
« This means that the models are much more limited in predicting outside of the range of past experience… They are often entirely useless when it comes to forecasting the impact of events that have never previously been experienced. »
On econometric modeling:
« you never really step in the same river twice, especially when so much is constantly changing. So while the technique is pretty good at telling you what was effective in the past, there’s no guarantee that the same approach will definitely be effective in the future, especially when the agency has changed the creative, the audience, the targeting criteria, etc., while the world has also changed in terms of economics, fashions and other trends. »
« Beware practitioners who always seem certain that they’re revealing some divine truth to you. Their models have the same uncertainty in them, they’re just not being open about it. »
Articles of Interest 