Understanding data: making the right comparison is hard to do

Whenever new data is published commentators and analysts are prone to providing their interpretations. If a policy is implemented between two data releases, the change between the two periods is often taken as the effect of the policy. When the subject is political, fierce debates can arise.

Making such analysis accessible to the public is often difficult – and this is largely because good analysis is often complex and quite technical, and the public does not want that level of detail. What can then happen in assessing whether a policy was good or bad is to simply compare the new data to the previous data, and the changes are attributed to the policy change. For instance, the ONS reports trade data for the third quarter of 2023 (2023Q3), and these are compared with the second quarter (2023Q2).

While this comparison is simple to understand at first sight, it can be very misleading. Continuing with our example of trade data, the implicit assumption is that, in the absence of the policy, the level of trade in 2023Q3 would have been the same as in 2023Q2.

This assumption is very restrictive. Many other factors could affect trade. Economists and statisticians have thought for a long time about these issues. One way forward is to think in terms of changes rather than levels. Take the preceding example. Suppose trade has been steadily increasing before 2023Q2, one might expect trade to have also increased in 2023Q3, irrespective of the policy change. Hence, it might be better to explore whether trade increased by more than what might have been expected – in other words, we can.

Even better, by adding data for another country not affected by the policy, we can compare the changes in trade between 2023Q3 and 2023Q2 for the two countries. This will allow us to control for shocks that affect everybody in each period (see for instance Gasiorek and Tamberi, 2023). Hence, in this case, we would be seeing if the UK growth in trade between 2023Q2 and 2023Q3 was different to another, comparable, country (or countries) over that period that did not introduce the policy change, but where both countries were subject to the same shock such as the increase in gas prices because of the war between Russia and Ukraine.

At times, we see comparisons of two periods which are both after a policy was implemented. By eliminating, or not doing, the before-after implementation comparison, any interpretation of the policy becomes very difficult if not quite meaningless.

All this still leaves many difficult questions. Which is the “right” set of comparison countries? What is the best before-after period to look at as results can be quite sensitive to this? Among economists, there is a rich debate over these questions, which sometimes take a long time to resolve. But while academic time lags might be long, the world tends to move on much more quickly and needs (or wants) faster answers.

Whenever we try to answer questions of cause-effect we must make a comparison. Making the right comparison is hard. Just comparing two periods is often wrong. Drawing strong conclusions without applying rigorous techniques, and without recognizing the limitations is poor practice.