This is a repost of a blog post from 2019 that was published also at the LSE and republished at Warwick Uni.
I become increasingly sceptical of much of UK polling. There appears to be a very strange disconnect. Eurobarometer polling suggests that attitudes to the EU, if anything, appear to have become more pro-European in the run up to the EU referendum — the share of respondents stating EU membership is a good thing peaked in 2015/2016 at around 45%.
I become increasingly sceptical of much of UK polling. There appears to be a very strange disconnect. Eurobarometer polling suggests that attitudes to the EU, if anything, appear to have become more pro-European in the run up to the EU referendum — the share of respondents stating EU membership is a good thing peaked in 2015/2016 at around 45%.
Respondents stating that EU membership is a bad thing stood at around 30%, neutrals around 25%.
Going by this it suggests that EU referendum must have been won on the backs of the neutrals/negatives — but it would literally require everybody in that category to support Leave. Couple this with some lower turnout among Remain-leaners, you get your marginal Brexit referendum win.
Strange patterns in British Election Study
The Eurobarometer study, as opposed to most opinion polling conducted in the UK these days, is conducted Face to Face. Most opinion polling in the UK is done using online recruited samples – some big providers are YouGov and Survation. Both of these firms have online communities or a pool of potential survey participants which are active on their platform. Participation in surveys or polls is compensated with monetary rewards provided in various forms or others.
But opinion polling, conducted e.g. by YouGov, relies on their online community to recruit participants. This is obviously very different from asking a random set of voters on the street.
How samples are recruited on these platforms is a bit of a black box. Below Figure plots support for Leave across the 15 BES waves in which the question was asked. Excluding don’t knows and using the survey weights provided, support for Leave in the YouGov administered British Election Study never reaches the 52% of 2016. It gets close to 50% for Leave in and around waves 7, 8 and 9. Wave 7 was conducted in April/May 2016, wave 8 was conducted in May/June just prior to the referendum, while Wave 9 was conducted just after the EU referendum. Since then, support for Leave has steadily declined with Remain leading 55% to 45% — a 10 percentage point margin.
Online polling as a black box
Here is a working hypothesis: politically engaged people are more likely to be active on YouGov persistently and, they are different from the population at large. This has the potential to skew results. Is there any evidence that people active on YouGov platform consistently over time are different compared to people who are only ad-hoc participants on YouGov’s platform?
The BES panel study has been conducted since 2014 in now 15 completed waves. And, it turns out that among the 30,327 respondents that participated in wave 1 – around 2,437 or 8% have participated in each of the 14 waves in which the EU referendum question was asked. Another 2,757 have participated in 13 out of 14 waves – a combined 30% of the Wave 15 respondents participated in 10 out of 14 total waves.
But what is the distribution of Leave versus Remain recognizing the sample weights that are applied to each group? The figure below is an attempt to shed some light on this question.
The figure has many lines on it, so please bear with me. The red line represents the overall weighted average of support for Leave of people asked in BES Wave 7 conducted just before the EU referendum. The blue line is the average that would emerge without using the sample weights – this suggests that weighting helps increase headline support for Leave in that wave by around 1 percentage points.
The solid line labelled as %Leave (weighted) computes the support for Leave versus Remain across the different cohorts of participants. The share of Leave versus Remain in each bin is weighted by the corresponding survey weights. This suggests that, indeed, the more likely you are to be a repeat YouGov participant, the higher the support for Leave.
And, turning to the remaining lines, indicated on the second axis, the group of individuals make up a larger share of the sample and of the sample weights. The dashed line with diamond markers suggests that survey participants that took at least 10 out of 14 waves, make up 44% of the sample in terms of sample weights and, they make up 47% of the observations.
Among individuals that have participated in the survey at each time it was conducted support for Leave is 56.8% — among individuals that only participated in that specific wave 7 (and never again) support for Leave is just 36.9% — a whopping 19.9 percentage point difference, suggesting that indeed, people who are repeat participants of the YouGov survey are significantly more likely to support Leave vis-à-vis the respondents that are less frequent on the platform.
This pattern, suggesting that more regular repeat YouGov BES participants are more likely to support Leave is stable – up until around Wave 13. From then onwards, the sample composition seems to change and include many more new recruits who have not participated in BES before. For example, individuals who have only participated in the single Wave 14 make up Yet, the Wave 14 sample includes nearly 8% of observations that are participating in BES for the first time – and these individuals make up to 11% of the total sample weight.
The noticeable result is that the overall weighted average of support for Leave now falls below the unweighted average (red line below the blue line).
In Wave 15, this pattern continues – the sample includes 15% of observations that were participating in the BES for the first time – support for Leave in this group is just below 35% — yet, they make up 20% of the sample in terms of weights – suggesting a major change in the BES sampling and the computation of underlying weights. The sample composition change, as becomes evidence below, is the most significant driver of the changes in the support for Leave relative to previous BES waves.
Is there something characteristic about those that participate regularly?
We can study a set of socio-economic characteristics stand out among the set of respondents that participate (very) regularly in the YouGov panel. Below is a table that summarizes the main differences. Among the patterns that stand out: they are significantly older. Among the (very) regular participants, 55% are aged 55 or above. According to the UK Census from 2011, the share of people aged 55 and older is only around 30% — this suggests indeed, that the regular participants are quite odd.
Similarly, the (very) regular YouGov participants are much more likely to live in owner-occupied housing (12.5 p.p. difference), they are 22.6 percentage points more likely to be married and they are 17 percentage points more likely to be aged above 55. Similarly, they are much more likely to express to like Nigel Farage (scoring above 5 on a scale from dislike to like from 1-10). In a hypothetical 2nd referendum with the choice being No Deal or another Extension or with the choice being between Remain or No Deal, this group also stands out with a significant larger share favouring No Deal.
If we study these respondents further – the sample that consistently participated in the BES waves appear indeed, much more pro-Leave and much more pro No Deal. Using the Wave 15 data which asks how people whether they would prefer to “Delay Brexit versus No deal Brexit”, the respondents that participated at least 10 times, stress a preference in favour of No Deal of 54%. Among the respondents that participated less than 10 times, support for No Deal is only 42%.
Note these differences are all statistically significant – though it is not immediately clear how to estimate the standard errors here.
What is notable is that these differences also emerge in the weighted data using the sample weights provided. This does suggest that the (very) regular YouGov respondents may be systematically different from the non-regular YouGov respondents – and, the very regular respondents appear to make up a sizable share of the survey each wave accounting for 47% of observations.
Support for Leave stabilized by regular BES participants
The above issues imply that there is likely going to be non-trivial selection bias on a whole of unobservable dimensions. The above results suggest that repeat survey participants appear to have distinctly different views to participants that are less regular survey participants.
The question obviously is whether and to what extent this skews the results – not just in the BES but in YouGov’s polling more generally as the idea of the platform is to invite
