There’s still something not quite right about the ARINS survey: Part Two (Surveys & Elections)

Colin Coulter is Professor of Sociology in Maynooth University. Today he continues to ask questions about the recent ARINS survey, and argues the case for making source data public and allowing legitimate criticisms into the public square.

In the first part of this essay, we suggested that most of the shift in constitutional aspirations recorded in the latest ARINS/Irish Times survey may, in fact, originate in the inappropriate weightings applied to ‘community background’ in its two previous editions. But what of the remainder, the 4% that cannot be simply attributed to shortcomings and inconsistencies in survey design?

The simplest, perhaps most likely, explanation is that the figure gives us a more plausible estimation of the level of ‘real’ change in popular opinion that has occurred over the last couple of years. And that may well prove to be the case. There are, however, at least two connected considerations that would counsel caution here.

The first is that there are no clear conditions that might have initiated such a rise in support for unification in such a short space of time. By the time the field work for the 2024 ARINS/Irish Times survey began, after all, the Stormont executive had been restored and its principal figures were, at the level of appearances at least, getting along famously; the prospect that Sinn Féin might form a government in Dublin had become more distant, mid-term at best; and the anxieties that accumulated in the immediate wake of Brexit had receded for most people living in Northern Ireland. It is simply hard, therefore, to envisage what it is about this specific period that might have reanimated nationalist passions on the scale that the ARINS/Irish Times survey would have us believe.

The outlier becomes the norm

That sense of mild incredulity only increases when we come to consider the other major opinion polls published in the lifetime of the ARINS project. In Figure 1 below, we have grouped together all of the surveys, all but one of them series, that have taken place since 2022. The numbers reflect the balance of constitutional aspirations in Northern Ireland before the “don’t knows” are excluded from our calculations.

Figure 1: % Support for Remaining in UK Minus Support for Leaving, All Polls, 2022-25

In a recent, fascinating Slugger article, Philip McGuinness noted that after the initial narrowing that followed Brexit, the unionist lead in opinion polls widened again at the turn of the decade and would remain more or less stable thereafter. Or, at least, that was the case until about a month ago. There have, of late, been two surveys registering an apparent surge in support for Irish reunification north of the border. The first to appear was, of course, the ARINS/Irish Times poll indicating a 7% net rise in support for a united Ireland in the last year alone. Then came the latest LucidTalk survey showing the same trend but on a lesser scale, with a 3% fall in the lead enjoyed by those who aspire to remain in the UK.

The patterns that McGuinness discerns in his article are broadly borne out in Figure 1 above. Looking across the thirteen surveys conducted since 2022, it is immediately apparent that those who wish to remain in the UK continue to outnumber those who aspire to a united Ireland. In percentage terms, their numerical advantage typically spans the low to mid-teens. There are, inevitably, variations between years, but most surveys tend to fall into that range.

The biggest exception to that rule is, clearly, the ARINS/Irish Times survey. That is the poll with by far the largest divergence in its findings over time. No other research comes close to showing a 7% net shift in constitutional aspirations in the space of just twelve months.

The ARINS survey is also the only one to record relative support for the Union at levels much higher than the 14% figure that represents the average across all the surveys published during its existence. That is hardly surprising, of course, given the excess weighting given to the Protestant community in its first two editions

Both of those observations suggest that the ARINS/Irish Times poll is, in fact, an outlier. Or at least it was until now. That claim should not be taken to imply that the 4% ‘real’ change unearthed by the latest edition of the survey is not actually happening. It might well be.

But, if it is, that means that the ARINS academics are divining trends that remain undetected by almost every other major opinion poll conducted during the period. And that, in turn, means that we should be a little reticent about accepting their findings at face value.

Another serious weight issue

If it does transpire that the ‘real’ shift in public sentiment documented in the ARINS/Irish Times poll is not really happening – or, at least, not on quite that scale – how then are we to account for it? One potential explanation may to be found in another critical error in survey design.

It is worth remembering that almost all of the shift in popular opinion discerned by the ARINS team occurred in the space between 2023 and 2024. If we examine the sample profiles for the those two years, however, another very peculiar weighting decision comes to light.

One of the ways in which the ARINS/Irish Times poll stratifies its respondents is in terms of electoral preference. In Table 5 below, we set out the weightings ascribed to supporters of the principal parties in the 2023 and 2024 editions of the survey and compare them to voting trends in elections held in those years.

Table 5: Weightings of Voting Preference (as % of eligible electorate, not votes cast) ARINS/Irish Times Survey, 2023-24

As we can see, the 2023 instalment overstates support for both the main nationalist and unionist parties, with the gap between them around a percentage point higher than would have been warranted on the basis of the local government elections that year. That was, of course, the first, and still only, occasion when candidates advocating constitutional change outperformed those in favour of the status quo.

When we turn to the profile of respondents in the 2024 survey, a very different picture emerges. In the Westminster election that year, the number of votes polled by candidates standing for the main unionist parties was 1.3% higher (as a proportion of the total eligible electorate) than the number received by representatives of the main nationalist parties.

Nonetheless, the ARINS team decided not only to retain the higher weighting given to nationalist voters but to accentuate it even further. Respondents who support Sinn Féin or the SDLP are now some 4% more numerous than those favouring the DUP, UUP or TUV.

In effect, therefore, those who weighted the 2024 survey have introduced, in the space of just one year, a net shift in the balance of electoral forces in Northern Ireland of 5.3% in favour of the nationalist community. Without access to the relevant database, it is hard to arrive at a precise sense of how that change might have altered the outcomes of the most recent ARINS/Irish Times poll. Its impact is likely, however, to have been substantial, accounting for at least some, perhaps even most, of the 4% ‘real’ change in public opinion that has occurred across the series as a whole.

The future flatlines

There remains, therefore, something not quite right about the ARINS survey. Its most recent edition grabbed headlines for seeming to show a major tilt towards the reunification project. But those changes tell us rather more about the original flaws in the design of the survey than they do about genuine shifts in constitutional preferences among the Northern Irish public.

The first couple of instalments of the ARINS survey applied strikingly skewed weightings for religious affiliation to the sample population. Once those errors were removed, two things were, simply, always going to happen.

The first is that there would be an increase in support, both relative and absolute, for the project of Irish reunification. The second is that the series would, finally, converge with the wider polling trend in showing that those who wish to remain in the UK currently enjoy a lead in percentage terms in the low to mid-teens. The outlier, quite simply, has become the norm. So there’s really nothing much to see here.

That feeling is only compounded when we consider that the ARINS/Irish Times poll draws upon the views of just 1,000 members of the Northern Irish public. A sample size that modest means that seemingly profound trends may, in fact, hinge upon the changing moods of only a few dozen people. The Irish Times is literally devoting tens of thousands of words to an opinion poll whose outcomes may have been determined by the equivalent of two or three rugby or camogie teams.

And yet, by far the most substantial survey of recent years – the project based in Liverpool and led by Professor Jon Tonge, with a sample of more than 5,000 respondents – continues to receive barely any media coverage. That really does make you wonder.

That the latest ARINS survey really just shows something very similar to what most other current polls are finding poses a clear problem for the project of which it is part. Now that the most glaring weighting error (‘community background’) has been resolved, it is entirely possible that future instalments will show little change in popular opinion on the constitutional question. And, if the other puzzling sampling anomaly (electoral preference) were to be corrected, it is possible that editions to come might even register small shifts towards a unionist position.

Both those scenarios would represent a real headache for the various players who have heralded the outcomes of the latest ARINS survey. It is going to be interesting to see how Ireland’s Future can accommodate an alternate reality – one where most of the rest of us reside already – in which the momentum towards a border poll has waned, or even reversed. It will be more interesting still to see how the Irish Times is going to stretch out data showing support for reunification has flatlined into several weeks of what will, doubtless, be thoroughly dispassionate commentary.

What is to be done?

In the current debates about the prospect of constitutional change, the ARINS/Irish Times series enjoys a profile rather higher than any other survey of popular opinion. That prominence rests largely on the assumption that the data arising from the project are the product of impeccable research practice. On closer examination, however, that turns out not to be the case.

The data emerging from the project are, in fact, the product of multiple inconsistencies and sundry arbitrary decisions. There are so many issues that ARINS, and their partners the Irish Times, need to address that it is hard to know where to start. But let’s start with these:

1. The ARINS team needs to explain clearly, and publicly, why the ‘community background’ weightings are so very different across the three years.
2. They need also to explain clearly, and publicly, why their weightings for voting preferences are so at odds with the actual voting patterns in the relevant years.
3. The ARINS project should, as a matter of urgency, release all of the datasets so that others can see for themselves the impact of their weighting decisions.
4. The Irish Times should open rather than close down the debate about the ARINS survey data. The unwillingness of the newspaper to publish letters asking serious, and reasonable, questions about a project of which it is part does very little to allay those legitimate concerns.

 That really isn’t too much to ask, surely?

This is a guest slot to give a platform for new writers either as a one off, or a prelude to becoming part of the regular Slugger team.