Background Quality Report - Public Awareness of and Trust in Official Statistics, Northern Ireland

Background Quality Report

Public Awareness of and Trust in Official Statistics, Northern Ireland

This report provides information on the quality of the data used to produce the official statistics: Public Awareness of and Trust in Official Statistics, Northern Ireland. It informs users about the quality of the information upon which they may be drawing conclusions and making decisions.

The report is structured around the five quality dimensions for statistical outputs (from the European Statistics Code of Practice, PDF (458 KB). The UK Statistics Authority’s Code of Practice for Statistics) requires that:

Q3.3 The quality of the statistics and data, including their accuracy and reliability, coherence and comparability, and timeliness and punctuality, should be monitored and reported regularly.

Introduction

The Public Awareness of and Trust in Official Statistics publication, produced by Northern Ireland Statistics and Research Agency (NISRA) Statistical Support Branch (SSB), presents findings on public awareness of NISRA and trust in official statistics in Northern Ireland, and examines changes over time.

The report also presents findings on trust in NISRA as an institution, compared to the Civil Service, the media and the NI Assembly (or elected bodies, such as the NI Assembly or UK Parliament). Information on the public’s opinion on the importance of NISRA statistics in understanding Northern Ireland is also included, as well as the public’s view on political interference and confidentiality.

Comparisons with the Office for National Statistics (ONS), the main producer of UK statistics, are made where there are appropriate data to do so.

Following the initial release in 2009, the report was produced biennially until 2018 with the survey carried out in ‘even’ years (2010, 2012, 2014, 2016 & 2018). Since 2019, the report has been produced annually. Due to an issue with the data, it was only possible to report on the Awareness of NISRA question in 2018.

Prior to 2018, the survey was included as a module within the NI Omnibus Survey. In 2018, the module moved to the Continuous Household Survey (CHS) due to the suspension of the NI Omnibus Survey. From 2014, the questions have been asked in the Autumn of the year.

The CHS is one of the largest continuous surveys carried out in Northern Ireland, designed and administered by the Central Survey Unit (CSU), NISRA. It is based on a representative sample of the general population, aged 16 and over, resident in private households. Prior to the COVID-19 pandemic, the CHS was based exclusively on face-to-face interviews carried out using computer-assisted personal interviewing (CAPI) where interviewers record responses to the questionnaire on tablets. During the pandemic, data collection switched to computer-assisted telephone interviewing (CATI). In July 2022, face-to-face interviewing was reinstated alongside telephone interviewing, thus making the 2022/23 CHS dual modality. This dual-mode approach continued throughout 2023/24, however, interviewers were advised that face to face interviewing should be used where possible. Interviews are spread over the 12 months from April to March, with the ‘Public Awareness and Trust in Official Statistics’ module included from September to November in 2023.

The data are used to monitor and report on awareness of NISRA, trust in NI official statistics and NISRA, as well as the public’s view on the value of official statistics, freedom from political interference and data confidentiality. This reflects the Trustworthiness, Quality and Value pillars of the Code of Practice for Statistics, which all official statistics and producers of official statistics must be compliant with.

Relevance

The degree to which the statistical product meets user needs in both coverage and content.

NISRA provides statistical services across government in Northern Ireland. Official statistics produced by NISRA are fundamental to good government, to the delivery of public services and to decision-making in all sectors of society. The Statistics and Registration Service Act 2007 states that official statistics should ‘serve the public good’ by informing the public about social and economic matters, and assisting in the development and evaluation of public policy. For the public to use official statistics effectively they must know who produces them and where to find them. The statistics in this release allow NISRA to monitor awareness levels and use these insights to inform communication and marketing strategies.

Official statistics must be produced in accordance with the Code of Practice for Statistics. The Code is structured under three pillars. Together these pillars support public confidence in official statistics.

Trustworthiness: Confidence in the people and organisations that produce statistics and data
Quality: Data and methods that produce assured statistics
Value: Statistics that support society’s needs for information

The Code explicitly states that “Those producing and releasing statistics should be free from conflicts of interest, including political pressures” (T1.2) and that “All statutory obligations governing the collection of data, confidentiality, data sharing, data linking and release should be followed” (T6.1). These statistics allow the public and NISRA to monitor the effectiveness of the processes which are in place to ensure adherence to the Code and how public perception varies over time.

T4.6 Independent measures, such as internal and external audit, peer review and National Statistics Quality Reviews, should be used to evaluate the effectiveness of statistical processes. Statistics producers should be open about identified areas for improvement.

Comparisons are made with other institutions (the Civil Service, the media and the NI Assembly) and the Office for National Statistics (ONS), the largest producer of UK statistics, where there are appropriate data to do so.

A core set of questions have been included in each survey (since 2014) allowing trends to be monitored over time. Questions have been aligned with the equivalent survey carried out by the Office of National Statistics (Public Confidence in Official Statistics and the OECD Committee for Statistics recommendations on Measuring Trust in Official Statistics.

Consultation is carried out prior to each run of the survey with key users to agree the questions to be included and to ensure they continue to cover the main requirements for users.

The statistical product includes a html publication containing charts and associated commentary, as well as detailed tables (which can be accessed either through the publication or downloaded separately) and standalone infographics to meet the needs of different users. The background to the publication, as well as key methodology and quality information is included within the publication. In addition, all data is provided in Open Document spreadsheets.

Accuracy and Reliability

The proximity between an estimate and the unknown true value.

The data in the most recent report are derived from the Continuous Household Survey (CHS). The CHS is one of the largest continuous surveys carried out in Northern Ireland. The survey is designed and administered by the Central Survey Unit (CSU), NISRA. It is based on a representative sample of the general population, aged 16 and over, resident in private households and has been running since 1983.

Sample Design and Sampling Frame

The sample for the survey consists of a systematic random sample of approximately 9,000 addresses selected from the NISRA Address Register (NAR). The NAR is developed within NISRA and is primarily based on the Land and Property Services (LPS) Pointer database which contains address information for every property in NI. The addresses on the NAR are sorted by district council and ward, so the sample is effectively stratified geographically. A starting point in the sampling frame is randomly selected and then every nth (fixed interval) address is selected by counting through the frame. As the survey is asked to a representative sample of the Northern Ireland population there are no major coverage issues. People living in institutions are excluded.

Response Rate

The target response rate for the CHS is 55%. That is, obtaining interviews at 55% of eligible addresses. Addresses which are vacant, derelict etc. and which do not contain people are deemed to be ineligible. The achieved response rate (50%) in the 2023/24 CHS was slightly lower than this target, partly due to the ongoing impact of the COVID-19 pandemic. Of the 2,340 addresses selected for interview from September to November 2023, 1,354 persons participated in this module of the survey.

CHS data collection and processing

In March 2020, due to the COVID-19 pandemic, data collection for the CHS moved from face-to-face interviewing using CAPI (computer-assisted personal interviewing) to telephone mode using CATI (computer-assisted telephone interviewing). Face-to-face interviewing was reinstated alongside telephone interviewing in July 2022, thus making the 2022/23 survey dual modality. This dual-mode approach continued throughout 2023/24, however, interviewers were advised that face to face interviewing should be used where possible. In computer-assisted interviewing, interviewers record responses to the questionnaire on tablets which allows plausibility and consistency checks to be incorporated to improve data quality. Data are returned from the field via FTP dial up. Data are then downloaded onto secure network drives and processed through the CHS data management system. Responses are coded and fully validated by CSU statistical staff.

The CHS interviews are spread over the 12 months from April to March, with the 2023 ‘Public Awareness and Trust in Official Statistics’ module included from September to November 2023.

CHS quality control and quality assurance

Questions are pre-tested before they go into field. The questionnaire is scripted in Blaise which allows CSU staff to define range and consistency checks for each question and to control routing throughout the questionnaire. The data are subject to further validation checks including treatment of outliers and detailed consistency checking including non-credible checks. A 100% check of all coded data is undertaken by statistical staff to ensure that coding procedures are of a high standard. A random sample of 10% of completed CHS interviews are back-checked with respondents to ensure that interviews are valid. Interviewers undergo intensive induction training and are trained (face-to-face) before working on the CHS. The quality of data returned by interviewers is assessed and is scored as part of their performance management.

Imputation

There is no imputation on the CHS.

Weighting

The CHS aspires to interview everyone aged 16 or over at a selected household, but non-response either by the household as a whole or by individuals within the household introduces non-response bias to the results. A chi square goodness-of-fit test shows that the CHS achieved sample is not representative of the population by age and sex when compared with the Population Estimates for Northern Ireland. Young males are particularly underrepresented. To counter this, weights are applied to analysis on the basis of age and gender. Weights are calculated based on how the age and gender profile of the survey respondents differ from that of the latest mid-year population estimates published by NISRA. The 2022 mid-year estimates (rather than the 2023 mid-year estimates) were used for 2023/24, due to the later publication of the 2023 mid-year estimates. This weighting process adjusts the results to those that would have been achieved if the sample had been drawn as a random sample of adults rather than of addresses, thus allowing inferences to be made about the population.

SSB quality assurance checks

On receipt of CHS data, SSB statisticians validate and perform checks on the data, with any issues queried with the CHS team. Range and plausibility checks are performed for returned and derived variables. The data are also checked for ‘missing values’. As part of their quality assurance processes, SSB statisticians compare the data with previous years to identify any unusual trends or distributions, or values that lie outside expected norms. All anomalies are queried with CHS staff who can re-check the coding and source data.

Sampling Error

Because CHS data are based on a representative sample rather than the whole population they are subject to sampling error. Sampling error is the difference between the estimate derived from a sample and the ‘true’ value that would result if a census of the whole population were taken under the same conditions.

Significance tests are carried out to determine if the observed differences in the responses given by various respondent groups are likely to be real differences or due to sampling error. Any statements in the report regarding changes over time or other differences are statistically significant at the 95% confidence level. This means that we can be 95% confident that the differences are actual differences and have not just arisen by chance.

Timeliness and Punctuality

Timeliness refers to the time gap between publication and the reference period. Punctuality refers to the gap between planned and actual publication dates.

The ‘Public Awareness of and Trust in Official Statistics’ survey questions are normally asked in Autumn and the report is published the following Autumn.

The delay is primarily due to the need to wait until three months after the end of the CHS survey cycle (which runs to March) for the validation of the explanatory variables to be completed and the weights produced based on the previous year’s population estimates which are usually published towards the end of June. The data are provided to SSB in mid-July and the reported is published in September/ October. For the 2023 run of the survey, the report was delayed until February 2025 due to resourcing issues.

Publication dates are pre-announced at least four weeks in advance (where possible) in the official statistics publication calendar on gov.uk and also on the ‘Public Awareness of and Confidence (Trust) in Official Statistics (PCOS)’ page on the NISRA website.

All the reports have been published on their planned publication dates, apart from the 2019 report, which was delayed by eight days to allow additional time for quality assurance. Any change to the pre-announced date is notified on both gov.uk and the NISRA website, including an explanation of the reason for the change.

Accessibility and Clarity

Accessibility is the ease with which users are able to access the data, also reflecting the format in which the data are available and the availability of supporting information. Clarity refers to the quality and sufficiency of the metadata, illustrations and accompanying advice.

The release can be accessed from the ‘Public Awareness of and Confidence (Trust) in Official Statistics (PCOS)’ page on the NISRA website, which is the same page as previous releases. It can also be accessed through the official statistics publication calendar on gov.uk

On the day of release, posts are issued from NISRA’s main social media accounts (Facebook and X) outlining some key findings from the publication, and linking to the release. A statistical press release is also made available to the media.

The statistical product includes a publication containing charts and associated commentary, as well as detailed tables including additional breakdowns (which can be accessed through the publication or downloaded separately) and standalone infographics to meet the needs of different users. All outputs are available to download free of charge.

This publication is published in html format and is compliant with the AA standard under the Web Content Accessibility Guidelines (WCAG) 2.1. The main part of the report includes key findings, followed by charts and accompanying commentary. The report is indexed meaning that users can easily find the section they are interested in. All charts have appropriate titles and labels. A ‘download data’ option is provided beneath all the charts which allows the user to download the table(s) containing the underlying data.

Appendix A includes relevant methodology and quality information, while Appendix B includes a complete list of all the accompanying tables, along with download links. All tables are provided in ODS (Open Document Spreadsheet) format. The survey questions are included in Appendix C.

The detailed tables can be accessed through the report or downloaded directly from the publication page. The publication page also includes infographics on ‘Awareness of NISRA’ and ‘Trust in NISRA Statistics’, summarising the main findings from the publication.

The contact details of the responsible statistician have been included in the bulletin, as well as in the accompanying tables.

In addition, the data from the CHS are available on the UK Data Service.

Coherence and Comparability

Coherence is the degree to which data that are derived from different sources or methods, but refer to the same topic, are similar. Comparability is the degree to which data can be compared over time and domain.

A core set of questions have been included in the survey (since 2014) allowing trends to be monitored over time. These are awareness of NISRA (which has been asked since the first run of the survey in 2009), trust in NISRA, trust in the Civil Service, trust in the media, trust in the NI assembly/ elected bodies, trust in NISRA statistics, political interference and data confidentiality, with the value question, i.e. the importance of NISRA statistics, introduced in 2016. The change in the wording of the NI assembly/ elected bodies question in 2019 and 2023 reflected the collapse of the NI assembly in 2017 and 2022, and feedback from respondents (in 2018) who queried the relevance of a question about a body that wasn’t sitting. The questions on respondent’s awareness of specified NISRA official statistics have been included since 2020. Questions on awareness of NISRA statistics were previously asked between 2009 and 2016.

For each of the key indicators (Awareness, Trust in NISRA, Trust in NISRA Statistics, Value, Political Interference and Confidentiality) the publication includes a comparison over time. Trust in NISRA as an institution is also compared with other institutions (the Civil Service, the media and the NI Assembly).

Prior to the 2023 survey results, the official statistics releases included analyses by age, employment status and highest qualification level for some or all of the key variables (Awareness, Trust in NISRA, Trust in NISRA Statistics, Value, Political Interference and Confidentiality). However, these analyses showed few significant findings, and therefore were of limited benefit. It was also apparent that much of the variation in the ‘agree’ and ‘disagree’ results for these subgroups was due to variation in the ‘don’t knows’. In order to focus on the most useful analyses, and reduce the time taken to produce the report, these analyses are no longer included.

A similar survey (Public Confidence in Official Statistics) is carried out by NatCen on behalf of the Office for National Statistics (ONS), the largest producer of UK official statistics. Where an equivalent question is asked and reported on in both surveys, the wording and response categories are aligned allowing comparisons to be made between NISRA and ONS. The Awareness of ONS/NISRA, Trust in ONS/NISRA, Trust in ONS/NISRA statistics, value, political interference and confidentiality questions are asked in both surveys.

The questions asked in the survey have also been aligned with the OECD Committee for Statistics recommendations on Measuring Trust in Official Statistics.

Impact of the COVID-19 pandemic

In March 2020, data collection for the CHS moved from face-to-face interviewing to telephone mode. This required a reduction in the number of questions in the overall questionnaire. While the Public Awareness and Trust in Official Statistics questions were unaltered, the change in data collection mode and the number of questions may have altered how people responded to the survey. In July 2022, face to face interviewing was reinstated alongside telephone interviewing, thus making the 2022/23 survey dual modality. This dual-mode approach continued throughout 2023/24; however, interviewers were advised that face to face interviewing should be used where possible.

The achieved response rate for the CHS was also lower than normal in 2020/21 (16% compared to 55%) and slightly lower than normal in 2021/22 (41%). As a result, the 2020 Public Awareness and Trust in Official Statistics module ran for an additional three months (October 2020 to March 2021). While this has maintained the precision of the 2020 estimates and enabled comparisons with 2021 and other years in the time series, the timing of the 2020 module does not align as well with other years. The achieved response rate for the CHS in 2022/23 was 51% and in 2023/24 it was 50%. More information is available on the NISRA website.

Change from NI Omnibus Survey to Continuous Household Survey (CHS)

When the NI Omnibus Survey was suspended prior to the 2018 run, the questions were moved to the CHS. Both surveys use probabilistic sampling to select a representative sample of the general population, aged 16 and over, resident in private households. This means that comparisons can be made over time and significance tests carried out to determine if observed differences are likely to be real or due to sampling error. The questions are included in the CHS for three months to maintain a similar sample size to the NI Omnibus Survey.

The ‘Public Awareness of and Trust in Official Statistics’ survey has been carried out in the Autumn since 2014. Prior to this it was carried out in the Spring run of the NI Omnibus.

Trade-offs between Output Quality Components

Trade-offs are the extent to which different aspects of quality are balanced against each other.

The main trade-off is timeliness against accuracy and comparability. Although the questions are asked in the Autumn interviews (September to November in 2023) finalised data from the CHS are not available until the following July meaning that there is a 9-13 month delay between the reference period and the results being published the following October.

Other surveys were considered but the CHS is the only survey which uses probabilistic sampling to select a representative sample of the general population, aged 16 and over, resident in private households at a similar cost to the NI Omnibus. Using the CHS means that the results can be generalised to the (household) population and that comparisons can be made over time.

Assessment of User Needs and Perceptions

The processes for finding out about users and uses, and their views on the statistical products.

Consultation is carried with key users prior to each run of the survey to agree the questions to be included and to ensure they continue to cover the main requirements for users.

Contact details for the responsible statistician have been provided in the report, and in the accompanying tables.

Performance, Cost and Respondent Burden

The effectiveness, efficiency and economy of the statistical output.

In 2020, the last year for which data are available, the CHS module took on average 3 minutes to complete. In this year, there were two additional questions asked of some respondents, so it is likely that the completion time for the module in 2023 was similar or slightly less.

Prior to the 2023 survey results, the releases included analyses by age, employment status and highest qualification level for some, or all of the key variables. However, these analyses showed few significant findings, and therefore were of limited benefit. It was also apparent that much of the variation in the ‘agree’ and ‘disagree’ results for these subgroups was due to variation in the ‘don’t knows’. In order to focus on the most useful analyses, and reduce the time taken to produce the report, these analyses are no longer included. This increases the feasibility of continuing to produce an annual report, which allows changes over time to be monitored more easily and increases the relevance of the results.

Confidentiality, Transparency and Security

The procedures and policy used to ensure sound confidentiality, security and transparent practices.

The CHS data received by SSB are anonymised (name/address information removed). In addition, the data represent a sample of the population and there is no way to identify the sample chosen from the population. The data are held within SSB’s secure record management system (CM) and on a secure drive, with restricted access and subject to approved record retention protocols.

Responses are presented as weighted percentages and there is no additional detail that would enable respondents to be identified.

Document last updated: February 2025