Further Education Outcomes: Methodology
Introduction
The Further Education (FE) Outcomes report outlines the findings from the Survey of FE College Leavers, in which leavers are surveyed approximately six months after qualifying from their course. It provides a snapshot of the destination of leavers following successful completion of their study, which helps provide evidence of the relevance of FE qualifications in achieving positive outcomes.
Generic Statistical Business Process Model
In order to produce the findings and statistical outputs, FE Outcomes follows the processes set out in the Generic Statistical Business Process Model (GSBPM). Figure 1 shows the full model, outlining all phases and associated sub processes. The model is designed to be flexible with many different routes and potential pathways and this document will outline the phases and processes undertaken to produce the FE Outcomes outputs.
Figure 1: Generic Statistical Business Process Model
1. Specify Needs
The first phase in the cycle is to ‘Specify Needs’ for statistics in a particular area. In the case of FE Outcomes, there was a need to identify what happened to leavers once they gained their FE qualification.
Why has this survey been carried out?
The Department for the Economy (DfE) is monitoring the performance of FE Colleges in terms of the quality and relevance of the curriculum to the needs of the NI economy and the learner. Therefore, there is a need to identify what happens to the thousands of people who complete and achieve their regulated FE courses. In particular, it is hoped that information on the destinations of leavers following successful completion of a programme of study will provide evidence of the relevance of this learning in achieving positive outcomes. This research will inform that process by measuring the impact and outcomes of student learning in terms of progression into further learning and employment. The research also forms a key part of the underpinning evidence for the development of future policy and the FE curriculum in terms of:
- the progression of learners to and within employment
- the progression of learners to higher levels of study
- the development of skills of use within the workplace
2. Design
The ‘Design’ phase plans out and defines all elements of the survey cycle.
Coverage
The survey and report findings cover those leavers who completed and achieved a regulated qualification at Level 3 or below in the FE Sector.The activity of those qualifying at Level 4 or above are researched separately, through their participation in HESA’s Graduate Outcomes survey.
Timing
The survey is carried out approximately 6 months after leavers achieve their FE qualification, providing a snapshot of their activity at that time. The gap between achieving their qualification and the leavers undertaking the survey is relatively short, so the contact details held should remain the same in most cases, thus encouraging better response rates for the survey.
Questionnaire
The survey questionnaire captures a snapshot of activities that leavers were engaged in approximately six months after completing their course. Details are captured on leavers’ employment, such as their mode of employment (full-time vs part-time), employment skills match and location of work. Those engaged in further study were asked where and what level they were studying. Leavers were also asked more generally about their satisfaction with their experience whilst gaining their FE qualification.
3. Build
The ‘Build’ phase entails setting up, testing and finalising the system that will collect the data.
Survey software
The current software used to issue the questionnaire and collect the data is ‘Snap Surveys’. Snap enables a census approach, providing an opportunity for all eligible leavers with up to date email contact details, to participate in an online survey.
4. Collect
The ‘Collect’ phase includes all aspects of the data collection from the initial point when the sample is established throughout the full collection process until the survey is closed.
Sampling strategy
The sampling strategy for the Survey of FE College Leavers has changed over the years, however the past 4 years have all adopted a census approach i.e. everyone in the population is surveyed.
The population for the 2022/23 survey were the 20,105 leavers who had completed and achieved a qualification at Level 3. However, this excludes
- Withdrawals
- Transfers
- Non-regulated enrolments where qualifications aren’t obtained
- Entitlement framework enrolments
- Individuals under 16 due to sensitivities involved with surveying children
- Null ULNs (Unique Learner Number), as ULN is used to ensure the same individual isn’t being contacted multiple times and thus overburdened
- Enrolments with missing level information
- Higher Education (HE) in Further Education (FE) as these leavers are surveyed on the same topics by Graduate Outcomes
- Individuals with Special Learning Difficulties and Disabilities (SLDD) taught outside mainstream college
For more information on these, please refer to Annex A - Commonly used terms and abbreviations.
Contact details were available for only 19,940 leavers, therefore, the population of interest are those students for whom contact details were provided by the FE Colleges.
A further 1,030 invitations were classified as undeliverable, either because the email address provided was incorrect, or the recipient opted out. This reduced the total eligible population to 18,910, the contactable population.
Field work
Prior to the commencement of fieldwork, the FE Colleges issue publicity emails to their leavers.
The fieldwork itself commences as email invitations are sent to all leavers in the population of interest for which email addresses have been provided. Over the course of the fieldwork period (4 weeks), there are 5 email reminders issued to leavers who had not completed the Survey.
Response Rates
Throughout the fieldwork, response rates are monitored and shared with each college to keep them informed and encourage responses from their leavers. The response rates are based on the number of survey respondents i.e. Number of individuals who were eligible and submitted a response to the questionnaire.
The final response rate was (23.5%).
Figure 2: Population of Interest to survey respondents
As shown in Figure 2, almost one-quarter (23.5%) of the total contactable population participated in the online survey, resulting in 4,440 completed responses. It was, therefore, very important to ascertain the extent to which the profile of the achieved responses differed from that of the population of interest.
In this respect, key variations among respondents included:
- 63% were female (compared with 55% in the population of interest);
- 27% were 19 and under (39% in the population of interest);
- 25% were studying full-time (34% in the population of interest).
5. Process
The ‘Process’ phase involves validating and transforming the data, so it is ready for analysis.
Data validation
The data is logic checked, and amendments were made to correct for errors in responses. The data is also cleaned to remove any duplicate responses.
Data linkage
The survey results are linked to information from the Consolidation Data Return (CDR). The CDR is comprised of data returns from each FE College. Each return consists of individual records for each enrolment recorded by the FE Colleges during the academic year. Linking to the CDR enables the results to be broken down by different personal characteristics e.g. Sex, age and categorical variables in relation to their qualification e.g. Mode and Level.
Weighting
Raked weighting is used to improve the representativeness of estimates produced from the Survey. By using an algorithm in R, raking is converged in 35 iterations using age, mode of attendance, level of study and sex, to provide a closer match between the characteristics of the participants and the population of interest.
While it is designed to improve the representativeness of survey results, it should be borne in mind non-response weighting assumes the characteristics of non-respondents are like those of respondents.
As a result of the raked weighting model:
the male share of respondents increased from 37% to 45%, whereas the female share decreased from 63% to 55%, both in line with their respective population shares.
the 25 and over share of respondents decreased from 63% to 48%, in line with their share of the population of interest.
the level of study proportions also aligned with their respective population shares.
The result of the raked weighting model for all categorical variables is shown below.
Table 1: Sex Comparison: proportions for the population of interest and survey respondents
| Variable Categories | Percentage of population (N=18,910) | Survey Respondents before weighting | Percentage of survey respondents before weighting | Percentage point difference before weighting | Survey respondents after weighting | Percentage of survey respondents after weighting | Percentage point difference after weighting |
|---|---|---|---|---|---|---|---|
| Female | 55% | 2796 | 63% | -8 | 2445 | 55% | 0 |
| Male | 45% | 1643 | 37% | 8 | 1994 | 45% | 0 |
| Total | 100% | 4439 | 100% | 0 | 4439 | 100% | 0 |
Table 2: Age Comparison: proportions for the population of interest and survey respondents
| Variable Categories | Percentage of population (N=18,910) | Survey Respondents before weighting | Percentage of survey respondents before weighting | Percentage point difference before weighting | Survey respondents after weighting | Percentage of survey respondents after weighting | Percentage point difference after weighting |
|---|---|---|---|---|---|---|---|
| 19 and under | 39% | 1195 | 27% | 12 | 1719 | 39% | 0 |
| 20-24 | 14% | 430 | 10% | 4 | 611 | 14% | 0 |
| 25 and over | 48% | 2814 | 63% | -15 | 2109 | 48% | 0 |
| Total | 100% | 4439 | 100% | 0 | 4439 | 100% | 0 |
Table 3: Mode Comparison: proportions for the population of interest and survey respondents
| Variable Categories | Percentage of population (N=18,910) | Survey Respondents before weighting | Percentage of survey respondents before weighting | Percentage point difference before weighting | Survey respondents after weighting | Percentage of survey respondents after weighting | Percentage point difference after weighting |
|---|---|---|---|---|---|---|---|
| Full-time | 34% | 1117 | 25% | 9 | 1513 | 34% | 0 |
| Part-time | 66% | 3322 | 75% | -9 | 2926 | 66% | 0 |
| Total | 100% | 4439 | 100% | 0 | 4439 | 100% | 0 |
Table 4: Level of Study Comparison: proportions for the population of interest and survey respondents
| Variable Categories | Percentage of population (N=18,910) | Survey Respondents before weighting | Percentage of survey respondents before weighting | Percentage point difference before weighting | Survey respondents after weighting | Percentage of survey respondents after weighting | Percentage point difference after weighting |
|---|---|---|---|---|---|---|---|
| Entry Level | 5% | 355 | 8% | -3 | 242 | 5% | 0 |
| Level 1 | 7% | 293 | 7% | 0 | 290 | 7% | 0 |
| Level 2 | 40% | 1753 | 39% | 1 | 1786 | 40% | 0 |
| Level 3 | 48% | 2038 | 46% | 2 | 2121 | 48% | 0 |
| Total | 100% | 4439 | 100% | 0 | 4439 | 100% | 0 |
6. Analyse
The ‘Analyse’ phase is the production of the statistical outputs for dissemination. This includes preparing the commentary, technical notes and ensuring the outputs are fit for purpose.
Prepare outputs
For FE Outcomes Academic year 22/23, the outputs consist of a HTML Bulletin that outlines the main findings from the survey results and a dashboard detailing the results and enabling the user to drill down into more granular detail using a series of filters.
The HTML Bulletin was created using an accessibility template, to ensure it meets the current accessibility guidelines. It details the outcomes for FE leavers once they have gained their qualification, providing commentary, tables and charts to bring the results to life.
The dashboard is created for users who want to know more than just the headline statistics. Users can explore each section of the HTML e.g. employment, learning etc, in more detail by using the filter options to find out more about the various cohorts of leavers:
- College
- Level of Study
- Mode of Study
- Broad STEM
- Narrow STEM
- Apprenticeship
- ESOL (English for Speakers of Other Languages)
- Essential Skills
- Subject Area
- Sex
- Age Band
- Disability
- Deprivation Quintile
The information detailed in both outputs are quality assured prior to release.
Disclosure control – Rounding Strategy
FE Outcomes conforms to the HESA rounding strategy to ensure the data disseminated does not breach confidentially rules. Due to the provisions of the Data Protection Act (2018) and the Human Rights Act (1998), HESA (and therefore DfE) implements a strategy, in published and released tabulations, designed to prevent the disclosure of personal information about any individual. This strategy involves rounding all numbers to the nearest 5, and suppressing percentages and averages based on small populations.
A full list of the rounding strategy includes:
- 0, 1 and 2 are rounded to 0;
- all other numbers are rounded to the nearest multiple of 5;
- percentages based on fewer than 22.5 individuals are suppressed;
- averages based on a small group of people (7 or fewer) are also suppressed;
- percentages and averages are based on unrounded figures;
- percentages are rounded to the nearest integer
- Total figures are also subject to this rounding methodology; so the sum of numbers in each row or column may not match the total shown. Similarly, percentages may not sum to 100%.
7. Disseminate
Produce products
The FE Outcomes outputs are available on the DfE website and this includes links to the bulletin and dashboard.
The target audience for the release is our colleagues across the Department for the Economy (DfE), Directors of FE colleges in Northern Ireland, Board members of FE colleges, educational professionals, academics, media and members of the public interested in the FE sector.
Interpreting the findings
When interpreting the findings presented, it is important to be aware of the following points:
Individuals in FE Colleges can enrol and achieve on multiple courses simultaneously. The methodology used to determine the population of interest was to select the highest level of regulated qualification (up to and including Level 3) achieved for each individual, which resulted in 20,105 individual FE College leavers being eligible for inclusion in the 2022/23 Survey.
Although it was intended that everyone who completed a qualification would be contacted and provided an opportunity to respond to the survey, not all had valid contact details, leaving a population of interest of 19,940. Of those participants who responded, not all answered all the questions. Therefore, variation occurs in the total number of responses provided for some questions. This affects the statistics derived from the survey when trying to infer patterns within the population of interest.
Inferences/statements about the population of interest can be made because the profile of survey respondents has been re-weighted by the profiling variables (age, mode of attendance, level of study and sex), so that it is broadly similar to that of the population of interest in respect of these key variables (Refer to Tables 1 to 4).
Due to rounding some figures may not add to 100%; differences reported may not equate to the difference between the two figures shown in the text; and summing of proportions may not equate to the value reported.
8. Evaluate
Review
Following a review of the Survey, this year marked the introduction of a new questionnaire. Underpinned by DfE’s Economic Mission, new questions were introduced on work quality, skills match for employment and learning, as well as regional balance of employment. As a result, the outputs have been updated to report on these valuable metrics and provide insight for policy and decision makers.
Feedback
To ensure we are providing relevant information and presenting this in a user friendly format, we are always keen to get feedback. All user engagement is important to us, so please let us know your thoughts in the feedback form below!
Annex A - Commonly used terms and abbreviations
Academic year: 1st August to 31st July.
Course: A course leading to a regulated qualification at Level 3 or below.
ESOL: An ESOL (English for Speakers of Other Languages) course is for students who want to improve their speaking, listening, reading and writing skills and for whom English is not their first language. Further information can be found on the ESOL page of the NI Direct Website.
Further Education sector: The six FE Colleges in NI -
- Belfast Metropolitan College (BMC);
- Northern Regional College (NRC);
- North West Regional College (NWRC);
- South Eastern Regional College (SERC);
- Southern Regional College (SRC); and
- South West College (SWC).
Leavers: FE College leavers who completed and achieved a regulated qualification at Level 3 or below during the 2021/22 academic year.
Leavers in employment: This includes: working in paid employment; self-employed; and waiting to start a confirmed job.
Leavers in learning: This includes: on a scheme/training scheme; waiting to start a confirmed scheme; at school in sixth form; at FE College; at university; or waiting to start a confirmed course at college/university.
Leavers unemployed: This includes FE College leavers who were unemployed and looking for work.
Leavers classed as other: This includes any FE College leavers not included in employment,learning or unemployed.
Main activity: Respondents were asked to select the activity they spent the majority of their time doing both before and after their course.
Main activity group: The four main activities (above) are grouped into four main activity groups: employment; learning; unemployed; and other.
Qualification level: Each regulated qualification has a level between Entry Level and Level 8. Qualifications at the same level typically have a similar level of demand or difficulty. The content and size of qualifications at the same level may be quite different. For further information on comparing qualifications across the UK, Republic of Ireland and overseas please refer to the GOV.UK ‘what different qualifications levels mean’ webpage.
Regulated Qualification: Any course at Level 3 or below offering a qualification that has a qualification code on the Register of Regulated Qualifications (RRQ) or any course at Level 4 or above.
SLDD: Special Learning Difficulties and Disabilities courses. These require specialised teaching, and are often taught in a reduced class size or in a day centre.
Sector Subject Area: Sector Subject Areas (SSAs) were developed according to industry sector and academic subject area by the Qualifications and Curriculum Authority (QCA), the Council for the Curriculum, Examinations and Assessment (CCEA) and the Department for Education and Skills (Wales), along with other key educational bodies. They aim to allow more consistent reporting of data across the UK. SSA codes are derived from the OFQUAL Register of Regulated Qualifications, based on the qualification or unit code. The SSA code registered against each qualification or unit has been assigned by the responsible awarding organisation. Further information can be found on the OFQUAL guidance on qualification descriptions.
ULN: Unique Learner Number. This is a unique code used to track each student through their Learning Cycle and is defined by the Learner Records Service (LRS). It is generated by education providers and held by the Learner Register.