Measuring Educational Attainment in Longitudinal Research: Challenges and Recommendations

AN0155632998;9eb01jan.22;2022Mar10.04:40;v2.2.500

Measuring educational attainment in longitudinal research: challenges and recommendations 

Demographic data, such as highest level of education attained, are often assumed to be relatively free from measurement error. As part of an evaluation of an early childhood parenting intervention, 654 parents reported their highest level of education via telephone interview at baseline and self-directed questionnaire at follow-up 5.7 years later. At follow-up, 14% reported a lower level of education compared to baseline, indicating measurement error in one of the data collections. Comparison with data collected by an external agency for a subsample of participants (n = 261) 3.2 years after baseline indicated error in both the baseline and follow-up data. Probable causes of error included respondent and interviewer confusion regarding the names of post-school qualifications and item construction incorrectly implying linear pathways through education. We make recommendations around question construction and data collection methods for reducing measurement error in self-reported educational attainment.

Keywords: Educational status; socioeconomic factors; longitudinal studies; measurement error

Introduction

Collection of self-reported participant demographic data is ubiquitous in quantitative social research and often includes a measure of educational attainment. Along with household income and occupational prestige, education is one of three common indicators of an individual's socioeconomic position (SEP) (Li et al., [12]). Participants may be sensitive about and decline to report their income, while accurate coding of the occupational prestige of a person's job can be time-consuming. In contrast, educational attainment is considered relatively acceptable, objective and easy to measure, and as a result, is often selected as the single proxy indicator of SEP when study constraints preclude inclusion of all three indicators.

In the early childhood field, it is particularly important to measure parents' educational attainment accurately and repeatedly. Higher levels of parent education are associated with greater exposure to rich language and learning environments with positive benefits for children's development (Harding et al., [9]; Zadeh et al., [17]). In many countries, an increasing number of parents combine parenthood and child-rearing with further study (Australian Bureau of Statistics, [2]). The resulting gains in parent education have demonstrated benefits for children (Harding, [8]; King et al., [11]), especially for those of parents with initially low levels of education (Magnuson, [13]; Magnuson et al., [14]).

In this paper, we report evidence from a longitudinal study of significant measurement error in self-reported educational attainment that would have gone unnoticed in the absence of a repeated measures design. In the course of reassessing a parent cohort approximately five years after recruitment to an early childhood intervention, we found 14% self-reported a lower level of educational attainment at follow-up compared to baseline. As it is not possible for a person's highest level of education to decrease over time these data indicate measurement error at one or both timepoints. We explore this further in the current research note. Specifically, we aim to: (1) identify the common types of disagreement between participants' highest educational attainment self-reported at baseline and five-year follow-up (2) use subsample data collected from an external source approximately three years after baseline, to confirm likely errors; and (3) propose strategies for improving the measurement of education, including in longitudinal research.

Methods

Sample

Data were collected as part of a longitudinal follow-up of a randomised controlled trial of a brief parenting intervention conducted in Victoria, Australia. Parents participated in the intervention when their child was aged 1–3 years and a school-age follow-up was conducted when children were aged 7–8 years. Of 990 eligible families from the original trial, 669 (67.6%) participated at follow-up. For the current paper, 15 participants were excluded due to missing education data at T2 giving a final analytic sample of N = 654. Almost all participants were female (96.9%) and one-third (31.0%) reported speaking a main language other than English at home.

Participants reported their highest level of education at baseline prior to intervention (T1) and again at follow-up (T2) a mean of 5.7 years later (range = 4.7–7.0). Self-reported participant education was also available from an independent external source, the School Entrant Health Questionnaire (SEHQ) collected by schools when children enter the state school system at age 5 (Department of Education and Training, [7]). These data were collected approximately 3.2 years after our baseline (T1.5) and were available for 261 parents (39.9%) of the analytic sample whose children were enrolled in the state school system in Victoria and had provided consent for linkage. Ethical approval was granted by La Trobe University Human Research Ethics Committee (15–028).

Measures

To orient the reader to how questions about educational attainment were worded in our study and in the SEHQ, we provide a brief description of the terminology and classifications used in Australia. Across most Australian states and territories, formal schooling is compulsory from age 6 to 16 years, with students completing their secondary school certificate in Years 11 and 12 (Australian Trade and Investment Commission, [5]). Post-secondary qualifications are obtained either through vocational settings, or from universities. Formal classification of post-school education levels by the Australian Qualifications Framework (AQF, Australian Qualifications Framework Council, [3]) describes 10 levels of education according to the type of education provider. At vocational institutions (most referred to as TAFE; 'Technical and further education' institutions), students obtain certificate qualifications (AQF Levels 1–4) and diploma or advanced diploma qualifications (AQF Levels 5–6). Higher level qualifications are obtained through universities, including Bachelor degrees (AQF Levels 7–8) and postgraduate qualifications (AQF Levels 8–10). However, these boundaries are blurred with some vocational institutions providing degree courses in partnership with universities. Moreover, education does not always follow a sequential pathway. For example, some certificate level qualifications can be undertaken without completion of Year 12 at school.

In the current study at T1, participants completed a computer-assisted telephone interview (CATI) with a trained interviewer (Nicholson et al., [15]) who asked participants 'What is the highest level of education you have completed?' The interviewer was instructed to not read out the response options, but to code the participant's reply into one of eleven categories. The alignment between these response categories and the AQF classifications are shown in Supplementary File 1. At T2, participants were visited at home and asked to complete an iPad®-based questionnaire with the researcher available to answer any questions. To facilitate self-completion, we simplified the response options from the 11 education categories used at T1 to 5 categories at T2 (See Supplementary File 1). In the independently collected SEQH at T1.5, participants were asked what is the highest level of education completed and selected one of four response options (See Supplementary File 1) or a fifth 'other' option (Department of Education and Training, [7]).

Data analysis

The 11-category education variable collected at T1 was recoded to the five categories used at T2 (see Supplementary File 1 for the alignment of recoded categories). Descriptive statistics were used to examine patterns of change in participant-reported education between T1 and T2 using StataSE Version 14 (StataCorp, [16]). Participant education at T1.5 was available for nearly half (41.0%) of the participants for whom we observed a discrepancy in education from T1 to T2. These data were used as a means of cross-validating the study-collected data to determine whether errors were occurring at T1 or T2.

Results

Figure 1 summarises the magnitude of change in participant-reported education between T1 and T2 across the five education categories used at T2. A zero indicates no change (i.e. participant-reported educational attainment was the same at both time points). Each point away from zero represents a change from T1, with positive values denoting higher education at T2 and negative values lower education at T2. Most participants (72.8%) reported the same level of education at T1 and T2. More than a quarter reported different levels, including 18.5% varying by one level, 7.5% by two levels, and 1.2% by three or more levels.

Graph: Figure 1. Difference between educational level reported at baseline and follow-up (n = 654)

Reports of a higher level of educational attainment at T2 compared to T1 were observed for 87 participants (13.3%). As shown in Table 1 the most common increases were from 'completed school' at T1 to 'trade certificate, diploma or apprenticeship' at T2 (n = 30) or from a 'bachelor degree' at T1 to 'postgraduate qualification' by T2 (n = 20). Five participants (0.8%) had self-reported increases of 3 or 4 levels, which seems an improbable gain over a 5–6 year period. Only one of these participants had independently collected data available at T1.5 (Table 1) and this matched the T2 data. For the remainder of participants with higher education at T2, the available T1.5 data were broadly consistent with study collected data. Specifically, they either matched T1 indicating a higher qualification had not yet been completed at T1.5 or they matched T2 indicating that the higher qualification had been completed by T1.5. Moreover, there were no examples of T1.5 values outside the range provided by T1 and T2.

Table 1. Participants reporting different education at T1 and T2 (N = 178): educational status at each study timepoint and from independent data

1 * For full wording of response categories in each data collection, see Supplementary Table.

2 ^Average number of years, as explained in the sample section under Methods

Reports of a lower level of educational attainment at T2 compared to T1 were observed for 91 participants (13.9%). As shown in Table 1, the most common decreases were from a 'postgraduate qualification' at T1 to a 'bachelor degree' or 'trade certificate, diploma or apprenticeship' at T2 (n = 47). Inspection of the original, non-recoded T1 data (data not shown) revealed that more than half of these participants (n = 29) reported a graduate diploma or graduate certificate at T1. For those who reported a 'post-graduate qualification' at T1 and a 'bachelor degree' at T2 (n = 28), comparison with T1.5 data was not helpful, as this dataset did not distinguish between bachelor and postgraduate qualifications. For those who reported a 'post-graduate qualification' at T1 and a 'trade certificate, diploma or apprenticeship' at T2 (n = 19), T1.5 data were available for 10 participants (Table 1), 80% of whom reported a TAFE or trade certificate or diploma at T1.5, aligning with T2. Reports of a 'trade certificate, diploma or apprenticeship' at T1 and partial or completed high school education at T2 were also common (n = 21). T1.5 data were available for 8 participants (Table 1), 88% of which aligned with T1.

Participants who spoke a language other than English at home represented similar proportions amongst those who reported a higher level of education at T2 (n = 25, 28.7%) and those who reported a lower level (n = 33, 36.3%).

Discussion

Using longitudinal data from a cohort of Australian parents, this paper highlights complexities in measuring educational attainment, a variable that is routinely collected in social science research. Over 1 in 4 participants reported a different level of educational attainment at T2 compared to T1 (5 years earlier), illustrating the importance of repeated measurement, especially for those in the early parenting years when combining study with child-rearing is common and known to be an important predictor of children's developmental outcomes (Harding, [8]; King et al., [11]). While educational attainment appears relatively straightforward to collect, our data suggest errors occurred at one or both timepoints for around one in seven participants: 0.8% whose self-reports indicated an improbably large increase over the 5 year period and 13.9% whose reports indicated a non-feasible decrease. Comparing our study data with participant reports collected by an independent source, we found evidence of likely errors occurring at both T1 and T2 suggesting that both methods of data collection (interviewer coded at T1 and self-completed at T2) had limitations. Inspection of where likely errors occurred suggests two contributing factors: confusion over the terminology for certificate and diploma qualifications obtained via vocational versus university institutions and non-linearity in educational progression.

Changes identified as potentially erroneous often involved qualifications with overlapping nomenclature. Three of the five participants with improbable increases, and 49 of the 91 decreases were interviewer- or self-classified as having a 'postgraduate qualification' at T1 or T2. Cross-validation data tended to suggest these participants had vocational rather than post-graduate qualifications. These errors likely reflect confusion between University conferred graduate certificates and graduate diplomas which require a Bachelors degree as a prerequisite, and vocational diplomas and advanced diplomas which do not. Adding to this confusion, some graduate certificates, graduate diplomas, diplomas and advanced diplomas are offered in both vocational and university settings (Australian Trade and Investment Commission, [4]).

Another common error observed for 21 participants was the report of a trade certificate, diploma or apprenticeship at T1, and high school education only at T2. Cross-validation data supported the T1 responses. It seems likely that when self-completing a single-item measure about their highest level of education, participants stopped at the high school items, not reading further and/or not recognising that their vocational qualifications were regarded as a higher level of education. For seven of these participants, vocational qualifications were attained without completing high school, highlighting the non-linear nature of educational progression.

It is also possible that participants who grew up in another country would have more problems reporting educational status due to a lack of familiarity with Australian qualifications and equivalence with overseas institutions. We did not have a measure of country of birth available. Our next closest measure was language spoken at home and there was no evidence that those from non-English speaking backgrounds had higher error rates than other participants.

The probable errors detected here were able to be identified because we collected educational attainment data at two timepoints using different methodologies, and while we faced limitations with missing data at 1.5, this third source helped increase confidence in identifying a 'true' value. Strategies for reducing error in future data collections include: providing greater clarity between vocational and post-graduate qualifications; cognitive testing to determine how participants are interpreting and responding to these questions; and asking about school education in a separate question from vocational and university qualifications to reduce the implicit linearity in response options. The latter approach is employed in surveys conducted by the Australian Bureau of Statistics (Australian Bureau of Statistics, [1]). Another method, used in the Longitudinal Study of Australian Children (Baker et al., [6]) and the UK Millennium Cohort Study (King et al., [11]), is to directly ask participants if they have undertaken additional study since the previous interview, with a follow-up question about the qualification obtained. Alternatively, in 'proactive dependent interviewing' participants are reminded of their response from the previous data collection, before asking about their current status (Jäckle et al., [10]). This can confirm the accuracy of the previous data, as well as explicitly identifying newly attained qualifications.

In summary, this paper highlights the likelihood of measurement error in interviewer-collected and self-reported methods for assessing educational attainment. Factors possibly contributing to error include question format and respondent understanding of the educational system. Lower participant education at follow-up assessments has also been reported in studies in the United States (Harding, [8]; Magnuson et al., [14]), indicating that these challenges are not unique to the context of the current study. In all social research and particularly studies where education is employed as a single proxy indicator of SEP, our experience highlights the need for thoughtful construction of education variables and scrutiny for error.

Acknowledgments

We thank all participating parents and children; the EHLS at School staff; and the many staff involved with the original EHLS, including those from the Parenting Research Centre and the Victorian Government. The views reported in this paper are those of the authors and do not necessarily represent the views of the Victorian Government or NHMRC.

Disclosure statement

The authors have no conflicts of interest to declare.

Supplemental material

Supplemental data for this article can be accessed https://doi.org/10.1080/13645579.2020.1818415.

By J. Love; S.K. Bennetts; D. Berthelsen; N.J. Hackworth; E.M. Westrupp; F.K. Mensah and J.M. Nicholson

Reported by Author; Author; Author; Author; Author; Author; Author

Miss Jasmine Love is a Research Officer at the Judith Lumley Centre, La Trobe University.

Dr Shannon Bennetts is a Research Fellow at the Judith Lumley Centre, La Trobe University, and Coordinator of the EHLS at School Study.

Professor Donna Berthelsen is an Adjunct Professor within the Faculty of Education, School of Early Childhood and Inclusive Education at Queensland University of Technology.

Dr Naomi Hackworth is a Senior Research Specialist at the Parenting Research Centre.

Dr Elizabeth Westrupp is a Senior Lecturer and Clinical Psychologist at Deakin University.

Dr Fiona Mensah is a Senior Research Fellow in Epidemiology and Biostatistics within the Intergenerational Health Group at the Murdoch Children's Research Institute.

Professor Jan Nicholson is the Director of the Judith Lumley Centre at La Trobe University and Chief Investigator of the EHLS at School Study.