Validation studies

This page lists publications and reports that provide information on the validity of data collections that are represented in the Centre for Health Record Linkage (CHeReL) Master Linkage Key.

NSW Admitted Patient Data Collection (APDC)

1.1 Australian Institute of Health and Welfare. Expenditures on health services for Aboriginal and Torres Strait Islander people, 1998-99. AIHW cat. No. IHW 7. Canberra: AIHW, 2001.

This report estimates that indigenous status was under-reported by 30% (range 26% - 34%) in the NSW APDC in 1998-99.

1.2 Taylor LK, Travis S, Pym M, Olive E, Henderson-Smart DJ. How useful are hospital morbidity data for monitoring conditions occurring in the perinatal period? ANZ J Obstet Gynaecol 2005;45(1):36-41.

This publication presents the results of a study that compared 1,000 NSW APDC records with information sourced from the actual medical record for that admission. There was a wide range of sensitivities among the 41 conditions selected, indicating a variable level of under-enumeration. Only 17 (41%) had a sensitivity of 80% or more. Sensitivities were more than 95% for gestational diabetes, prolonged pregnancy, perineal laceration, single live birth, low birth weight, and intracranial haemorrhage. Specificities were generally high. Neonatal conditions tended to be more accurately reported than maternal conditions.

1.3 Centre for Epidemiology and Research. NSW Department of Health (now known as the Ministry of Health). NSW Mothers and Babies 2004. NSW Public Health Bull 2005;16 (S-4).

Results of a validation study are presented in Chapter 13 of the report: How useful are hospital morbidity data for monitoring conditions occurring in the perinatal period? The results of this study were originally published in the ANZ Journal of Obstetrics and Gynaecology (see 1.2 above). The chapter includes additional and extended tables of results. The NSW mothers and Babies 2004 report may be found on the NSW Ministry of Health website.

1.4 Roberts CL, Bell JC, Ford JB, Hadfield RM, Algert CS, Morris JM. The accuracy of reporting of the hypertensive disorders of pregnancy in population health data. Hypertens Pregnancy 2008;27(3):285-97.

Validation data for 1200 women were linked to both hospital discharge and birth databases. Hypertension could be reported in one, both, or neither database. Of the 1184 records available for review, 8.3% of women had pregnancy-related hypertension and 1.3% had chronic hypertension. Reporting sensitivities ranged from 23% to 99% and specificities from 96% to 100%. Using broad rather than specific categories of hypertension and more than one source to identify hypertension improved case ascertainment.

1.5 Roberts C, Algert C. Antenatal transfer of rural women: how does the NSW Inpatient Statistics Collection compare with an audit of hospital records? Aust Health Rev 2000;23(3):78-82.

The aim of this study was to determine whether the NSW Inpatient Statistics Collection (ISC), a census of hospital admissions, could be used to estimate the magnitude of, and reasons for, antenatal transfer of rural women. Data from the ISC were compared with results of a clinical audit of all antenatal admissions of rural women to perinatal centres in NSW during 1997-1998. While the overall number of perinatal centre admissions identified by the ISC and the audit were similar, the ISC identified only about 70% of antenatal transfers. Rural hospitals identified 12% of women as indigenous compared with 9% at perinatal centres. The ISC showed 28% of rural women admissions and 42% of transfers were for threatened preterm labour compared with 21% and 30% respectively from the audit.

1.6 Hadfield RM, Lain SJ, Cameron CA, Bell JC, Morris JM, Roberts CL. The prevalence of maternal medical conditions during pregnancy and a validation of their reporting in hospital discharge data. ANZ J Obstet Gynaecol 2008;48(1):78-82.

This study compared medical records from a random selection of New South Wales (NSW) women who gave birth in a NSW hospital in 2002 with coded hospital discharge records. The researchers estimated the population prevalence of maternal medical conditions during pregnancy and found a tendency towards underreporting although specificities were high, indicating that false positives were uncommon.

1.7 Lain SJ, Roberts CL, Hadfield RM, Bell JC, Morris JM. How accurate is the reporting of obstetric haemorrhage in hospital discharge data? A validation study. Aust N Z J Obstet Gynaecol. 2008 Oct;48(5):481-4.

This study aimed to assess the accuracy of hospital discharge data in identifying obstetric haemorrhage diagnoses and procedures. The medical records of 1200 randomly selected women were reviewed and compared with obstetric haemorrhage diagnoses and procedures in the hospital discharge data. Sensitivity, specificity, and positive and negative predictive values were calculated using the medical records as the 'gold standard'. Estimated population incidence for any antepartum haemorrhage was 1.8 per 100, and post partum haemorrhage was 7.2 per 100 women. Obstetric haemorrhage diagnosis and procedure codes tended to be underreported, with sensitivities ranging from 28.3% to 100%. All codes had specificities of 98.9% or greater. The identification of obstetric haemorrhage differed between levels of severity.

1.8 Lain SJ, Hadfield RM, Raynes-Greenow CH, Ford JB, Mealing NM, Algert CS, Roberts CL. Quality of data in perinatal population health databases: a systematic review. Med Care. 2012 Apr;50(4):e7-e20.

This meta-analysis of 43 validation studies found that under-enumeration was common among perinatal population health databases and that hospital discharge data were generally more accurate than birth data.

1.9 Ford JB, Algert CS, Kok C, Choy MA, Roberts CL. Hospital Data Reporting on Postpartum Hemorrhage: Under-Estimates Recurrence and Over-Estimates the Contribution of Uterine Atony. Matern Child Health J. 2011 Nov 23. [Epub ahead of print]

This study aimed to explore whether recording of a prior adverse pregnancy outcome (postpartum hemorrhage) in a medical record increases the likelihood that recurrence of the same event is reported in hospital data. Using a sample of 588 pregnancies [2 consecutive pregnancies for 294 randomly selected women with at least one postpartum hemorrhage (PPH)], the study compared 'coded' recurrence rates in hospital data with those obtained from medical record audit. 'Coded' recurrence in a second pregnancy was also compared for women with or without a documented history of prior PPH. The study found a 'coded' recurrence rate of 18.5% and an 'audited' recurrence rate of 28.4%. The 'coded' rate of recurrence among women who had a documented history of PPH was 27.4% compared to 19.1% when the previous PPH was not noted in the second pregnancy medical record. Medical record reporting of uterine atony as the cause for postpartum hemorrhages in first and second births was 37.9 and 34.0% while 'coded' hospital data reporting attributed 79.8 and 73.9% respectively to atony. The study results indicate that a history of postpartum hemorrhage may be a stronger risk factor for subsequent PPH than previously demonstrated. The contribution of uterine atony as a cause of postpartum hemorrhage is over-estimated using hospital data.

1.10 Marsden DL, Spratt NJ, Walker R, Barker D, Attia J, Pollack M, Parsons MW, Levi CR. Trends in stroke attack rates and case fatality in the Hunter region, Australia 1996-2008. Cerebrovasc Dis. 2010;30(5):500-7.

This study assessed stroke attack rates in the Hunter Region of NSW and found that rates decreased from 1996 to 2008. There was a 97.5% proportion of agreement between audit and hospital discharge coding.

1.11 Lam MK. How good is New South Wales admitted patient data collection in recording births? HIM J. 2011;40(3):12-9.

This study found that out of a total of 79,173 confirmed hospital births that were matched to corresponding records in the APDC; 2,249 (3%) confirmed hospital births were not found in the APDC. For matched records, 95% and 99% of records were found to be coded consistently between the APDC and MDC datasets for outcome of delivery and discharge status respectively. With a high level of coding concordance between the APDC and MDC datasets and only a small percentage of hospital births not being recorded in the APDC, the obstetrics subset of the APDC dataset was found to be of good quality.

1.12 Roberts CL, Ford JB, Lain S, Algert CS, Sparks CJ. The accuracy of reporting of general anaesthesia for childbirth: a validation study. Anaesth Intensive Care. 2008 May;36(3):418-24.

This study aimed to determine the accuracy of the reporting of peripartum general anaesthesia in single and linked population health data. Data from a statewide validation study of 1200 women provided the gold standard for delivery and postpartum general anaesthesia use. Among the 1184 records available for review, 7.7% of women had a general anaesthetic during the birth admission, of which 6.2% were for delivery and 1.5% were for a postpartum procedure. Reporting sensitivities ranged from 26% to 97% and specificities 94% to 100%. Identifying general anaesthesia from either the birth or hospital data improved general anaesthesia ascertainment. Limiting analysis to caesarean sections resulted in very accurate identification of general anaesthesia for delivery (sensitivity 97.0%, specificity 99.8%) while limiting to vaginal births was moderately accurate for identifying postpartum general anaesthesia (sensitivity 73.2%, specificity 99.8%).

NSW Perinatal Data Collection (PDC - formerly the Midwives Data Collection, MDC)

2.1 Centre for Epidemiology and Research. NSW Department of Health (now known as the NSW Ministry of Health). New South Wales Mothers and Babies 1998. NSW Public Health Bull 2007;9(S-2).

Chapter 8 of the report "Validation Study - NSW Midwives Data Collection 1998" presents the results of a validation study carried out on a random sample of 1,680 records from the 1998 MDC. Information collected on the MDC record was compared with the medical record. Percentage agreement, kappa statistics, sensitivity and specificity for each variable are presented. The report may be found on the NSW Ministry of Health website at: www.health.nsw.gov.au/pubs/2000/mds98.html.

2.2 Centre for Epidemiology and Research. NSW Department of Health. New South Wales Mothers and Babies 2005. NSW Public Health Bull 2007;18(S-1).

Records of births reported to the Midwives Data Collection (MDC) are linked to birth registration records from the NSW Registry of Births, Deaths and Marriages to examine the quality of reporting of Aboriginality to the MDC. This information is published annually in the NSW Mothers and Babies Report. The percentage of births to Aboriginal and Torres Strait Islander mothers reported to the MDC ranged from 65 to 70 per cent between 2001 and 2004.

2.3 Roberts CL, Bell JC, Ford JB, Hadfield RM, Algert CS, Morris JM. The accuracy of reporting of the hypertensive disorders of pregnancy in population health data. Hypertens Pregnancy 2008;27(3):285-97.

Data from a validation study of 1200 women provided the 'gold standard' for hypertension status. The validation data were linked to both hospital discharge and birth databases. Of the 1184 records available for review, 8.3% of women had pregnancy-related hypertension and 1.3% had chronic hypertension. Reporting sensitivities ranged from 23% to 99% and specificities from 96% to 100%.

2.4 Lam MK. How good is New South Wales admitted patient data collection in recording births? HIM J. 2011;40(3):12-9.

2.5 Xu F, Sullivan EA, Madden RC, Black D, Jackson Pulver LR. Improvement of maternal Aboriginality in NSW birth data. BMC Med Res Methodol. 2012 Jan 30;12:8.

This study was based on linked birth data from the Midwives Data Collection (MDC) and the Registry of Births Deaths and Marriages (RBDM) of New South Wales (NSW). Data linkage was performed by the Centre for Health Record Linkage (CHeReL) for births in NSW for the period January 2001 to December 2005. The study found that maternal Aboriginal status was under-ascertained in both the MDC and RBDM. The proportion of births to Aboriginal mothers in the non-registered birth group was significantly higher than in the registered group.

2.6 Dwyer C, Achat H M. Examination of the completeness of routinely collected antenatal care data in New South Wales. Aust N Z J Public Health. 2013 Apr;37(2):187-8. doi: 10.1111/1753-6405.12039.

This investigation examined the completeness of information about antenatal care prior and subsequent to the introduction of a revised Perinatal Data Collection (PDC) form in 2006. Prior to 2006, the 'ancare' variable in the PDC captured only the number of weeks of pregnancy at the first antenatal visit; the PDC did not provide a yes/no response option to antenatal care access. In 2006, a new information collection form introduced two questions about antenatal care; the 'anc' variable allows a yes/no response to whether antenatal care was received and another variable identifies the model of antenatal care. The 'anc' variable appears to enable more comprehensive reporting of the occurrence of antenatal care than the 'ancare' variable, identifying antenatal care in 14% of confinements with missing 'ancare' data.

New South Wales Central Cancer Registry (CCR)

3.1 Barraclough, H., Morrell, S., Arcorace, M., McElroy, H. J. and Baker, D. F. (2008), Degree-of-spread artefact in the New South Wales Central Cancer Registry. Australian and New Zealand Journal of Public Health, 32: 414-416. doi: 10.1111/j.1753-6405.2008.00271.x

This article describes an unexpected data artefact in degree-of-spread data in the NSW CCR during 1993-1998, related to the introduction of the Electronic Notification System, and leading to a reporting of the proportion of localised cancer cases in 1993-1998 of approximately 5% lower than expected

3.2 Yu, X. Q., O'Connell, D. L., Gibberd, R. W., Abrahamowicz, M. and Armstrong, B. K. (2008), Misclassification of colorectal cancer stage and area variation in survival . Int. J. Cancer, 122: 398-402. doi: 10.1002/ijc.23043

This study found 70% overall agreement between stage of colorectal cancer at diagnosis recorded by the CCR and that recorded in a survey of treating practitioners.

3.3 Webster, A. C., Supramanium, R., O'Connell, D. L., Chapman, J. R. and Craig, J. C. (2010), Validity of registry data: Agreement between cancer records in an end-stage kidney disease registry (voluntary reporting) and a cancer register (statutory reporting). Nephrology, 15: 491-501.

This study assessed agreement between incident cancer reported to the Australian and New Zealand Dialysis and Transplant Registry (ANZDATA) and to the Central Cancer Registry (CCR) in New South Wales. Agreement of cancer records between both registries was high and differences largely explicable. The authors concluded that it is likely that both ANZDATA and CCR have some inaccuracies, for reasons that are now more explicit, with themes similar to those likely to be experienced by other registries.

3.4 Goldsbury DE, Smith DP, Armstrong BK and O'Connell DL. Using linked routinely collected health data to describe prostate cancer treatment in New South Wales, Australia: a validation study. BMC Health Services Research 2011, 11:253 doi:10.1186/1472-6963-11-253

This study linked data from the NSW Prostate Cancer Care and Outcomes Study (PCOS), a population-based survey of patterns of care for men aged less than 70 years diagnosed with prostate cancer in NSW, with the NSW Cancer Registry, electronic hospital discharge records and Medicare and Pharmaceutical claims data from Medicare Australia. Cancer Registry and hospital inpatient data accurately captured radical prostatectomy and brachytherapy treatment, but not external beam radiotherapy or disease stage. Medicare claims data substantially improved the accuracy with which all major treatments were recorded. The authors concluded that these administrative data combined are valid for population-based studies of some aspects of prostate cancer care.

3.5 Stavrou E, Vajdic CM, Loxton D, Pearson SA. The validity of self-reported cancer diagnoses and factors associated with accurate reporting in a cohort of older Australian women. Cancer Epidemiol. 2011 Dec;35(6):e75-80.

This study assessed the validity of self-reported cancer diagnoses in women born 1921-1926 recruited to the Australian Longitudinal Study in Women's Health (ALSWH) by comparing records with the NSW CCR. Sensitivity and specificity of self-reported cancer diagnoses in this cohort of older women (aged 70-75 years at baseline) was high. Overall sensitivity was 89.2% (95% CI 86.0-91.7%) and exceeded 90% for breast, lung and colorectal cancer at baseline. Overall specificity was 96.9% (95% CI 96.3-97.5%). The authors concluded that the use of linked data from population-based cancer registries is to be recommended for studies of cancer epidemiology.

3.6 Yu XQ, O'Connell DL, Gibberd RW, Coates AS, Armstrong BK. Trends in survival and excess risk of death after diagnosis of cancer in 1980-1996 in New South Wales, Australia. Int J Cancer. 2006 Aug 15;119(4):894-900.

This article explored reasons for lowered mortality from cancer over a period of years using data from the NSW CCR. The article acknowledged the limitations of incomplete data on stage of cancer at diagnosis in the CCR.

3.7 Vajdic CM, Kricker A, Giblin M, McKenzie J, Aitken J, Giles GG, Armstrong BK. Incidence of ocular melanoma in Australia from 1990 to 1998. Int J Cancer. 2003 May 20;105(1):117-22.

This study found that an estimated 20% of melanomas, mainly those lacking a tissue diagnosis, were not notified to Australian cancer registries, but only 1.3% were not notified by ophthalmologists. Expert reviewers agreed that a high proportion (95%) of clinically diagnosed lesions were probable or possible melanomas.

NSW Registry of Births, Deaths and Marriages (NSW RBDM)

4.1 Comino EJ, Titmuss A, Harris E, Craig P, Pulver LJ. Identification of Aboriginal infants at an urban hospital. J Paediatr Child Health 2007;43(9):623-6.

In this study, data on the Aboriginal status of all infants who were delivered at the hospital to mothers who resided in the surrounding Local Government Area during 2002 were extracted from the Obstetrics Data Package (ODP). These data were supplemented with local health worker knowledge about the Aboriginal status of infants and compared with NSW Birth Register. There were 1739 deliveries at the hospital to mothers from the Local Government Area. The study showed that 71.4% (n = 90) of Aboriginal and 77.5% (n = 1649) of non-Aboriginal infants identified through ODP were included in the Birth Register. The proportion of Aboriginal infants identified through the ODP was 5.2% and the Birth Register was 5.6%. The 90 Aboriginal infants included 38 with an Aboriginal mother, 34 with an Aboriginal father, and 18 with two Aboriginal parents.

4.2 Xu F, Sullivan EA, Madden RC, Black D, Jackson Pulver LR. Improvement of maternal Aboriginality in NSW birth data. BMC Med Res Methodol. 2012 Jan 30;12:8.

NSW Emergency Department Data Collection

5.1 O'Toole L, Muscatello DJ, Zheng W, Churches T. Can near real-time monitoring of emergency department diagnoses facilitate early response to sporadic meningococcal infection?--prospective and retrospective evaluations. BMC Infect Dis. 2010 Oct 27;10:309.

This study evaluated the effectiveness of reporting of ED diagnoses of meningococcal infection. The study involved a prospective and retrospective component. During the prospective evaluation, 31 patients were diagnosed with meningococcal infection in participating EDs. Of these, 12 had confirmed meningococcal disease, resulting in a PPV of 38.7%. All confirmed cases were notified earlier to public health authorities by conventional reporting.Of 149 cases of notified meningococcal disease identified retrospectively, 130 were linked to an ED visit. The sensitivity and PPV of the ED diagnosis for meningococcal infection was 36.2% and 36.7%, respectively. The authors concluded that based on prospective evaluation, it was reassuring that existing mechanisms for reporting meningococcal infection perform well and are timely.

5.2 Boufous S, Finch C, Close J, Day L, Lord S. Hospital admissions following presentations to emergency departments for a fracture in older people. Inj Prev. 2007 Jun;13(3):211-4.

The aim of this study was to estimate the proportion of older people who were hospitalised following a presentation to an emergency department for hip, pelvic and wrist fractures. This study stated that "The findings indicate that hospitalisation data do not accurately reflect the incidence of low-trauma fractures, particularly wrist and pelvic fractures, in older people."

5.3 Indig D, Copeland J, Conigrave KM. Comparing methods of detecting alcohol-related emergency department presentations. Emerg Med J. 2009 Aug;26(8):596-600.

The aim of this study was to assess the strengths and limitations of different methods for detecting alcohol-related emergency department (ED) presentations and to compare the characteristics of patients who present to the ED with an alcohol-related presentation with ED patients who are found to be risky drinkers by a questionnaire. Two Sydney Australia EDs were surveyed over four weekends of 389 patients. Alcohol-related presentations were identified using a range of methods and were compared with presentations in ED patients who reported risky drinking using the alcohol use disorders identification test (AUDIT). Overall, 20% of ED patients had alcohol-related presentations and 28% were identified as risky drinkers by AUDIT. Diagnostic codes detected only 7% of all alcohol-related ED presentations, compared with 34% detected by nursing triage text, 60% by medical record audits and 69% by self-report. Among risky drinkers, just over half (51%) were not attending for an alcohol-related reason, whereas among alcohol-related ED presentations, nearly a third (31%) were not identified as risky drinkers by AUDIT.

External datasets

6.1 Larney S and Burns L. Evaluating health outcomes of criminal justice populations using record linkage: The importance of aliases. Eval Rev. 2011 Apr;35(2):118-28. doi: 10.1177/0193841X11401695. Epub 2011 Mar 10

This study was undertaken to determine the impact of aliases on sensitivity and specificity of record linkage and how this affects ascertainment of mortality. Records for a cohort of prisoners were linked to methadone maintenance treatment (MMT) and mortality records. The record linkage was conducted in two stages. First, the linkage was undertaken using the participant's name and date of birth as recorded in a prior study. Then, a second linkage was undertaken using these identifiers, plus all known aliases. Sensitivity was 64%, and specificity 100%, for the first linkage. When aliases were added to the linkage, sensitivity increased to 86% and specificity remained 100%. The standardized mortality ratio was 4.3 for the first linkage, increasing to 6.1 when aliases were used. These results suggest that the potential effects of participant aliases on linkage outcomes, and methods for mitigating these effects should be carefully considered when planning and undertaking record linkage studies with criminal justice populations.

6.2 Centre for Epidemiology and Evidence. Quality and coverage of the NSW Register of Congenital Conditions using Admitted Patient Data: A record linkage study. NSW Ministry of Health, 2016.

The purpose of this project was to assess the quality and coverage of reporting to the NSW Register of Congenital Conditions (RoCC) compared to admitted patient records. This study found that about 30% of cases of congenital conditions were reported to the APD and not to the RoCC. We found that the quality and coverage of the RoCC depends on the condition of interest. While some congenital conditions are well reported on the RoCC, the APD has potential to substantially improve reporting of congenital conditions to the RoCC. The NSW Register of Congenital Conditions using Admitted Patient Data: A record linkage study can be foudn on the Ministry of Health Website.