Series details: CRDA/20

The Historic Mortality Data Files database was originally created by the Office of Population Censuses and Surveys (OPCS) as a basic tool for researchers studying mortality in England and Wales. The two versions of the database which have been transferred to NDAD record the numbers of deaths registered in England and Wales from 1901 onwards by year, sex, age group and underlying cause of death. They also provide estimates of the population at risk of dying by year, sex and comparable age groups. The data was designed to allow for the calculation of national mortality rates, the analysis of trends in mortality and differences in mortality by age and sex, epidemiological research, and local studies of mortality which required national rates for comparative purposes.1

The creation of the database was prompted by the creation by the World Health Organisation (WHO), in the mid 1970s, of a database of death rates from 1950 onwards for a number of countries by year, sex and a limited range of causes. This led other institutions to construct similar but more detailed databases to allow for the computer analysis of their own national mortality data. In 1979 OPCS decided to construct a mortality database for England and Wales using readily available sources in published form and on computer (for details of the sources used to construct the database, see How data was originally captured and validated).2 The database was updated annually by OPCS and its successor, the Office for National Statistics (ONS), to include additional years of data. It was distributed commercially to outside purchasers from at least 1985 onwards.3

In 1997 the database was redesigned by ONS, and a new version was issued to the public on CD-ROM under the title "Twentieth Century Mortality Files". This included data down to 1995, and had a number of features which distinguished it from the earlier version of the database (see Scope and content).4 After 1997, annual update CDs were issued by ONS covering data from 1996 until 1999. In 2003 ONS issued a revision of Twentieth Century Mortality Files covering data from 1901 to 2000, which incorporated revised population estimates based on the 2001 Census.5

Historic Mortality Data Files is believed to have originated in the Medical Statistics Division of the Office of Population Censuses and Surveys (OPCS). By 1998, when the first dataset was transferred to NDAD, the database was apparently the responsibility of the Demography and Health Division of the Office for National Statistics. Responsibility later passed to ONS's Health and Care Division, which transferred the second dataset. For further information about these divisions, OPCS and ONS, see the Administrative History of the Statistical Departments.

The two datasets which have been transferred to NDAD reflect different versions of the database. The first dataset covers 1901-1992, and reflects the Historic Mortality Data Files database before it was redesigned in 1997. The second dataset covers 1901-1995, and is the version which was re-issued by ONS in 1997 as Twentieth Century Mortality Files (see Aim and purpose). As might be expected, there is a significant overlap between them, and many similarities as well as differences which are detailed below.

Tables

The 1901-1992 and 1901-1995 datasets contain the following types of tables:

• Historic Deaths tables (both datasets)
• Population tables (both datasets)
• ICD Dictionary tables (1901-1995 only)

The Historic Deaths tables record the number of deaths in England and Wales in each year broken down by age group, sex and the underlying cause of death. From 1911 onwards, the cause of death is coded according to the contemporary version of the International Classification of Diseases (ICD). For the period 1901-1910, causes of death follow a classification scheme which was used in England and Wales before the ICD was adopted. Each dataset thus contains an Historic Deaths table for 1901-1910, and a table for each period in which a different revision of the ICD was in force. Down to 1992, the data relates to deaths which were registered in the year in question; from 1993 onwards, the figures represent deaths which occurred during the year.6 As with other published mortality statistics, the Historic Deaths tables include persons dying in England and Wales who were not normally resident in England and Wales, and exclude deaths outside of England and Wales of persons who were normally resident in both countries.7

Each dataset also contains a single Population table which contains estimates of the population of England and Wales (the "population at risk of dying") by year, by sex, and by age groups. The age groups correspond to the age groups used in the Historic Deaths tables.8 The Population tables provide a yardstick against which the deaths data recorded in the Historic Deaths tables can be measured (e.g. to calculate mortality rates).

Finally, the second dataset includes ICD Dictionary tables which explain the codes used for causes of death in the Historic Deaths tables. There is one ICD Dictionary table for each Historic Deaths table. These tables are not present in the 1901-1992 dataset, in which codes for causes of death were not explained except in the accompanying documentation.

ICD codes

The ICD originated in a draft nomenclature of causes of death which was presented to a session of the International Statistical Institute by Dr Jacques Bertillon in 1893. The first revision of the ICD was adopted at an international conference in 1900. New versions have been issued at roughly 10-year intervals. Maintenance of the standard rests with the WHO, and its use is mandatory for WHO member states under the Organisation's nomenclature regulations.9 The various revisions of the ICD have provided an international standard for describing causes of death, and a way of expressing these causes as numeric or alphanumeric codes.

In England and Wales the ICD was first adopted in 1911, in the form of an amended version of the second revision. The following table summarises the periods covered by the revisions of the ICD which have been used to classify and code causes of death in England and Wales:10

During the period 1901-1910, causes of death in England and Wales were classified by the General Register Office using a list of causes which was a variant of the first revision of the ICD, but did not employ ICD codes. When the Historic Mortality Data Files database was developed by OPCS, codes were assigned to causes in this unnumbered list. This is the basis for the codes for causes of death in the Historic Deaths table for 1901-1910, in both datasets.11 The other Historic Deaths tables in the datasets cover the periods of the second through to the ninth revisions of the ICD.

In the 1901-1992 dataset, ICD codes are represented by "computer codes", which can differ substantially from the ICD codes. This is particularly true in the case of ICD revisions 2-5, for which the alphanumeric ICD codes were converted into purely numeric codes in the dataset. Explanations of the computer codes and the ICD codes were not included in the dataset. However, the documentation accompanying the dataset allowed for the matching up of computer codes and ICD codes, and explained the meaning of the ICD codes for ICD revisions 2-5 and the codes used in the period 1901-1910 (see the Dataset Documentation Catalogue, reference CRDA/20/DD/1/1). By contrast, the 1901-1995 dataset includes actual ICD codes in the Historic Deaths tables from 1911 onwards, with explanations of the codes being provided in the ICD Dictionary tables. The most significant difference is that where the ICD employed a 3-digit code, a "0" was added at the end in the dataset to ensure that all codes had 4 digits.12

Data on causes of death recorded in the Historic Deaths tables represents data on the underlying causes of death, and is ultimately derived from the system for registering deaths for civil purposes. "Underlying cause of death" was defined in the 9th revision of the ICD as (1) the disease or injury that initiated the train of events leading to death, or (2) the circumstances of the accident or violence (e.g. suicide) that produced the fatal injury.13 The methods of registering causes of death and of selecting the underlying cause of death have changed over time, which affects the interpretation of data in the Historic Deaths tables (see Constraints on the reliability of the data). Where death was not due to natural causes, ICD revisions 6-9 allowed two codes to be assigned to each death: one covering the external cause of injury and the other the nature of the injury. To avoid any double counting of deaths, only counts for external causes of injury are included in the Historic Deaths tables for these revisions, in both datasets.14

Age groups

In both datasets, data in the Historic Deaths tables and the Population table is divided into standard age groups. These age groups vary according to the period covered by the data. In most cases, five-year age groups from age 5 up to age 85+ are used. However, there are variations from this for some of the periods corresponding to the earlier ICD revisions. The variations result from the published sources which were used for deaths data in these earlier periods (see How data was originally captured and validated). The following table summarises the age groups used in the datasets by period and by ICD revision:15

From 1986 onwards, data in the Historic Deaths tables for deaths under the age of 1 excludes deaths in the first 28 days of life. This resulted from the introduction of a new form of death certificate for stillbirths and neonatal deaths in that year, which abandoned the concept of an underlying cause of death. Instead, physicians were required to supply details of maternal and foetal contributions to mortality.16

Year of registration/year of occurrence

In both datasets, down to 1992, the years assigned to data in the Historic Deaths tables represent the year when the death was registered. In 1993 OPCS began publishing mortality statistics by the year in which the death occurred, rather than by the year in which the death was registered. This affects data in the Historic Deaths tables in the 1901-1995 dataset, where the year represents the year of registration up to 1992, and the year of occurrence of death for 1993-1995.17

Further accessions of datasets in this series are expected. It is anticipated that the Twentieth Century Mortality Files dataset covering 1901-2000 will be transferred to NDAD once an updated version of the database has been published by ONS.

The datasets and related dataset documentation are public records under the Public Record Acts 1958 and 1967. The National Archives has assigned these datasets and documentation the series reference RG 69.

The datasets and related dataset documentation are open without restriction. Data is available for browsing on demand by users of NDAD and does not have to be booked in advance.

The datasets and related dataset documentation are Crown Copyright. Copies may be made for private study and research purposes only.

The datasets are not subject to registration under the Data Protection Act.

The language of the materials is English.

A number of items of documentation relating to the datasets have been transferred to NDAD, including explanatory notes which were issued to purchasers of the datasets. These and other documents can be consulted via the Dataset Documentation Catalogue.

The 1901-1992 dataset is also held in the UK Data Archive, where it is known by the title "Historic Mortality and Population Data, 1901-1992" (study number 2902).

The documentation transferred to NDAD includes an article by researchers at the Office for National Statistics ("Twentieth Century Mortality Trends in England and Wales"), which is based on analysis of the 1901-2000 version of the database (not yet transferred to NDAD). See the Dataset Documentation Catalogue, reference CRDA/20/DD/2/1.

Few details are available regarding the hardware used to run the database. In 1985 the data was held by OPCS on an IBM mainframe and an ICL 2900 series mainframe, and had also been held on an ICL 1900 series mainframe.18 Data for the period 1959-1968 was derived from computer tapes recording data on individual deaths (see How data was originally captured and validated). For most of this period the original processing of mortality data and other vital statistics data was done on an IBM 1401 mainframe, which came into use in 1963 and was still being used in 1969.19 The version of the database which was reissued in 1997 was intended to be run by purchasers on personal computers (see Application software).

OPCS's ICL 2900 series mainframe had an ICL VME operating system.20 The software formats in which the 1997 version was supplied to purchasers (see Application software) would normally have presupposed a DOS or Windows type operating system.

OPCS held the 1901-1992 dataset as ASCII text files, which is believed to be the format in which it was issued to purchasers.21 The version of the database which was reissued in 1997 as Twentieth Century Mortality Files was made available in four formats: Microsoft Access version 2.0, Microsoft Access 95, dBase III and comma separated text files.22

Both datasets contain Historic Deaths tables for each ICD revision (and the period 1901-1910), as well as a single Population table. In addition, the 1901-1995 dataset includes ICD Dictionary tables corresponding to each Historic Deaths table, which explain the codes used in the Historic Deaths tables for causes of death. For further information about the structure of the datasets, see Scope and content and the catalogues of the individual datasets (Links to dataset catalogues).

The database could be said to be dynamic, in the sense that new versions were periodically issued to replace versions previously sold to the public (see Aim and purpose). However, OPCS and ONS apparently kept copies of superseded versions of the database without overwriting the data on these copies. As these copies formed the basis of the datasets transferred to NDAD, the datasets could be considered to be closed.

Data sources

The data in the datasets is taken from a mixture of published and unpublished sources. The information in all of these sources was originally gathered for purposes other than the compilation of the database. The published sources, in turn, have imposed certain limitations on the database (see Constraints on the reliability of the data).

In both the 1901-1992 and the 1901-1995 datasets, data in the Population tables was taken from mid-year population estimates which were periodically issued by OPCS and its predecessors through several series of publications. These included the OPCS Monitor Series, the Registrar General's Quarterly Return for England and Wales, the Registrar General's Statistical Review, the Registrar General's Decennial Supplements and the 73rd Report of the Registrar General (1910). Details of the sources used for population data are set out in the explanatory notes accompanying the datasets (see the Dataset Documentation Catalogue). The population estimates also reflect periodic revisions made in light of data from the decennial Census.23

Deaths data in the Historic Deaths tables for the period 1901-1958 was derived from published sources. Information on the numbers and causes of deaths in these years was manually transcribed from tables published annually in the Registrar General's Statistical Review (also known as the Annual Review). From 1959 onwards mortality data was available in electronic formats. For the period 1959-1967, these took the form of archived computer tapes of data on individual deaths, which had been used in the production of annual reference volumes. Counts of deaths were retabulated from data on these tapes. For deaths data after 1967, the compilers of the database were able to use computer summaries of mortality data which had already been created for routine tabulation purposes.24

Original checks on the data

The data in the Historic Deaths tables was systematically checked against the sources used to compile the data. For data from the period 1901-1958, this involved summing the data on computer by age group for each cause of death, and by cause of death for each age group, and checking the results against the published sources. Where the process of transcribing the data detected printing errors in the published sources, the data in the database was adjusted to achieve consistency rather than to agree with the incorrect published figures. It was also noted that 265 deaths in a colliery disaster in 1934 had not been registered until 1938-39, and were not included in the published statistics for either period. These deaths were allocated to 1934 in the database.

Data for 1959-1967 was checked against published figures, to detect any errors arising from the corruption and loss of data on the archived computer tapes. The discrepancies which were detected are summarised in the notes accompanying the datasets (see the Dataset Documentation Catalogue). For data from 1968 onwards, checking involved making sure that every ICD cause of death had been carried across, and spot checking the values carried across.25

Data coding

As previously indicated (see Scope and content), the creation of the 1901-1992 dataset involved converting ICD codes for causes of death into computer codes, which are substantially different in some cases from ICD codes. The rules governing this process were complex and are explained in the Dataset Catalogue for the 1901-1992 dataset (see Links to dataset catalogues). These considerations do not affect the 1901-1995 dataset, in which the actual ICD codes are reproduced with relatively minor modifications. See the Dataset Catalogue for the second dataset (Links to dataset catalogues) for details.

The datasets are limited by the published sources from which much of the data was derived. They are also also affected by certain general considerations which apply to all data originating from the civil registration of deaths.

Constraints arising from published sources

These constraints affect data in the Historic Deaths tables for 1901-1958, in both datasets. The age groups into which the data is divided were determined by the age groups used in the Registrar General's Statistical Review in this period. These age groups, in turn, are the basis of the age groups used in the Population tables covering the period for 1901-1958 (with the exception of the period 1940-1941, the same age groups are used in the Population and Historic Deaths tables in both datasets to allow for direct comparison of the data). Similarly, data on causes of death is limited to those causes which were reported in the Registrar General's Statistical Review. In other words, where no incidences of a particular cause of death were recorded for a particular year, that cause will not appear in the datasets.26

General constraints affecting mortality data

Regardless of whether published or unpublished sources were the immediate source of data in the Historic Deaths tables, the data was ultimately derived from the system for registering deaths in England and Wales. Like all mortality data, data in the Historic Deaths tables was affected by changes in the methods of certifying deaths and of identifying the underlying cause of death (for a definition of "underlying cause of death", see Scope and content).

The civil registration of deaths became mandatory in England and Wales in 1837. Data on causes of death was gathered as part of the registration process from then on (see the Administrative History of the Statistical Departments). From 1874 the medical practitioner who attended the deceased in his or her final illness was required to provide a medical certificate of cause of death. This formed the basis of the information about cause of death which was recorded on the registrar's death certificate, unless the death was the subject of a coroner's inquest (e.g. sudden or violent deaths), in which case the coroner's certificate of cause of death was used. Medical certificates and coroner's certificates were transmitted periodically by superintendent registrars to the Registrar General. The format of medical certificates changed over time. The most radical change occurred in 1927, when a two part medical certificate was introduced: in the first part the doctor recorded the disease or condition leading directly to death and causes antecedent to it, while the second part was reserved for "other significant conditions contributing to the death, but not related to the disease or condition causing it". From 1940 onwards the entry in the first part of the certificate was taken to be the underlying cause of death. Before then the underlying cause of death was selected by the General Register Office using a complicated set of rules, in which conditions of various types were given an arbitrary order of precedence, regardless of the order in which they were specified on the medical certificate.27

Further changes to the methods of identifying underlying cause of death occurred in 1984. OPCS adopted a broader interpretation than that previously used of a WHO coding rule, that when the cause of death in the first part of the death certificate was a direct sequel to a condition mentioned in the second part, the latter condition should be preferred as the underlying cause of death. This resulted in an artificial decrease in the numbers of deaths from certain causes (e.g. bronchopneumonia) and corresponding increases in other causes (e.g. chronic conditions such as diseases of the nervous system and mental disorders). The anomaly was reversed in 1993, when an overhaul of OPCS's computer systems led to the introduction of an automated system for coding cause of death which followed the internationally agreed interpretation of the WHO's rules for selecting underlying cause. Recent data on causes of death for the under 1 age group was also affected by the introduction of a new form of certificate for neonatal deaths in 1986 (see Scope and content).28

The above factors need to be considered when assessing mortality data in the datasets. Two other factors affect mortality data from specific periods:

• During the periods 1915-1920 and 1940-1949, death statistics only record deaths in the civilian population (i.e. deaths of service personnel in the First and Second World Wars are excluded). Population statistics in these periods reflect the civilian rather than the military population, and will also be distorted.29
• Mortality data for 1981-1982 was affected by industrial action by some registrars. This particularly affected the coding of deaths due to accidents or violence, and cases where more detailed coding was required.30

Details of the content and transformation validation checks performed by NDAD on the datasets are recorded in the catalogues of individual datasets: see Links to dataset catalogues.

Dataset catalogues provide more detailed information about individual datasets, and are currently available for the following dataset(s):

1. Dataset Documentation Catalogue, reference CRDA/20/DD/1/1, p. 1; Office for National Statistics web site, page on "Products: Twentieth Century Mortality" (http://www.ons.gov.uk/data/cds/mort.htm), consulted on 2 December 1998.

2. Dataset Documentation Catalogue, reference CRDA/20/DD/1/1, p. 1-2.

3. NDAD holds explanatory notes dating from 1985 relating to the first dataset (CRDA/20/DS/1), which were apparently produced by OPCS to be issued to purchasers of the dataset: see the Dataset Documentation Catalogue, reference CRDA/20/DD/1/1.

4. Office for National Statistics web site, page on "Products: Twentieth Century Mortality" (http://www.ons.gov.uk/data/cds/mort.htm) consulted on 2 December 1998; Government Statistical Service web site, page on "The Source: Statistical Output and Services Catalogue" (http://www.statistics.gov.uk/publish/healrev.htm), consulted on 12 November 1998; note of telephone conversation between NDAD and ONS on 11 March 1999; comments by ONS on draft catalogues for CRDA/20.

5. Email from NDAD to ONS, 22 October 2003; email from ONS to NDAD, 7 November 2003.

6. Dataset Documentation Catalogue, reference CRDA/20/DD/1/1, p. 2-3; reference CRDA/20/DD/1/2, pp. 1, 3.

7. John Charlton and Mike Murphy, "Monitoring health- data sources and methods", in Charlton and Murphy, eds, The Health of Adult Britain 1841-1994, Volume 1, Chapters 1-14, Registrar General's Decennial Supplement no. 12 (London: Stationary Office, 1997), p. 8; M. Britton, "Sources of data and limitations", in M. Britton, ed, Mortality and Geography: A Review in the mid-1980s, Registrar General's Decennial Supplement no. 9 (London: HMSO, 1990), p. 6.

8. Charlton and Murphy, "Monitoring health- data sources and methods", p. 8.

9. General Register Office, "Vital Statistics Registration in England and Wales", n.d. (unpublished draft loaned to NDAD by the Public Record Office), pp. 7-8; General Register Office, Registrar General's Statistical Review for England and Wales for the Year 1963, Part III, Commentary [reprint] [London: General Register Office, 1966?], p. 10.

10. Dataset Documentation Catalogue, reference CRDA/20/DD/1/1, p. 14 (table 1); CRDA/20/DD/1/2, p. 3; CRDA/20/DD/2/1, pp. 5-6; Charlton and Murphy, "Monitoring health- data sources and methods", p. 4-5 and table 2.1.

11. Dataset Documentation Catalogue, reference CRDA/20/DD/1/1, p. 8; CRDA/20/DD/1/2, pp. 1, 3; Charlton and Murphy, "Monitoring health- data sources and methods", p. 5 (table 2.1).

12. Dataset Documentation Catalogue, reference CRDA/20/DD/1/2, p. 3.

13. Charlton and Murphy, "Monitoring health- data sources and methods", p. 6

14. Dataset Documentation Catalogue, reference CRDA/20/DD/1/1, p. 7; CRDA/20/DD/1/2, p. 3.

15. Dataset Documentation Catalogue, reference CRDA/20/DD/1/1, pp. 6, 15 (table 2); CRDA/20/DD/1/2, p. 4.

16. Dataset Documentation Catalogue, reference CRDA/20/DD/1/1, p. 20; CRDA/20/DD/1/2, p. 7; Charlton and Murphy, "Monitoring health- data sources and methods", p. 7.

17. Dataset Documentation Catalogue, reference CRDA/20/DD/1/2, p. 1; Charlton and Murphy, "Monitoring health- data sources and methods", p. 8.

18. Dataset Documentation Catalogue, reference CRDA/20/DD/1/1, pp. 17A-19C.

19. General Register Office, Registrar General's Statistical Review for England and Wales for the Year 1963, Part III, Commentary [reprint], p. 13; General Register Office, Registration and Vital Statistics in England and Wales (London: General Register Office, 1969), p. 24.

20. Dataset transfer form for CRDA/20/DS/1 (completed by ONS), held in NDAD accession files.

21. Dataset transfer form for CRDA/20/DS/1 (completed by ONS), held in NDAD accession files.

22. Dataset Documentation Catalogue, reference CRDA/20/DD/1/2, p. 1

23. Dataset Documentation Catalogue, reference CRDA/20/DD/1/1, pp. 4, 16 (table 3); CRDA/20/DD/1/2, pp. 4-5, 8; CRDA/20/DD/2/1, p. 6.

24. Dataset Documentation Catalogue, reference CRDA/20/DD/1/1, pp. 3-4, 9; CRDA/20/DD/1/2, pp. 4-6.

25. Dataset Documentation Catalogue, reference CRDA/20/DD/1/1, pp. 9-10, 75 (table 9); CRDA/20/DD/1/2, pp. 5-6.

26. Dataset Documentation Catalogue, reference CRDA/20/DD/1/1, p. 3-4; CRDA/20/DD/1/2, p. 2, 4.

27. Charlton and Murphy, "Monitoring health- data sources and methods", pp. 4-6; General Register Office, Registration and Vital Statistics in England and Wales, pp. 11-13.

28. Charlton and Murphy, "Monitoring health- data sources and methods", pp. 5-6; Dataset Documentation Catalogue, reference CRDA/20/DD/1/2, pp. 6-7.

29. Dataset Documentation Catalogue, reference CRDA/20/DD/1/2, p. 6.

30. Charlton and Murphy, "Monitoring health- data sources and methods", p. 5 (table 2.1); Britton, "Sources of data and limitations", p. 9.