Process re-engineering: a brief history of Government computing

Kevin Ashley, NDAD Project Manager

In his recent presentation to the Association of Departmental Records Officers, Kevin Ashley demonstrates that there are more reasons than might at first be obvious for selecting datasets for preservation, and explains why NDAD staff go to considerable pains to uncover background documents of all sorts relating to the datasets they preserve.

In my experience, it is true to say that most people, if they think about the issue at all, would say that the datasets worth preserving are those that contain interesting data. This is true in a sense, but it isn't the only reason a government IT system might be of interest. Forty years of civilian government computing have had a huge effect on how government works and what it can do, for good or ill. Computing changed the nature of the Civil Service: its size, skills mix and working environment. In the 1960s and 1970s particularly, government also shaped the products of the computer industry. This was particularly true in Britain where for some years the fortunes of what we once called the British computer industry were inextricably intertwined with government spending, and its enthusiasm, or lack of it, for computing solutions. The end result is that the history of government computing is relevant to a wide range of research interests: social history, politics, economics, the history of computing and the study of government itself.

*

It hasn't taking computing to make some citizens and politicians wary of the way governments collect data. In 1753. in the House of Commons, William Thornton was provoked to declare:

I was never more astonished and alarmed since I had the honour to sit in this House [...]. And what purpose will it answer to know where the kingdom is crowded [...] except we are to be driven [...] as graziers do cattle? [...] As to myself, I hold this project to be totally subversive of the last remains of English liberty.

The scheme that aroused his anger was a plan to count the number of able-bodied males in each parish of the country: a census. In the first glimmerings of central planning, someone had realised that it might be a good idea to know how many people there were in the country, whether they were all staying in one place and how many of them might be able to put up a fight if someone tried to invade. The Honourable Mr Thornton clearly didn't agree (and the Data Protection legislation that might have appeased him was going to be a long time coming) but more than two centuries later his words were to inspire a remarkable study of the impact of counting and tracking people on our lives, which I will describe in more detail below.

I would like to think there are number of reasons why the PRO might want to select datasets, and their documentation, for permanent preservation. ¹ Interesting data clearly must be one reason, but as someone with an interest in computing techniques and history, I believe that interesting implementations - either technological or organisational - are also worthy of consideration. Systems that changed what was possible, rather than just re-implementing manual processes, are of great historical interest to many; striking successes of the use of IT in government (I am sure we can find some if we look hard enough) are good candidates for preservation, as learning tools for the future if nothing else.

Failures can be equally instructive. One of the most widely read books on how to manage IT development projects is The Mythical Man-Month by Frederick Brooks, which relates the sorry story behind the development of IBM's OS/360. Despite being a disaster in many ways, this operating system went on to become the single most successful operating system in the world before the advent of Microsoft. Useful lessons could equally be learned from the story behind recent changes in the Passport Office and the immigration service of which we read so much in the press last year, or the troubled history of the London Ambulance CAD system (already a topic on degree courses in computing).

Sometimes failures will have been so total that they never actually produced any data to preserve, and NDAD is probably not the right home for them; but the documentation that survives may be of considerable interest, for example as evidence of particular trends or modes of working: outsourcing versus in-house development; decentralised IT versus large central data warehouses.

The literature

Three publications examine these issues in detail, from somewhat different perspectives, are:

• James Rule, Private Lives and Public Surveillance: Social Control in the Computer Age (Allen Lane, 1973)
• The report of the Economic & Social Research Council (ESRC) Programme on Information and Communication Technologies (PICT) study group (ESRC, 1992)
• Helen Margetts, Information Technology in Government (Routledge, 1998)

Unfortunately I still haven't located the 1992 ESRC report, although I am assured it is of great interest and other authors make much reference to it. Of the other two, the first is probably the reason I am doing this job today, having read it as a 19 year old student and been fascinated by the issues it raised. The latter is a striking study of contemporary issues in how government is being affected by its increasing reliance on IT and in particular IT outsourcing.

Rule's interest was primarily as a sociologist, in issues related to mass surveillance and the social control it offered. By 'mass surveillance' he doesn't mean spy cameras on every corner, but rather systems which gather information about a large sector of the population. He looked at the US and the UK, and considered more than just government systems: Bank Americard (the first widely available credit card), criminal records, National Insurance, and the Driver and Vehicle Licensing Centre (DVLC) were amongst the subjects. Many of these systems were not yet computerised, at least not fully, at the time he wrote, however, all were in the process of computerisation. Rule looked not only at the impact on citizens but also at the working lives of civil servants. One of the most enduring images of this that remains with me is of an early attempt to improve job satisfaction and variety at the National Insurance centre in the north-east of England. After some six months working on surnames beginning with A-B, staff were given a welcome change by moving to L-M. Computerisation was meant to remove much of this drudgery. Twenty-five years on, we have the drudgery of call centres instead.

Twenty-five years after Rule's, Helen Margett's book takes a somewhat different approach and provides us with other reasons for considering what we might wish to preserve for posterity. It is an inward rather than an outward look, concerning itself with how government operates and makes decisions related to IT. It has little concern with how this affects citizens, but a great deal to say about how recent trends have affected what is possible in government and what skills the Civil Service loses when it moves increasingly towards outsourcing as a solution. Margetts also chooses to look at US and UK institutions, although all of her examples are from government. Interestingly, she suffered from many of the same problems as Rule in getting information, particularly 'on the record', and in most case the same departments proved the most difficult in this respect.

Contracting out

Although primarily concerned with contemporary issues of the late 1990s, Margetts spends some time looking at the history of IT usage in government and how this history affects present-day ways of working. One noticeable contrast is made between the preferred route of the Department of Social Security (DSS) and its predecessors, and that of the Inland Revenue. The latter had long had a close working relationship with ICL, unusual even in government, but had been keen for many years to divest itself of IT development even before outsourcing was popular. Commenting on the 1970s IR COP (PAYE) system, it was clear that ICL was not up to it and that the technology was being pushed to its limits; 'R&D is not what we should be doing' was the main feeling. The result was the loss of all IT development skills within the Inland Revenue. Margetts comments that the British Civil Service always prided itself on being the world's best at what it did: can it now be 'best in the world' at writing and supervising contracts?

IT systems have changed forever what can be done, but not always for good. One story, covered in detail by Margetts, is of the problems that attended the failure of a new computerised fingerprint system after two years of unhappy trials in 1997. Rather than waste more time and money, the system was withdrawn, but the sudden return to manual systems caused a severe strain in many parts of the service, and demonstrated that many skills that staff had once had had been lost, even after such a short period. Another example is the computerisation of tax and pay, both within and without government, which means that a lead time of 2 years is now needed to make any significant change to the PAYE system in this country. This is a severe constraint on what any Chancellor or Government can choose to do, however radical they might wish to be. Bob Asseratti (then head of the CCTA) warned in 1997 that the eventual implementation of European Monetary Union in the UK would stretch IT resources to breaking point.

What particularly interests Margetts is the way outsourcing may involve more than simply having someone else run your computers for you. In the US, and increasingly in the UK, it leads to the outsourcing company wanting to take a greater and greater degree of control over the entire process of which the computer system is a part. This type of relationship is shown to have worked well in the past, ICL and the Inland Revenue being seen as a useful and symbiotic partnership. But Margetts points out that companies such as EDS and SEMA see running the computer systems as only Phase One: eventually they see the chance to make real money by taking over the entire process, civil servants and all. Phrases such as 'vertical integration' and 'process re-engineering' are commonly heard. PFI contracts lead to the suggestion of what is effectively profit-sharing: 'we'll collect more tax for you if you allow us to change the tax system', perhaps. Whatever your opinion of these changes, the way in which they have happened and the results of the process are clearly of interest to the study of the present and recent past, and will be of interest to historians of the future.

Census and births

In 1961 the General Register Office and OPCS - now integrated as the Office of National Statistics (ONS) - considered one of the largest civilian computing systems ever commissioned at the time (the process actually began in 1959, two years before the census.) It was hoped to produce census statistics on computer to improve on what was seen as very poor performance in 1951, when it had taken many years before useful statistics had emerged from the census. The idea was to share this system with another dealing with registrations of births, but this proved to be an unrealistic hope. The planning papers show those involved grappling with problems they had never had to deal with before and, in my opinion, acquitting themselves extremely well. They realised they had to recruit hundreds, if not thousands, of people with skills which were not widespread, or even well understood, such as punch card operators and programmers. It was believed that some programmers were available from another government project, but unfortunately nobody seemed to have any way to test their competence. One prescient administrator raised the question of whether computer printout could be truly said to be 'a record'; others with greater vision saw the possibility of using computerised death certificates to analyse patterns of ill health, perhaps even to catch epidemics before they took hold. All on systems which would seem Heath Robinson-like today, the output being produced on specially converted typewriters.

Coastal defence

The MAFF Coastal Defence Survey is one dataset in NDAD that resulted from a contracted out process: not just the computer system but the entire data collection and interpretation process. Perhaps this would be a good test for the theory about de-skilling of the civil service as a result of outsourcing. We have certainly ensured that we have preserved documents that describe not just the data itself, but the contracts that governed how it was collected and how quality control was to be carried out. They certainly don't seem to show any lack of relevant skills in the department - quite the reverse, in fact. However, they don't throw much light on why the survey has not been updated. The insurance industry is certainly concerned that we don't have current information on the threat of flood and subsidence from weakened coastal defences: the unquantified risks are reaching very large amounts.

Crime statistics

Also in NDAD is data from the Metropolitan Police Crime Statistics system, another dataset where understanding of the process is key to understanding the data. It existed only to produce statistics from crime reports, but it is clear that it could have done much more. What is apparent from its documentation is that the system's developers were also aware of this. However, cost constraints and time constraints chipped away at the specification bit by bit, and problems encountered during testing limited the functionality even further. Political and commercial concerns meant in the end that something had to be delivered without further delay, and it ended up performing one particular job very well, until superseded by another system five years later. ² If we didn't have the minutes of the planning groups and system design documents, future researchers would never know how much more ambitious were the hopes of its original developers.

*

This, I hope, has gone some way towards explaining why NDAD is interested in collecting a lot more than just the data in databases and why we are so persistent in trying to collect the contextual material that surrounds it. My aim has been to think a bit more about what we might want to keep and why. I also hope anyone involved in processing our accessions has gained a greater understanding of why we ask the questions we do. We are privileged to be able to deal with the history of government computing from its very earliest days (if we discount its funding of Charles Babbage's ill-fated engines). The processes of change that computing wrought brought in their wake outsourcing, process re-engineering and vertical integration. These changes are still taking place as we experiment with allowing greater and greater control of the interaction with the citizen to be handled by others. The advent of e-government will doubtless bring with it further changes of type, rather than just degree for the civil service, for government and its citizens. When we choose which parts to remember, lets make sure we record the abject failures along with the sparkling successes: suffice it to say that at NDAD we are looking for the complete picture.

1. In talking about what should or should not be preserved, I realise I am straying from the role assigned to us by the PRO: appraisal is not something we are asked to deal with. Nonetheless, I hope I can speak as an interested citizen rather than as NDAD's manager for once.

2. The successor, the Crime Report Information System (CRIS), did provide considerable extra functionality; the story of the development of CRIS is even more interesting.