the problem with “data sharing”

datasharing.pngThe consultation closed recently on the “Better use of data in government” proposals. It has been some two years in the making and yet seems to be a superficial retread of many of the ideas repeatedly surfaced by civil servants to previous administrations – Transformational Government (2005), the Identity Card Act (2006), and the Coroners and Justice Bill (2008) amongst them.

Here’s a quick rundown of just some of the issues where I found the proposals lack rigour.

Lack of definitions

The paper provides no objective analysis of the problem(s) it aims to fix and no evidential reason as to why “data sharing” is proposed as the (only? best?) solution. Oddly there is no definition of what is meant by “data sharing”. Does it mean data duplication / copying / distribution, or data access, or alternatives such as attribute / claim confirmation? These are all quite different things with their own distinct risk profiles.

Paragraph 54 p.15 implies some potential use of attributes (“flags”), but without detail or context, and paragraph 77 p.21 some levels of granularity of types of data access, but these issues need to be at the core of the paper. Similarly, there is no definition of “public” and “private” data, and hence no quantified levels of granularity of sensitivity within those definitions (e.g. for at risk children, protected witnesses, undercover officials, etc.).

The paper hence displays little understanding of or references to current best practice. For example, there is an inadequate exploration of how technology could be mandated that enables confirmation of attributes (e.g. “this person is over 21”, “this household is entitled to a fuel discount”) without disclosure of any personal data records. The document is discursive and verbose when it needs to be analytical, evidence-based and precise.

There is an odd claim that APIs are a “new” technique (they are not). Neither by themselves do they “allow the access to the minimal necessary information” (p.4, para 12): any API will merely do what is specified for it, including distributing an entire sensitive personal data set for fraudsters or hostile governments to mine and exploit.

This lack of definitions is also exhibited in the “Illustrative clauses”, where references are made to “disclosure of information” without defining what that means – whether copying information to third parties; providing them with controlled one-time access; or whether it would merely be to confirm e.g. “this person is in debt” or “not eligible for benefit X”.

So what’s the problem we’re aiming to solve?

In terms of better and more efficient services for citizens, the description of the fragmentation of experiences of users of public services suggests the core problem is not data but poor service design. The fact that service “design” (and hence data) is fragmented across organisations is a reflection of services designed around organisational structures and their needs rather than citizens. The paper however contains no analysis of whether better services can be created by redesigning them around users’ needs rather than by trying to reverse engineer a solution through “data sharing”.

The idea of applying “data sharing” to problems that actually frequently derive from inadequate organisational and service design makes this paper read as if its purpose is to paper over the cracks and inefficiencies of existing public sector organisations and hence protect them and their existing poor processes rather than fixing the underlying problems. The paper appears rooted in a bureaucracy-centric viewpoint when what is required is a user / service-centric one.

In terms of better search and statistics to inform better decision-making, there is inadequate distinction between public and private data. It provides no specific detail on the process of de-identifying personal data nor any reference to the known problems of achieving this successfully, although paragraph 107 p. 30 proposes putting into legislation the key criteria of the de-identification process. This will need to be about more than just removing personal identifiable information before disclosure however, also placing an obligation on the data owner to ensure no re-identification can be made using the data released in combination with other data. This is a much more complex issue than focusing on a single data set released in isolation and requires co-ordination and risk assessment across and between data sets.

Where public data rather than private data is concerned, a useful policy position would be that by default all such data should be automatically published or accessible via open APIs. Neither should it be limited to government’s own internal needs, but provide a public good, open resource for the wider UK economy. There should be no additional cost in doing this: the same interface can serve ONS alongside everyone else.

Lack of policy alignment

The paper does not make clear how proposals to “data share” comply with the government policy of citizens’ data being under their own control rather than civil servants’ (para 8 UK Government Technology Code of Practice). Instead, it appears to place the bureaucracy at the centre, weakening citizens’ control over their data in order for public bodies and their employees to “share” it around between their organisations rather than by improving the design of public services. It thus appears out of step with the focus on user needs and better services being pioneered by the Government Digital Service.

The paper appears unaware of, or unaligned with, other government initiatives. For example, it is notable for the complete absence of any reference to the Verify user identification programme. If Verify is to be used, why is it not included? If it is not to be used, why not?

Its absence suggests that citizens and their needs do not lie at the centre of this paper. What alternative identification, authentication and verification mechanism will be used by citizens to ensure secure and authorised access to personal data if Verify is apparently to be ignored? And what identity and access management approach is going to be used within and between public sector bodies? No such system currently exists.

What alpha or proof of concept work has taken place over the 2 years this paper has been running to explore and validate different models and inform the policymaking process? Why not “show the thing” rather than just spend 2 years producing paperwork?

Illustrations are simplistic and unrealistic

The illustrations provided are well-meaning but overly simplistic. For example, the illustration given of registration of a birth makes no mention of user identification, authentication or verification. As a result, the examples as they stand are more likely to increase fraud rather than help mitigate it: fraud often arises from poor data management and inappropriate data access and access controls (including social engineering to exploit such weaknesses), providing fraudsters (both insiders and external agents) with the ability to game the system. “Data sharing” more widely – providing an even larger pool of people and organisations with access to useful personal data – will remove it from current domains of control and context, exacerbating and increasing problems of fraud.

Security is mentioned only 11 times in the entire paper, but there is no detail of the computer security techniques to be applied. In particular, the paper makes only one mention of encryption. Along with the absence of any detail about identification, authentication, access controls (authorisation), confidentiality, integrity, non-repudiation, audit, protective monitoring etc., the proposals are inadequate in determining how opening up personal/private data will reduce fraud rather than increase it. There is a risk of repeating the poor design of earlier central government programmes (e.g. see the lessons learned on New Tax Credits [1], which took a well-meaning but simplistic approach to simplifying a complex data issue).

In the absence of such details, they could effectively lead to these proposals becoming a fraudsters’ charter.

Drilldown into an Example

Let’s take just one example – that provided on p.17 – to explore how these proposals lack detail and an explicit understanding of the problem domain and the issues that need to be tackled:

Screen Shot 2016-05-09 at 15.32.08.png

For a paper asserting to be the major proposed policy on sharing citizens’ private data with more civil servants and more organisations, it is notable for its failure to provide detail on the processes to be applied to the protection of data, whether it is attributes that are being confirmed, how users are authenticated, how audit happens etc.

The questions lack a meaningful context

The weaknesses above undermine the questions posed in the paper, since they lack an objective basis against which they can be assessed and answered. Let’s take an example, that of Question 8:

“Should a government department be able to access birth details electronically for the purpose of providing a public service, e.g. an application for child benefit?” (p.17)

Of course government services should be more efficient and online and seamless and painless: no-one argues with that. But this is not the issue here: it is unclear how anyone will be able to answer this question with any meaning given the absence of any description of how the system would work. Details missing include:

  • how will a civil servant in the “government department” identify themselves as a person with a legitimate interest in the birth details and with an appropriate level of clearance?
  • how will their access be monitored and how will it be audited – and will this be real-time protection or retrospective? (particularly important if someone is accessing an at risk individual’s personal data)
  • will civil servants be able to trawl all records or only the specific one related to the event on which they are currently working, and how will such mapping/matching happen?
  • how will the data be “shared”? Will it be copied to their system, will they get access to the full record, will they merely view the record on the system where it is currently retained, or will the system merely confirm attributes (e.g. “this parent has a child and is eligible for child benefit”) without disclosing any details about the child or their data?
  • how will data be secured, what levels of protection are being applied to data at rest and in motion? What levels of granularity are being applied to access controls to ensure more sensitive data is not disclosed without appropriate authority?
  • how does the civil servant prove they are acting on behalf of a legitimate parent or guardian and not participating in a potential or actual fraud and merely “fishing” for data?
  • how does the parent or guardian initiating the claim for child benefit prove who they are and prove that the child that they are asserting is theirs *is* theirs?
  • how is the data of those most at risk going to be “shared” whilst ensuring preservation of security and without tell-tale flags that in turn reveal that a sensitive record has been “hidden”?

In the absence of a definition of how this will work, how are the questions asked in this paper going to be answered with any credibility or meaning? Without such detail, the potential for more widespread and automated fraud and the compromising of potentially at risk people, such as vulnerable children, will be compounded.


We all need government to become smarter in the way it works, and to play a more positive role in our economy. Better use of data must play an essential role in making this happen. The problem is that this paper provides inadequate detail about basic, fundamental areas (such as security, privacy, accountability) – or indeed any early proof points – that will determine the success of the systems it proposes to put into place.

Without these details being clearly defined, in either the paper or the draft illustrative clauses, the proposals to “data share” will expand the pool of people and organisations able to access citizens’ personal data. In an increasingly digital economy, expanding access to useful personal data is more likely to increase the risk of fraud, not reduce it. There are smarter ways of tackling these problems – from improved service design to technical measures to protect data whilst enabling it to inform decision-making.

Disappointingly, they are inadequately covered by this paper.

[1] See for example “Online tax credit system closed” and

Posted in digital, future Britain, identity, IT, IT strategy, open government, privacy, public services, security, social exclusion, social inclusion, technology, technology policy | 2 Comments

UK Government’s single Website since 1994

I tweeted four screenshots a couple of days ago showing the main UK Government Websites since 1994. A few folks have asked for better quality resolution, so here are the best I currently have (if you have better, happy to share them).

I briefly covered the history of government attempts to duplicate the single Website / portal model of AOL and CompuServe in my piece Happy 20th anniversary online government.


1994 – Government Information Service


2001 – UKonline

2004 – Directgov

Gov.UK 2016.png

GOV.UK – 2012

Posted in digital, IT, IT strategy, public services, technology, technology policy | 2 Comments

King Canute, diffusion and the Investigatory Powers Bill


King Canute rebuking his advisors for suggesting he could hold back the waves

We can all learn something from King Canute. At least he had the humility to know, contrary to popular misconception, that he could not hold back the waves.

The same humility is absent however from the Investigatory Powers Bill – which seems to imply it can hold back waves of diffusion.

Diffusion is the way in which a new innovation, such as the car, television or the telephone, starts from a small niche market of early adopters through to being a commodity used by all. It’s best known from the classic Everett Rogers curve.

Think of any new innovation or technology – from flat screen TVs to contact lenses to electric cars – and it becomes apparent how diffusion works across multiple markets, industries and organisations.

Let’s consider how diffusion will impact the IP Bill by taking a look at encryption.

Computer encryption was initially limited to those with the compute power and expertise to use it – such as the intelligence agencies. Early work on public-key cryptography was kept secret. But as with any innovation it was only a matter of time before what was available to a very limited and select few followed the diffusion curve. Encryption moved out of its niche market and into the mainstream.

The impact of this process has been good for us all – better security for our online financial and commercial transactions, and better security for devices such as laptops and mobile phones. Successive waves of technical innovation have provided the intelligence agencies with short-term advantage. But over the longer term, those advantages flow out and diffuse to us all.

There’s a big downside too of course. This same pattern of diffusion happens in less helpful ways  – such as criminal hacking.

At one time hacking was limited to those with in-depth technical capabilities. Now hacking is increasingly commoditised. Today someone without any technical knowledge can download and run automated hacking scripts and launch potentially damaging criminal attacks without any real technical understanding. What was once niche and specialist has become mainstream.

And this is where diffusion and the IP Bill clash. Big-time. Here’s why.

The Bill talks about being able to demand “the removal of electronic protection applied by a relevant operator to any communication or data”. The Bill also seeks other significant powers, such as making it legally permissible to remotely hack computers.

So let’s assume the Bill passes. Someone creates a way to “remove electronic protection” from any communication or data. So too hacking tools are created and exploited so that computers can be remotely compromised and their contents accessed. So far so good – just what the Bill’s authors wanted. Trebles all round.

Ah. But we haven’t yet considered the impact of diffusion. Unfortunately what starts today as a specialist way of compromising security and enabling remote hacking will tomorrow become a commodity, available to all. A universal way to remove “electronic protection” from every device, communication or data.

It’s hard to believe anyone considers this a good idea. Consumers will no longer be able to trust their devices or online financial and commercial transactions, or businesses their mission critical information systems. Without trust, our online commerce and financial environment will fail. Worst of all, the intelligence and law enforcement communities will find their own operations and security progressively, and fatally, compromised.

The IP Bill in its current form will lead to the very opposite outcome to that its authors foresee.

More time is needed to get the Bill right. The wrong decisions now would prove devastating. Not just to our trust in technology – but to our personal and national security.

Just as King Canute accepted that he could not hold back the waves, the civil servants authoring the IP Bill need to recognise that they can’t hold back diffusion.

Posted in digital, future Britain, IT, IT strategy, privacy, public services, security, technology, technology policy | 1 Comment

security, privacy and the Internet of Things

photoI had the pleasure recently of opening a Cambridge Union Society debate on the topic of “This House Fears The Large Scale Collection Of Personal Data“. This theme is partly what inspired my CIO Column on “The Internet of Thieves“. The issue of enterprise and Internet security (or more usually in my experience the lack of security) has occupied much of my career – and unfortunately seems likely to continue to occupy much of the rest of it!


Jerry Fishenden proposing the motion at the Cambridge Union Society 2016.
Photo © Chris Williamson 2016.

Alongside me proposing the motion was Heather Brooke, the investigative journalist and freedom of information campaigner. And opposing us were solicitor and academic Professor Christopher Millard and journalist Edward Lucas. Both sides of the debate were complemented by a student speaker: supporting us with the proposition was Katherine Dunbar, student and competitive debater; and the opposers were supported by Katie Heard, Durham student and competitive debater – both of whom demonstrated their expert familiarity with the format and a commendable ability to have read and learned about the topic in remarkably short time.


Edward Lucas and Katie Heard consider their response, while Professor Millard looks on.
Photo © Chris Williamson 2016.

The formality of the setting and format of the debate all seemed a very long way from the “debates” we used to have at my old south London comprehensive. It made me realise how rarely I engage in formal debate – most conference events consist of “panel discussions” and tedious broadcast-mode slideware instead.

The core of my opening proposition was that far too many digital businesses rest on a profit model centred on the relentless commercial exploitation of our personal data. Some of our personal data of course we may share voluntarily with others in return for a benefit – store loyalty discounts for example. But a great deal are taken, analysed, manipulated, sold and exploited without our consent – often indeed without even our knowledge.


Professor Christopher Millard makes his case for opposing the motion. 
Photo © Chris Williamson 2016.

Not so long ago it was a unique, unpleasant characteristic of totalitarian states that citizens were permitted no secrets – no private, personal spaces. No freedom. We pointed wagging, righteous, critical fingers at such regimes.

So it’s ironic that a worryingly similar invasion of nearly every aspect of our personal lives has now been adopted as the routine, prevalent, business model of many Western companies and even, shamefully, some of our governments.

Let’s think about this another way. What would we make of somebody we discovered rummaging daily through our dustbins to examine our discarded letters, beer bottles and food packaging? Of somebody who stalks a few paces behind us everywhere we go to observe who we meet and to eavesdrop on and record our conversations?

We would, I think, regard such an individual to be perverted. Possibly even insane. Certainly not someone you’d invite to your birthday party – somebody probably best subjected to a restraining order.

unspecified 5.jpg

The debate in full flow at the Cambridge Union Society. 
Photo © Chris Williamson 2016.

And now imagine this person also randomly and obsessively runs up to us from time to time and shouts “I think you might want to buy this car!”, or “Are you looking for a new house?” or “You’re drinking too much!”.

Yet this invasive and obsessive behaviour is precisely how our technology behaves. Every second of every day. We should regard this use of technology – to trawl, monitor, gather and mine our personal data ­– as no less perverse.

Instead of using new technology to partner with us as equals to our mutual benefit, far too many organisations are obsessed with fleecing us of our personal data for short-term gain, without any regard for the consequences. All in the vainglorious hope that it will provide them with the power of precognition, the ability to understand us better than we understand ourselves – in order to take even more money from us.


Heather Brooke debating in favour of the proposition: “I believe in privacy for the private citizen going about their private business, and transparency for the public official making policy decisions which affect us all.”
Photo © Chris Williamson 2016.

But but but!“, my critics will counter, I am concerned for no good reason. We should all just enjoy the benefits bestowed on us by this largescale collection and use of our personal data. Where’s the harm?

Well, in response, consider the expert advice given by those who safeguard our critical national infrastructure. They warn of the grave risks of aggregating bulk personal data – creating a pool of valuable information that will be targeted, exploited and abused by everyone from foreign hostile powers to opportunist hackers.

We should heed such warnings.


Katie Heard opposing the motion.
Photo © Chris Williamson 2016.

If our bulk personal data is collected it will, without any doubt, sooner or later flow into the hands of whoever wants it. Whether by accident or design. So what? “Nothing to hide, nothing to fear.” Isn’t that what we keep being told? The same self-serving line trotted out by those totalitarian governments we once rightly criticised.

In any case, try parroting that nonsense phrase to a battered spouse, abused child, whistle-blower, informant, witness to a serious crime, journalist source, barrister and their client, or undercover law enforcement official. Do you really think they have “nothing to hide”? Of course they do, and for very good reason – this is part of the reality I was arguing in my column “Securing digital public services“. Access to personal data can, literally, become a matter of life and death.

This abuse of our personal data threatens us all in other ways too. It undermines our everyday security. What’s the point after all of protecting an online financial account with “secret” details of your first car, favourite colour and memorable place when those very same details are being Hoovered up and sprayed around the world?


Katherine Dunbar argues passionately in favour of the motion.
Photo © Chris Williamson 2016.

The irony is that all of this sucking up of our personal data isn’t even necessary: it’s the by-product of a badly broken and ill-conceived business model. How much simpler it would be if we had better business models, ones designed to enable and secure the Internet age. Empowering technology that lets us maintain and control our own personal data, and choose with whom we wish to share it.

What a terribly brilliant, but dangerous idea that is. Rather like democracy itself. Yet we urgently need to adopt this type of imaginative new approach if we are going to end the toxic legacy of analogue thinking in the digital age. The intrusive and dangerous large scale collection of our personal data needs to end, whether by businesses or ­governments. Our democratic right to safeguard and control our own personal data must be strengthened.

Until this happens, we must do everything in our power to protect our data – by using ad blockers, virtual private networks, cookie wipers, onion routing, end-to-end encryption. Whatever it takes to keep our data, and us, secure.

unspecified 6.jpg

Edward Lucas makes the closing case for voting against the proposition.
Photo © Chris Williamson 2016.

The large scale collection of our personal data must not be seen as some sort of ransom or blackmail we have to pay in order to enjoy the benefits of our digital age: quite the opposite in fact. I  supported the Society’s proposition that we should fear the current abuse of our personal data – because it has become the biggest risk to this emerging, amazing, exciting, digital age.

Reflecting on the debate afterwards, I think there was little significant distinction between either side – the underlying consensus seemed to be that we should all have better control over our own personal data. You can’t have security without privacy and vice versa.


… relaxing after the debate. 
Photo © Chris Williamson 2016.

The post-debate Press Release from the Society provides a high level summary of the debate, as does the short, edited highlights video below (I understand the “Director’s cut” full version will be available at the end of this academic term). This is an important topic that needs much more discussion and understanding – and not just in the debating hall of the Cambridge Union Society.


Posted in digital, future Britain, identity, IT, privacy, public services, security, technology, technology policy | Leave a comment

more CIO bits and pieces

Here’s a summary and links to my last few CIO columns of 2015. Looking forward to what 2016 brings.

Service design thinking

December 2015.

The use of technology in government is all too often relegated to simply making business as usual more efficient, or improving co-ordination between and across the arbitrary divisions of the public sector. Rarely has it managed to achieve the type of fundamental re-engineering of public services typical of modern digitally-enabled organisations, despite a long-held desire to do so. The real challenge is not technology – but to move from a bureaucracy-centric culture to a service-oriented one.

Securing digital public services

November 2015.

The IT reform agenda of the past five years has successfully demonstrated that government technology itself is rarely “special”. Public services can often be designed using commodity goods and services – much as in any other organisation. Yet one area where governments do face a unique challenge is in their duty to protect some of the most vulnerable and at risk individuals in our society. Government must resist calls to weaken cyber security if it is to continue protecting the most vulnerable and at risk in our digital society.

Posted in digital, future Britain, IT, IT strategy, open government, privacy, public services, security, social exclusion, social inclusion, technology, technology policy | 1 Comment

CIO roundup

Here’s a summary, with links to the full pieces, of some of my recent CIO columns:

Data-enabled service design

October 2015

The chorus of people calling for personal “data sharing” in the public sector seems to grow by the day. Yet rushing to propose “data sharing” is to start in the wrong place. If “data sharing” is the answer, what was the question?

Of lipstick and pigs in government

September 2015

The UK government is “in the vanguard of developing common IT architectures, and ahead of most in developing the IT core to enable secure transactions with citizens”. Is this part of the ongoing debate about the future of the Government Digital Service (GDS), GOV.UK and the role of “Government as a Platform”?

No. It’s from an international survey carried out in 2002, reflecting an earlier UK Government strategy of developing a cross-government infrastructure covering common services such as payments, authentication, transactions and secure messaging.

Time for the rise of the platform mutuals?

August 2015

Draw back the curtain of hype from many so-called “digital innovators” such as Uber and you reveal familiar pyramid shaped organisations that share many negative characteristics with the heyday of the railway and oil tycoons. Even claimed innovations such as “dynamic pricing” are merely shallow re-brandings of the economics of the barrow boy – putting up prices when something’s in high demand, reducing prices when it’s not. Yeah, very original. And, just like those earlier tycoons, these new businesses operate in a largely unregulated environment — beneficiaries, for now, of governments’ habitual failure to keep up with the times.

Re-establishing trust in technology

July 2015

It’s becoming difficult to remember what it felt like during those early, pioneering days of the internet. It seemed to hold out so much promise. I don’t mean the transactional convenience that dominates our usage today — online banking, booking holidays, music downloads, photos of kittens — but its more Reith-like ambitions. For a fleeting magical moment in time it held out the promise of new forms of political, social and personal expression and a more empowered, participative and inclusive society.

Then reality intervened.

Improving front line services

June 2015

Digital organisational practices enable us to make significant improvements to the way our public services work. But instead of seizing hold of this opportunity, many existing bureaucracies appear to place their own internal interests before front line services. There remains a widespread misunderstanding of “digital”, miscasting it as being about yet another generation of online versions of paper forms and processes rather than the catalyst for a wholesale reshaping and improvement of the public sector.

The result is damaging and unsustainable.

Posted in digital, future Britain, IT, IT strategy, open government, privacy, public services, security, social exclusion, social inclusion, technology, technology policy | 1 Comment

online citizen accounts

The idea of an online government account where we can see everything in one place has been kicking around since the late 1990s and the “” all-in-one portal. Despite several generations of government portals over the last 21 years (GIS, UKonline, Directgov, and now GOV.UK) we still don’t seem to be any closer to fulfilling that original vision.

HMRC’s online self-assessment portal for example looks almost identical to when it launched 14 years ago, in 2001, in that heady, optimistic flurry of early “e-government” services. Online government today still generally presents us with a set of silo transactions that mirror the paper-based processes that went before — despite the original plans to use technology to re-think and redesign government services.


Figure: HMRC’s self-assessment service some 14 years on

In this same time frame, even the high street banks have dragged themselves into the Internet age. They’ve gradually provided significantly improved services, through the use of PCs, smartphones and tablets, the deployment of chip and PIN cards and contactless payments, and quicker ways of transferring money electronically, such as Faster Payments. Partly as a result, total cash payments have been overtaken by non-cash payments.

In the best of the public sector we’ve seen organisations such as TfL (Transport for London) taking strategic advantage of these improvements to streamline and improve their own services, embracing the use of contactless payment for example.

Whilst the banks face similar challenges to government – including their complex brownfield IT estates with creaking mainframes in the back office – they’ve made meaningful progress in transforming their front-end operations. And even though the banks compete with each other, they’ve still managed to collaborate so that we can use our bank cards in any ATM and see our current account overdraft regardless of whether that machine is run by our bank, another bank or a third-party provider.

Making it happen – building on what’s in place

But just how accurate is this somewhat cynical view of how well government is handling technology to transform the way our public services operate? How hard would it be to now provide the type of online citizen tax account that the Chancellor mentioned in his budget? How far are we from realising that long-held vision of a more widely integrated online government account for citizens, and businesses, alike?

There’s been a lot of negative commentary questioning the ability of government to deliver the online tax service the Chancellor outlined. Much of the media commentary has focused on this becoming yet another project ripe for “government IT disaster” headlines. That it’s just a grandiose pipedream and will be far too complex to implement. Yet such understandable cynicism overlooks the infrastructure that the UK already has in place. The move towards the real time information (RTI) tax system implemented by HMRC over the last few years has already demonstrated how government can make much more rapid progress in delivering new services if it’s smart about how it works.

The implementation of RTI means that PAYE (Paye As You Earn) data now flows automatically into government. RTI’s success relies on the way in which it has aimed to integrate reporting obligations alongside the actual payments. For example, when an employer sends information about their employees’ salaries and deductions to government and simultaneously makes the actual salary payments to employees’ bank accounts, they fulfill all their obligations without the need for later reconciliation.

Whilst the interim, and incomplete, solution of RTI currently implemented by HMRC means in the short-term that employers haven’t been able to completely integrate payment and reporting, it does give a promising pointer for the future.

RTI interim

HMRC’s real time information (RTI), simplified view of current (interim) design

As the figure above shows, at present RTI data currently flows through two separate channels. For the 70,000 largest submitters, these two processes are automatically joined up by means of a “BACS hash”. This is a convoluted but workable solution that enables payments made over the BACS system to be verified and matched to employers’ payment declarations. This less than ideal interim solution was adopted largely because of representations from the payroll industry, who expressed concerns about changing both what was sent and the channel it was sent over. It’s intended as a staging post on route to the original fully integrated system.

The full solution for RTI envisages a much more streamlined re-use of the existing banking infrastructure. The UK’s banking network processes around 10 billion transactions each year (about ten times the volume of transactions that HMRC handles), with a combined value of about £5 trillion. It provides the central infrastructure for BACS Direct Debit and Credit payments and the Faster Payments Service, and connects the world’s busiest ATM network of over 69,000 machines.

The UK payments industry has already recognised that this existing central infrastructure easily has the capacity to carry the extra data required to meet government’s requirements (a meagre 18 characters per transaction). It has therefore publicly announced its commitment to work with the UK government to develop a strategic, complete solution that would replace the interim RTI phase — an initiative now nicknamed “Richer Data”.

HMRC richer data

The proposed “Richer Data” approach (the original strategic design for RTI)

The significance of these developments goes well beyond PAYE. Although the above Figure illustrates a payroll payment, the infrastructure would enable any payments and the information associated with them to flow regardless of the type of financial transaction. The architecture will enable not only transactions of interest to government, but will also enable businesses to include relevant data with other forms of payment, helping them rationalise and automate many of their other business processes.

Given these existing developments, the next logical step would be to enable citizens (and potentially businesses too) to login online to see and manage their data for themselves. Over the year we would be able to see in near real-time everything we have been paid as employees and all the taxes deducted. It would help to provide the type of experience we already have grown accustomed to with online banking, where we can keep track of our finances in near real time.

So the first step towards providing us with online government accounts should not be “a major IT project”, but a programme that enables us to access our existing data based on enhancements to the UK’s core national payments infrastructure — a programme not run as a “government IT project”, but a joint programme in partnership with those who currently own and run it.

HMRC could surface this information via their own portal, where they already provide other HMRC services, elsewhere on the GOV.UK infrastructure, or through other access channels. For logging in to such a government-based service, there’s already the old Government Gateway authentication system, although any new services are likely to use the replacement Verify identity assurance system to enable us to login as securely and easily as possible. Equally, another option would be for us to access this data through our existing secure, trusted and familiar online banking services — via online banking services for example, and to view or print the data at an ATM. It will be interesting to see whether we are given a choice of channels as plans for the citizen account develop.

Making it worthwhile

By making smart use of the UK’s existing payments infrastructure, within a fairly short time period we could have an online service that lets us see everything we earn and everything we have contributed to the government from an employee perspective. This would be a useful first step. We would be able to see in near real time our current year’s contributions and earnings and, over the years, we would begin to build up an historic record of our cumulative contributions and earnings. So far so good.

But this system would not provide a complete picture. Any other earnings, such as from savings, that do not go through PAYE would not appear. And our relationship with the state is not one-sided: wouldn’t it be useful to see any benefits or welfare payments being made to us too? After all, one of the other reasons for the implementation of RTI was to provide up-to-date information about employment and pension income so that the Department for Work and Pensions (DWP) can determine and adjust claimants’ Universal Credit awards.

What we ideally need is not a partial set of data, but an equivalent to our online bank accounts, showing monies in and monies out. It wouldn’t be much use only having part of the picture, of seeing what we pay into the system without the balancing information about how much we benefit too.

Surprisingly, enabling us to access this additional information needn’t be such a big subsequent step. The banks already send to HMRC the taxes deducted from our savings accounts. So this information could also be rolled into our citizen account. On the welfare side, DWP uses a duplicate of the same data gathered by RTI to help inform the calculation and determination processes that decide our entitlement to Universal Credit payments. So this data could be made visible in our online accounts too.

Given that much of the infrastructure and data already exist, in a fairly short space of time we could have an online service that enables us to view all of our earnings, our payments into government and our payments received from government. Over time, this would also build up into a useful lifetime record of our earnings, contributions to the state and receipts from the state. A true citizen account rather than only a partial and incomplete view.

citizen account mockup

A simple mockup of an initial citizen online account

Extending the model

Would this be enough? It would certainly be useful, but in some aspects it would also still be incomplete. What about other taxes, such as indirect taxation like VAT? For most of us, the tax we contribute via indirect taxes such as VAT will also be significant on an annual basis — as are other taxes or duties if we fly, drive, smoke or drink. It would be good to see all of this too, but how feasible would that be?

Not as difficult as you might think.

Part of the way that the “Richer Data” initiative will work is via the use of a financial data standard (likely to be based on an open standard such as ISO 20022). This would enable additional data to be carried alongside financial information – such as the additional payroll data that now flows with RTI.

For example, suppose we pay £12.00 using a credit card in a shop, which is actually £10 + £2 VAT. At the moment, when we receive our credit card bills we don’t see this breakdown, merely the gross amounts including tax and the total amount due: we lose insight into the amount of tax incurred in our daily lives. However, if these transactions used the ISO 20022 standard, both our credit card statements and the online citizen portal could reflect this breakdown, not just the total amount, as at present. It would enable us to keep track automatically of our VAT contributions. Indeed, it could conceivably cover all national or local government payments – council tax, parking and congestion charges, even library fines and prescription charges.

Given that many of these transactions already take place over the banking network, capturing this information could be automated using the same processes and data standards. In the same way, if we pay for an evening out in a pub or fill up our car with petrol or pay for a holiday flight, the data captured and shared back to us could also include other related taxes, such as beer, petrol and air passenger duty.

new payments infrastructure

How payments data could flow from e.g. retail outlets

This approach would enable the much-promised online citizen account to become a rich resource to us in terms of our interactions with the state. Yet a single “citizen portal” should not be the only option. As with the current ATM system, which lets us see our account balance anywhere we choose (even at a competing bank’s ATM) we should also have choice about what channels we can use to see and track our information. It shouldn’t just be accessible from a single government website: that would take us back in time to the sort of top-down design and massive, monolithic “government system” thinking typical of the late 1990s.

But whoah! Hold on a moment – won’t this system I’m describing also enable the state and our financial providers to know far too much about us in terms of where we shop, what we buy, how much we drink and smoke and so on?

Making it private and secure

Whilst a data rich online citizen account could be enabled relatively simply, and in an incremental fashion that avoids the “big bang” chaos of some previous government programmes, clearly there are significant privacy and security concerns that must be addressed.

Would we be happy for all of this information to be gathered and stored in a single place? It would be incredibly valuable data in the wrong hands, providing rich insight into many aspects of our private lives, where we spend and on what, and how dependent we are on the state. Inappropriately accessed and used, it would be a potentially toxic resource and effectively function as a confessional self-reporting system on where and how we live our daily private lives.

If we are to have a useful online citizen account, security and privacy need to be built into the system by design. It should aspire to comply with the sort of principles Kim Cameron set out in his “Laws of Identity”. The system must avoid the inadequate technical design of earlier government initiatives, such as the national identity register and its associated identity cards which based themselves on a simplistic model from the 1930s. Such systems demonstrated poor systems design and engineering, neglecting to use modern technologies that provide stronger data protection and hence enhanced levels of security and privacy.

It’s been possible, for example, to conduct a transaction such as confirming someone is a higher rate taxpayer or in receipt of child benefit without revealing anything else about them since at least the early 2000s [1]. Such techniques need to become commonplace rather than the sloppy “data sharing” approaches which assume personal data needs to be copied and shared everywhere, with the inevitable leaks and abuses reported so frequently by the media. Newer technology options, such as homomorphic encryption [2], blockchain [3],  certificate transparency [4] and ‘Guardtime’ [5] should also be robustly evaluated to see what potential role they could play in a secure, privacy aware and citizen-centric service.

If government applies good privacy and security engineering to an online citizen account, it would also have beneficial impacts on the wider financial system. It would raise the bar for example on the security used within banking and retail operations, which remains relatively leaky today. They too could move to take advantage of the more secure technologies that an online citizen account will require.

Such a system would also need to give us control over our personal data (in line with government policy).

government policy

Government policy on citizens’ personal data (source: Government Service Design Manual)

Better than that, it must also be engineered so that it is impossible for the system to hold inappropriate detail or enable anyone to reverse engineer our interactions without our explicit consent (or under due process of law). That is why ensuring the right technologies are engineered into the design before it is developed is essential. This would enable transactions to be verified and authenticated, and for data to answer questions such as “Is this person entitled to a tax credit?” but without any intrusive “panoptic” central authority holding all the details.

Such a system must also enable citizens to continue to use cash where they wish, but to obtain a point of sale receipt that enables them to manually enter records if they want to keep track of all interactions, including those that don’t use digital technology. (For insight into the potential dubious consequences of moving to a completely cashless society, this article is worth a read).

Many of us already interact with what is regarded as a generally safe, secure and ubiquitous financial system that we feed data into. After all, using online payments we’re becoming accustomed to services that let us transfer our hard-earned cash out of our own accounts and to a sequence of numbers that we trust to be the account of our intended beneficiary. We must be able to trust this system to protect our data and use it for the purposes we designate if we are going to increase our dependency upon it even further.

So all the components and technologies are in place, or in near-development, to make this vision a reality. A step-by-step approach can build on what is already there. It now requires a clear political commitment to ensure the whole design remains secure and privacy-friendly. The design of such a system must be done in the open, so that the UK’s expertise in privacy and security engineering can contribute, helping to review and improve the design. Building on the Cabinet Office sponsored private/public open collaboration on identity, represented by the OIX open forum, we also need a payment equivalent where payments standards and interoperability can be developed to meet these requirements, but with sufficient commercial incentives to maintain a competitive and innovative drive.

Where some of the necessary security and privacy technology is not quite ready for prime-time, government should play an essential catalysing role in encouraging further research and development – another good reason for working on this in the open. Not only will the citizen account then become an exemplar of how to create modern, secure and private citizen services – but it will also provide a unique competitive advantage to the UK, helping researchers, public services, and commercial and financial businesses to develop world-leading secure online computing and personal data models, technologies and products that will be in high global demand.

Wider benefits

There should be many other beneficial and far-reaching side effects to engineering the online citizen account well. For example, it would become simpler in real time to see other data, such as which major businesses are being subsidised by government – where employees are in receipt of tax credits to top up their low wages for example. Such information should also be easily accessible in the public domain.

insight into business data

Visibility of which businesses receive taxpayer subsidies

Such transparency would help inform the debate about the extent to which taxpayers subsidise apparently profitable businesses and enable better modelling of things like whether increasing the minimum wage to the living wage would produce a better outcome for all than business subsidies made via tax credits. Equally, it might be appropriate for government to consider reclaiming such taxpayer subsidies from a business’s annual profits. Without accurate and complete data, many such policy considerations are little more than a stab in the dark at the moment – but the moves towards a well designed system, with the right policy and engineering safeguards built in, would provide far wider benefits than just the immediate citizen account itself.

There are also likely to be considerable knock-on benefits to government’s own operations. It would make it possible, for example, to decide whether to continue to maintain two separate organisations with many duplicated functions, one which takes money from us (HMRC) and another which gives it back (DWP), with all the costs and friction (and citizen inconvenience and personal hardship) this artificial split can currently create. Given the type of data that RTI already collects and the type of data that would be available in a citizen account, it would be much simpler in future to have a single set of calculations that offset both deductions (taxes) and allowances (welfare). Doing so could improve the services we receive, cut their operational costs and inconvenience, and reduce the levels of fraud in the system.

The result is that government will progressively be able to streamline its own processes and organisation to better meet the needs of citizens and businesses, providing the type of digital transformation centred on improved public service design that we discuss in our book “Digitizing Government”. For the majority of citizens and businesses the administrative burden and frictional and human costs would reduce and government could better focus its efforts on those whose need more support and a helping hand — as well as homing in on those who intentionally fail to comply and contribute to our society.

Such a system should be incrementally developed, proven and improved over time rather than trying to do too much all at once. For example, we receive many other benefits from the state that are harder to quantify – from education to healthcare, policing to defence, and from road building to public transport. Working out how to determine the shared benefits we take from these will remain a much more complex challenge. But the route for the journey ahead is clear to see.

What we need now is a much better informed public debate about how such a system can be made to work in the best possible interests of us all. After all, this is not just about delivering another “government technology project”, but about addressing wider societal and policy issues too.


[1] See for example some of the techniques developed by Stefan Brands and his former company Credentica here.

[2] Homomorphic encryption

Homomorphic encryption is a form of encryption that allows calculations to be carried out on encrypted data. It generates a result that is the same as if the data had not been encrypted. This means, for example, that it is possible to perform calculations on financial data that has been encrypted without revealing the actual data. These characteristics make it useful for deployment in a system where a considerable amount of personal data is collected in one place – such as the proposed citizen online portal. It would potentially enable data to be encrypted and hence inaccessible to anyone except the owner (i.e. the citizen) but would still enable e.g. HMRC or banks to perform calculations on that data. As with blockchain technology (see below), it is not yet fully mature – but another area where government can play an important role in helping drive its development and adoption. You can read a bit more about it in this “American Scientist” article.

[3] Blockchain

A blockchain is a public ledger of all transactions that have ever been executed. It provides proof of all the transactions on a network and a full history of transactions. Transactions are entered chronologically in a blockchain and the blockchain database is shared by all nodes participating in a system so that no single node can ever be in the position of falsifying or tampering with it. The full copy of the blockchain has records of every transaction ever executed. It could be used to ensure that our transactions have happened and cannot be tampered with, yet it can also potentially retain a degree of anonymity – which is why it has provoked such interest with Bitcoin, the digital currency, since it enables secure financial transactions to take place without necessarily revealing who is involved in those transactions.

Within a system using blockchain technology, users can be identified only by their public keys. The mapping of a user to their public keys is held on that user’s node only and each user can generate as many public keys as they want, using each in a different context (such as a transaction with a particular retailer), and potentially also using one-time public keys to further reduce the risk of anonymity being compromised.These characteristics – proof that a transaction, such as a purchase or payment of tax, has happened, combined with potential anonymity – make it a candidate technology to be considered for use in a twenty-first century system. However, the ability to maintain anonymity may require better design than that currently used in the Bitcoin network, as this paper (PDF) points out, and its vulnerability to concerted manipulation by an adversary with sufficient computational power remain a concern. This piece is also interesting about its current limitations and (possible) future direction.

[4] Certificate Transparency

This technology is centred on a public, verifiable, append-only log — see Ben Laurie’s 2014 post here.

[5] Guardtime

This technology provides real time detection and mitigation of integrity loss in network infrastructure. It aims to combat cyberattack and data breaches through the use of a blockchain-based digital signature system for real time validation of electronic data. See this PC Advisor article and this Wikipedia entry.

Posted in digital, future Britain, identity, IT, IT strategy, open government, privacy, public services, security, social exclusion, social inclusion, taxation, technology, technology policy | 1 Comment