Digital, data, and technology (DDaT) are being used to build shared platforms and infrastructure, and to improve the user experience of government services. But improving services within their current policy and administrative silos won’t achieve the “digital transformation” much spoken of and little delivered.
Let’s step back briefly to 1996 to understand why. In July that year, the UK government set out a radical plan to streamline processes within and between departments:
The techniques and technologies which could enable Government to transform its service delivery operations—process rationalisation and the effective application of technology—could also transform the administrative procedures of Government. This would remove the need to continue with stovepipe processes which cannot interact with one another, and would free Government from organisational constraints.
It was this re-engineering of processes that would simplify the design and delivery of integrated services across Whitehall:
After a change of government in 1997, work started on designing and delivering the technical infrastructure for cross-cutting processes that would help services be redesigned around users’ needs. However, despite being a technical success, it didn’t produce the desired political outcome.
Delivery of the plan
From 2001 onwards, the UK Government implemented a series of cross-government platforms to support the delivery of joined-up services. The best known of these is the Government Gateway. It provided identification and authentication services for individuals, businesses, and individuals acting on behalf of others—an updated version of which is still in use today for HMRC services.
Less well known are some of the other platforms implemented under the Government Gateway brand. One of these was the Transaction platform. It delivered a transformation in the user experience by supporting the delivery of integrated services.
Users could, for the first time, complete a single online form rather than repeatedly entering the same information into a series of separate forms from different government departments. To achieve this level of integration, the Transaction platform ensured interoperability with existing department systems and processes via a Departmental Integration Service (DIS).
How cross-departmental services worked
As the user submitted the integrated form from the cross-government website, the Transaction platform applied a set of business rules. These transformed the data contained within the submission into a series of messages that were exchanged securely and reliably with relevant government departments.
For example, a user could submit one form with details such as their name, date of birth, address, and personal circumstances. The platform managed the submission of subsets of data to different departmental systems. It would then await the responses, collate them, and present a single response to the user. This happened in real-time where possible, or asynchronously where needed—for example, a response might not arrive until the following day if a department needed to run an overnight batch process.
One function of the Transaction platform was to act as a state machine. For those unfamiliar with the concept, applications use a state machine to manage the state of a process or processes and their associated events. It can determine what to do on successful completion of a process or event, or what to do if it fails. As the nerdy joke went, it was a state machine for the state machine. Geddit?!
These state management functions were important. Integrated services interacted with multiple departments, some of which could respond immediately (synchronously) and some of which could not (asynchronously). The Transaction service tracked and managed the overall state of these interactions.
So where did the DIS systems fit in? Departments installed them in their data centres to provide the reliable messaging end point for interactions with the Transaction platform. They could queue, store, and manage messages and data. This was important where a department’s system proved unable to handle the speed and volume of messages involved.
DIS systems used open standards and interfaces. They were available from a range of suppliers, or departments could develop and maintain them. On the department-facing side, they offered a variety of integration adaptors to connect and interoperate with whatever technology was already in place.
What did we learn?
The Transaction platform and associated DIS systems provided the technical capability for joined-up services at the front end and interoperability and data integration with departments’ existing systems and processes at the back end. Great—so wasn’t this “job done”, and time to crack open the beers and pizza and party, party, party? Unfortunately, not.
The technology was the relatively easy bit. Managing state and orchestrating messages and data across multiple systems, both synchronous and asynchronous, is hardly rocket science. Departments successfully used the infrastructure to deliver their online services, but they did so solely on a service by service basis. For example, HMRC received self assessment returns and sent acknowledgements in response to users. But there were no joined-up, cross-cutting services to be seen.
One of the biggest problems was the absence of a senior, Cabinet-level minister or group of heavy-hitting departments tasked with the delivery of joined-up thinking, free to operate outside the traditional administrative boundaries of Whitehall. As a result, there were no integrated policies, processes, or funding.
Departments understandably remained focused on delivering their own services onto the cross-government website, presenting them to users as a series of discrete services with no shared processes or integration. Online services, and the processes behind them, remained as separate as they were during the paper era, just as they do today.
What needs to change?
The cross-government platforms designed and implemented from 2001 onwards took at face value the stated political ambition to redesign services around users and streamline common processes. But that never happened in any meaningful way.
Meanwhile, here we are in 2022. There remains a near total absence of cross-cutting policies and the services to go with them. DDaT remains confined to work within the longstanding administrative silos and boundaries of the past, delivering waterfall government. If governments want to design and deliver joined-up, integrated policies and services, funding models and incentives need to break out of the vertical mindset and focus on outcomes and effectiveness instead.
This situation won’t improve until governments better understand how to use DDaT, and integrate it into the heart of the political and policymaking process. This is where it can add real value, as the best digital organisations demonstrate. DDaT can help provide continuous feedback and data to inform and update policymaking; enable rapid experimentation to learn and adapt faster; improve organisation design; engage citizens in the co-creation and co-design of their services; and exploit platform technologies for innovation, efficiency, agility, and scale.
Improving the real state machine requires far more than using digital, data, and technology to polish the status quo. But then the real blocker of “digital transformation” has never really been technical—it’s political:
A sense of crisis and an ideology of reform are necessary, but not sufficient conditions for reform. There must also be a third element: that of the political will and power to enact reform.
Reinventing government in the information age. Richard Heeks. 1999
My forthcoming book Fracture | The collision between technology and how we fix it (published autumn 2022) explores the interplay of DDaT and politics.
New tech observations from the UK (ntouk) 