A (brief) history of UK Government moves towards a platform-based architecture

The vision — 1998 onwards

There’s been a long-standing vision to use technology in a way that doesn’t fossilise the old paper-based way of running the public sector — but which enables services to be redesigned and improved. Part of this vision has seen a move over the last few decades towards common services that remove duplication and enable the delivery of services designed around the citizen (rather than the owning department or agency).

The Parliamentary Office of Science and Technology (POST) back in 1998 provides the first recorded effort I’ve come across that considers the impact of technology on government and all of its services in a more holistic way. Their report on “electronic government” included the following schematic as part of a discussion about common processes across government and the need to avoid merely using technology to serve up online versions of paper forms.

Figure: the 1998 view of common services

[Source: POST Electronic Government 1998]

The UK government’s architectural approach to development of these more joined-up services was set out in the 1999 “Portal Feasibility Study”, prepared for the Central IT Unit (CITU) in the Cabinet Office. It proposed the three-tier architecture shown below.

Figure: The 3-tier conceptual architecture, 1999

[Source: Portal Feasibility Study]

This architecture was designed to support a wide range of access channels, and to enable the existing IT estates of departments and agencies to integrate without requiring a big bang, rip and replace approach. It proposed that

A three-tier architecture can be used to insulate the access channels from the complexity of the Government Back Office with web technology providing the portal, or gateway between the channels and the individual service requested. The key concept of the three tier architecture is the use of middleware technology to provide a brokerage capability, a concept that sits well with the idea of a portal. The middleware will link components to allow them to interact without the need to have knowledge of the other component’s location, hardware platform, or implementation technology. [p.7]

 It included an emphasis on the value of open standards:

The technical implementation of the three-tier architecture must provide the glue to link existing Departmental services and systems to a wide range of different access channel technologies. This means that open standards need to be proscribed and that the interface standards needed to ensure good interworking must be defined.

An open architecture will maximise the flexibility and opportunities for infrastructure provider competition. Every major interface in the architecture will need to have an interface specification defined for it. This will allow architectural components, services and supplier systems to be replaced easily and a ‘plug and play’approach to be taken to architecture components, services and supplier systems. [p.8]

For the specific standards, reflecting some of the dominant technical models of the time, it recommended:

Tier	Element	Recommendation
Front	Web server protocol	HTTP
	Application server candidates	CORBA ORB (supporting services written in JAVA, or DCOM and Microsoft Transaction Server)
Middle	Application server architecture candidates	CORBA ORB
Back	Suggested protocol for communication with Government departments	SMTP
	Suggested messaging standard	S/MIME

Table: Proposed Technical Standards

[Adapted from “Portal Feasibility Study, 1999”]

Confidentiality was to be assured through the use of encryption between the desktop and the Web server, specifically SSL or HTTP/S. Public Key Infrastructure (PKI) would be used in the middle tier, with a firewall deployed to limit network access between the front and middle tiers. Messaging between the middle and back tiers would use structured messages digitally signed by the Web site to: protect the integrity of the information, authenticate the source and prevent repudiation. The internal government networks (the Government Secure Intranet, GSI, and xGSI) would be used to secure the physical transmissions within government’s own estates.

From concept to engineering

The approach — building out a single online web presence for the whole of government —  outlined in the “Portal Feasibility Study” was later to be iterated through a series of implementations. The initial site was “UKonline”, replacing the early portal efforts of the Government Information Service (GIS) at open.gov.uk first established in 1994. This appeared as a soft launch, beta site in November 2000, providing time to gather feedback and enable refinement and improvements prior to its formal launch in February 2001. It was redesigned, rebranded and relaunched on 1st March 2004 as Directgov. The current GOV.UK first appeared in May 2011 and went officially into full service in October 2012.

However, my focus in this brief blog is not the front-end website but a range of other, associated platform components, such as those developed to secure the processing of transactions and to enable the reliable identification, verification and authorisation of citizens, businesses and intermediaries.

From late 2000 onwards, the UK government built out a series of web-service / SOA-based components. There was a clear business case for taking this “whole of government” approach: it was already evident that without some degree of common services, each part of the public sector would proceed with its own local initiatives — from websites to identity systems to payments systems and so on. The same needs would be met time and again in multiple places, resulting in pointlessly duplicated expenditure, processes and systems.

This would not only be expensive, and duplicate common infrastructure in multiple places across the sector, but also produce a poor user experience:  government services would be scattered across a technological landscape that replicated the physical world of multiple overlapping providers. Leaving everyone to do their own thing would effectively fossilise existing, paper-based services in the online world rather than taking the opportunity to improve and streamline them.

Open standards were the bedrock on which these initiatives were based, using published APIs (for programmatic access) and XML (for data interchange). Although collectively referred to as the “Government Gateway” (reflecting terminology first used in the “Portal Feasibility Study”), the initial component services consisted of several discrete functional components: 

1. Registration & Enrolment (the identification, authentication and verification of online users — citizens, businesses and intermediaries — utilising a range of credentials, from user ID and passwords to digital certificates to — later — EMV chip and PIN cards)
2. Transaction Engine (handling the processing of transactions between citizens, businesses and departments, including the management of state and orchestration across multiple service providers where a transaction involved multiple entities)
3. Secure Messaging (handling two-way secure communication between departments and citizens)
4. Payments Engine (handling in-bound payments to government departments)
5. Gateway Helpdesk (an internal service enabling departments to integrate a degree of management of these common components services within their existing helpdesk services)

These API-based components are illustrated at a high level below.

Figure: the API-based components developed from late 2000 onwards

The rationale for adopting this cross-government approach has been explained as follows:

“The Gateway was designed to simplify and accelerate the UK e-Government programme. It achieves this by ensuring that the common building-block components of e-Government services are provided once, in a flexible, modular and scalable way.”

[Source: “Delivering e-Government Services to Citizens and Businesses: The Government Gateway Concept”. Jan Sebek, p.127. Published in “Electronic Government: Second International Conference, EGOV 2003, Volume 2”. Editor Roland Traunmüller]

The services were exposed via the following open standards:

• TCP/IP
• HTTP
• HTTP 128 bit SSL (TLS 1.0)
• HTML
• XML
• X.509 certificates
• W3C XML digital signatures
• SOAP
• SMTP
• SAML

The schematic below shows the relationship between these initial component services and also illustrates how the provision of common platform components avoids duplication of the same needs across multiple government services. It came about, in part, because each department was developing its own local solutions to its online service needs, with a resulting duplication of expenditure and fragmentation of services from a user perspective. Moving to a strategic, platform-based approach for common services made sense from an economic, architectural and user experience perspective.

[Source: “Delivering e-Government Services to Citizens and Businesses: The Government Gateway Concept”. Jan Sebek, pp.125-128. Published in “Electronic Government: Second International Conference, EGOV 2003, Volume 2”. Editor Roland Traunmüller]

All of the technical standards used were stipulated by government to be open, enabling interoperability regardless of the software or vendors involved across the many connecting organisations:

“A wide range of systems have interoperated with the Government Gateway since its launch, including systems running Sun’s J2EE technology, IBM technologies, Apache, Tomcat and other technologies and applications including standalone PC application software.”

[Source: Preliminary Study on Mutual Recognition of eSignatures for eGovernment applications NATIONAL PROFILE UK April 2007. p.16. IDABC European e-Government Services.]

Building on the Platform

Following on from the successful release and implementation of the initial cross-government components, in 2003 a further set of services were considered.

Some initial work was done to prototype various of these additional platforms, but internal issues relating to funding and agreement between the multiple stakeholders involved became increasingly problematic. The recurrent issue of how much government should itself try to build in-house and what it should consume or purchase from others was also as much a debating point then as it is now. Hopefully this time around the debate about continuing to develop the vision of government at the technical platform level will be informed by the use of techniques such as Wardley mapping, which can help identify when and where in their evolution the various needs are and therefore how they might best be sourced (from bespoke build to rental).

 Figure: Wardley Mapping 

[courtesy Simon Wardley]

For those few who have not seen it yet, Mark Foden’s short but useful video on “the Gubbins of Government” is essential viewing. It sums up much of what has been intended and what remains to be done very succinctly to implement the platform model at scale:

Current era

In 2010, “Better for Less”, which set out many of the ideas implemented in government during the 2010-2015 period, emphasised the value of the component-based approach.

The use of a common infrastructure is critical … applications must be developed to work together using open standards for data and interoperability. It is critical that the common infrastructure is available for all the identified stakeholders: officials, citizens, third sector organisations, potential and actual solution providers …

… Government IT is well placed … if it adopts this construct: service oriented architecture, open standards, re-use of components and a carefully thought-through incentivisation environment encouraging and supporting innovation and re-use. It can take advantage of a cheap, commodity platform, competitive suppliers, constant innovation, and automatic sustainability.

In 2013, GDS published in its Service Design Manual its intention to continue this journey towards “Government as a Platform”. The more detailed text available at the time appears to have been removed from the site, but is still viewable in GitHub here. It recognised the need to work smartly including:

A move to platforms does not mean that government has to develop everything in-house: many of government’s needs can be met by existing cost-efficient utility services. However, government can help to establish best practice in areas such as personal data privacy.

Wherever appropriate, the government should use existing external platforms, such as payments services (ranging from third party merchant acquirer services to the UK’s national payments infrastructure). Deciding to develop platforms in-house will happen only where that is the best way to meet users’ needs in the most flexible and cost-effective way.

Government will draw upon the experience of other organisations that have already made this journey. Many have created platforms that initially sit across the top of existing silos to expedite agile and effective digital service delivery.

This co-existence enables the benefits of the platform model to be realised quickly, even in a brownfield environment such as government, while the silos below the waterline are gradually reduced in importance and eventually made obsolete.

This is the approach that government will follow, ensuring that it develops a well-defined schedule for switching off legacy environments as the platform model is progressively implemented.

Another announcement about this commitment was made in September 2014 by Sir Jeremy Heywood, Cabinet Secretary and Head of the Civil Service, in a blog entitled “More than just websites“. It seems clear that the continuing move towards a platform-based model is now generally accepted not only at the engineering level but, as importantly, at the business and services level too.

Where next?

We shall see after the election 🙂

Technical Annex

I originally planned to include the following more technical detail in the body of my blog. However, it’s a bit distracting for those only interested in the history of efforts to realise the higher level principles and benefits of developing a platform based model. But for those who want to look under the bonnet in a bit more technical detail, here it is … It aims to provide insight into two earlier components developed from late 2000 onwards — the Transaction Engine, and Registration and Enrolment.

Transaction Engine

The Transaction Engine (TxE) was developed to provide a transaction handling and routing service. It supports the exchange of data (usually in the form of documents and business forms) between government organisations and external organisations, intermediaries and citizens, as well as between government organisations. It’s able to manage state between both single point to single point, as well as to orchestrate more complex interactions involving multiple parties where state needs to be understood and managed (e.g. an incoming piece of data may successfully update 3 of 4 departmental systems, but not the remaining one: clear state-management rules can be applied in such circumstances, ranging from rolling back all of the updates through to providing response information that indicates which systems have been updated and which have not and leaving the client side to decide how to progress the interaction).

TxE provides two main methods of utilisation:

1. for ‘ad-hoc’ use, a defined submission protocol involving the exchange of XML documents posted into a URL destination point

2. for permanently connected government and related entities, two-way messaging via Departmental Integration Servers (DIS) between the hub and those spokes involved in the messaging exchange

Although the original TxE was based on proprietary technology, the protocols developed and used were vendor neutral, open standards-based rather than utilising Microsoft’s BizTalk Framework. The development of these platform components were managed within the context of the broader e-Government Interoperability Framework (e-GIF) and its associated GovTalk initiative to use open interoperability standards (which drew primarly upon the IETF, W3C and WS-I interoperability standards).

XML was used as the standard data format for all messages into and through the TxE. The specific standards utilised were defined by the former GovTalk initiative. This is now defunct, with the UK Government’s Open Standards Board taking renewed and reinvigorated ownership of the open standards to be used in government for software interoperability, data and document formats.

Interoperability between the TxE and existing departments is accomplished through what became known as the “Departmental Interface Server” (DIS). DIS provides SOAP reliable two-way communication between the connecting organisation and the TxE. It combines compliance with TxE’s open standards (XML, HTTP and SOAP) together with reliable once-only delivery and separation of the integration requirements of the systems within the connecting organisations.

Figure: Standards

[Source: Government Gateway technical documentation]

Some departments and other organisations deployed DIS services developed by third parties rather than using the Microsoft Biztalk-based offering: SoftwareAG for example supplied HMRC and others, demonstrating the practical benefits of open standards in maintaining an open market rather than restricting it to a single supplier.

TxE protocols

Three message protocols are supported by the Transaction Engine:

• Document Submission Protocol: the interface used by ISV Applications and Department Portals. It allows ISV applications or Portals to submit business transactions/forms/documents to Department services
• Hub and Spoke Interchange Protocol: used for Department to Department business transactions/forms/documents submission
• Administration Message Protocol: an asynchronous message based version of a more recent SOAP Admin Interface

GovTalk Message Envelope and Document Submission Protocol

DIS uses the GovTalk Message Envelope in conjunction with the Document Submission Protocol (DSP) from 2008. The DSP routes business transactions (e.g. Self Assessment tax forms) submitted from either a Department Portal (e.g. the HMRC Online service Web site) or directly from an ISV application (which could be hosted on a website, server-based, PC-based, smartphone-based etc.), through the TxE, to the appropriate Department (back-end) system and retrieves the corresponding response. The TxE acts as a “hub” to numerous “spokes”, which can be websites or dedicated DIS endpoints that provide onward integration and interoperability into departmental local systems.

DSP uses the GovTalk Message Envelope to encapsulate business transaction documents. GovTalk documents are XML formatted and use the UTF-8 encoding standard. Messages are transported using the Hypertext Transport Protocol (HTTP). Portals, ISV applications and Departments (using DIS) must be capable of generating HTTP 1.1 POST requests and receiving and interpreting HTTP 1.1 response messages.

The transaction architecture is componentised, with applications and portals separated from the systems with which they interact. So, for example, HMRC’s portal does not talk directly with its backend, but via the DSP and GovTalk Message Envelope, routed and authenticated via the TxE. Connections with the TxE are made either over the internet or via government networks (formerly the GSI, now the PSN).

Figure: end-to-end overview of the transaction-handling system

[Source: Government Gateway technical documentation]

The typical sequence followed when a client application submits a document to a target spoke (i.e. a Department service) is shown below (assumes no errors).

Figure: overview of the submission-response protocols

[Source: Government Gateway technical documentation]

The protocol makes extensive use of the envelope portion of the GovTalk schema, requiring XML documents submitted to TxE to include a Qualifier element immediately after the Class element. Together these two elements denote the message type.

Messages issued by the client application:

• SUBMISSION_REQUEST
• SUBMISSION_POLL
• DELETE_REQUEST
• DATA_REQUEST

Messages issued by the Gateway: 

• SUBMISSION_ACKNOWLEDGEMENT
• SUBMISSION_ERROR – error detected in message received from the Client
• SUBMISSION_RESPONSE/ERROR – business response/error from Target Spoke (i.e. Department)
• DELETE_RESPONSE
• DATA_RESPONSE

When submitting any message type to TxE it’s the responsibility of the client to ensure each message conforms to the relevant syntactical rules for that particular type of message. A client application does not necessarily have to process each document sequentially as described above. Instead it could operate in a batch mode, submitting a number of documents over a period of time and then later using: 

• DATA_REQUEST to examine the state of these submissions
• SUBMISSION_POLL to retrieve the corresponding response for each submission
• DELETE_REQUEST to delete each submission from the Gateway
• SUBMISSION_REQUEST to resubmit documents with recoverable errors

Registration and Enrolment (R&E)

The registration and enrolment (R&E) service was launched in January 2001. The initial authentication and authorisation component supported two credential types, UserID/Password combinations and digital certificates. The service continued to be enhanced after launch and later expanded to include WS-Security tokens (offering the potential for other trusted issuers to federate their tokens) and the use of EMV Chip and PIN card (from around 2008, offering the potential for users to authenticate using a third party card, such as one issued by their bank).

Citizens, businesses and intermediaries were therefore able to have single sign-on facilities across all government services — national, regional and local — although the option also existed (should a citizen wish) to utilise a separate method of authentication to each separate government service. This was to enable citizens to maintain the segmentation of their identities across government should they desire to do so.

Part of the problem tackled by this common service is the need to associate an online identity and its associated credential with the different identifiers by which that same user is known within different parts of government.

Figure: mapping an identifier to other known identifiers

[Source: UK Government Gateway Frequently Asked Questions 2005] 

Two interfaces are provided:

• User Interface (web pages): a series of on-line conversational dialogues consisting of web page interactions with end users that support the Gateway registration and enrolment process (see www.gateway.gov.uk)
• Programmatic: APIs available over the Internet, comprising two types:
  • a secure web service interface. This is made available only to trusted users (e.g. government websites and applications), providing authentication and authorisation support, including the ability to register and enrol a user
  • a public web service interface with reduced privileges, available for any website  or application to conduct routine tasks including the authentication of users

Figure: API methods exposed

[Source: UK Government Gateway Frequently Asked Questions 2005]

A Simple Object Access Protocol (SOAP) interface is required for websites and applications to interact programmatically with the R&E API-based service. The parameters included in SOAP messages are required to be well formed XML documents conforming to the XML Schema (XSD) defined by the UK government. This enables maximum re-use across SOAP APIs and for both parties to correctly and consistently validate XML documents.

Figure: MoD use of the Chip and PIN support for online access to shared services

[Source: “The UK Government Gateway Remote Authentication. Jim Purves, E Delivery Team, Government Gateway. Department of Work and Pensions 24/10/2008]

PS. If you’ve got this far, well done. You might also find this previous blog useful in terms of some of the underlying architectural design: high level cross-government architecture — 2003 style