Written evidence submitted by Professor Thiemo Fetzer (UAIG0006)
Response by Professor Thiemo Fetzer, Professor of Economics, University of Warwick and University of Bonn. Visiting Fellow at the London School of Economics and a Fellow at the National Institute for Social and Economic Research (NIESR) and Scientific Advisor to the Applied Economics AI group.
Professor Thiemo Fetzer has extensive experience collaborating with public sector entities in the realm of public data, including the Office for National Statistics (ONS) and the Competition and Markets Authority (CMA). He serves as a Visiting Researcher at the Bank of England. He has worked with big data startups and was involved in the Royal Society DELVE – Data Evaluation and Learning for Viral Epidemics group during the pandemic producing rapid research insight. His research leverages AI across various domains related to public sector service delivery with a specific focus on the official statistics and administrative data.
This submission addresses the risks and opportunities associated with AI adoption in government, focusing particularly on data and skills issues within the public sector, and provides two case studies drawn from Professor Fetzer’s ongoing work.
1. EXECUTIVE SUMMARY
The deployment of Artificial Intelligence (AI) in the public sector presents an unparalleled opportunity to redefine the role of the state and revolutionize governmental operations.
By easing skills constraints, enhancing data integration and management, increasing the accessibility of public services, and improving data collection and dissemination, AI can deliver significant improvements in efficiency, service delivery, and citizen engagement.
Embracing AI thoughtfully and proactively will enable the public sector to meet future challenges effectively, delivering better outcomes for citizens while upholding transparency, accountability, and ethical standards.
Key areas of opportunity include
● Easing skills constraints by simplifying complex data processes that currently require specialized knowledge and automating tasks.
● Enhancing data integration and management
● Reducing interoperability barriers between proprietary software systems
● Increasing the accessibility of public services by reducing barriers and streamlining interactions
● Enhancing the quality and speed of data collection and dissemination, essential for informed decision-making and responsive governance.
Prerequisites and barriers:
● High-quality data is a prerequisite for successful AI deployment and to maximise AIs full potential.
● However, experience with public sector data reveals significant challenges around data quality, siloed data, and low levels of public trust.
● There is a multi-layered skills gap which will hinder AI deployment in the UK
Recommended actions
● Invest in Training and Skills Development: Implement comprehensive training programs to equip public sector staff with the necessary skills to leverage AI tools effectively and responsibly.
● Lean on research and development partnerships between higher education, private- and third sector partners to develop sandpit-style testing applications
● Enhance Data Quality and Integration: Prioritize data quality initiatives and develop robust data integration frameworks, including the use of unique identifiers to facilitate seamless data merging while preserving privacy ideally with federation and decentralisation.
● Adopt Secure AI Solutions: Deploy self-hosted AI systems within government IT infrastructure to harness AI capabilities without compromising data security or privacy.
● Foster Trust through Transparency: Build trust by establishing clear guidelines and transparent practices for data handling and AI deployment, addressing privacy concerns proactively.
● Encourage Organizational Change: Promote a culture of innovation and adaptability within public sector organizations to overcome institutional inertia and embrace the transformative potential of AI.
● Leverage Open-Source Tools: Utilize open-source AI and software solutions to reduce dependency on proprietary systems, enhancing interoperability and cost-effectiveness.
2. The Value of AI in the public sector
The deployment of Artificial Intelligence (AI) in the public sector presents an unparalleled opportunity to redefine the role of the state as a platform provider, enhancing service delivery, improving operational efficiency and lower barriers to access. AI can reduce barriers to access to public services, reducing digital divides through the use of conversational AI and may further lower transaction cost with the public sector owing to regulation. The widespread adoption of AI can foundationally change the role of the state, it may redefine the boundaries of the firm and may lead to a redefinition of informational boundaries more broadly.
There are at least three broad elements relating to present constraints and challenges around deploying AI:
3. Easing Skills Constraints within the Public Sector Workforce
The COVID-19 pandemic exposed critical deficiencies in the public sector’s IT capabilities, particularly in agile data collection, analysis, and dissemination. These shortcomings hindered effective pandemic response and underscored the need for enhanced technical skills across various tiers and branches of government.
● Relief to Technical Staff Shortages: AI tools can alleviate pressure on scarce technical staff by simplifying complex data processes that currently require specialized knowledge. By automating tasks such as data entry, transformation, and standardization, AI enables staff with less technical training to perform advanced data handling actions.
● The AI-enabled increased accessibility of more complex software or technical backends can reduce the often common specific narrow but deep skill sets that are scattered across functional units empowering even users with only cursory training e.g. in one type of software or programming language to meaningfully engage with software and technology without having deep knowledge. This can reduce challenges that may arise e.g. with older IT systems or architectures which age with the staff around them, reducing hold-up problems or insider/outsider conflicts that may be a barrier to adoption of new technology or may hinder the public sector to play a more proactive entrepreneurial role.
● Adoption of No-Code Platforms: Large language models can help translate complex code into plain text, making it easier for non-technical staff to engage and communicate with more technically minded colleagues or external stakeholders. This may facilitate inter-departmental operations and improves engagement with external entities such as private and third-sector organizations working collaboratively with the public sector delivering services; carrying out pilots or carrying out functional evaluations.
● Automating Routine Tasks: AI can automate many routine tasks such as data collection; data entry; data transcription; information or data retrieval; basic lookups, and regular reporting. Conversational AI with function calling capabilities can increase the quality of many chatbot style systems as are presently used, reducing the need for waiting times for customer service. This may encourage the staff to become more specialised. This shift leads to enhanced productivity and reduces the risk of errors in data handling, while maintaining traceability.
4. Enhancing Data Integration and Management
AI deployment can enhance the quality of data integration across different users, which is key to unlocking transformative insights that inform policy and decision-making.
● Improved Data Quality and Linkage: AI can facilitate the seamless integration of heterogeneous data sources, placing a strong emphasis on data quality. Enhancements in data integration methods can overcome existing barriers, such as linking datasets without a common identity layer. For example, many non-standard tasks that private- or third sector collaborations with government entities involve require the merging of datasets from different sources based on standardized address data. This can be improved through the rollout of numeric spatial identifiers like Unique Property Reference Numbers (UPRNs), lowering the costs of administering and maintaining records.
● Accelerated Data Processing: AI enables more rapid data processing, leading to timely updates of crucial statistics and indicators. This acceleration is vital for responsive governance and effective public administration.
● Privacy Compliance and Data Security: Advanced AI tools can streamline data handling while ensuring compliance with data protection regulations (e.g., GDPR) and maintaining the integrity of sensitive information.
● The value of AI and in particular large language models increases exponentially in the degree to which data are integrated on the backend – that is, if the large language model is provided with the ability to query data from different data sources. This is challenging in the UK due to the absence of a strong common ID layer (absence of national ID) that creates the data merging complications highlighted above.
5. Increasing Accessibility of Public Services
AI can transform the interface between citizens and public services, reducing operational frictions and making governmental assistance more accessible to disadvantaged groups.
● Efficiency in Service Delivery: Conversational AI and automated decision-making systems can streamline interactions between citizens and government bodies in multi-modal fashion, e.g. via telephone, email helplines or semi-supervised chat bot interactions that leverage bespoke instances of large language models using function calling. This can reduce waiting times for helplines and (semi-)automate personalised responses to common inquiries, improving the overall user experience.
● Enhanced Regulatory Compliance: By automating compliance tasks, AI can lower the cost and complexity of meeting regulatory requirements, ensuring more transparent and accountable government practices, lowering cost of doing business substantially. Future business tax filing, for example, for VAT can be made fully automatic with the integration of accounting systems, with banking systems through open banking, together with digitalized receipts with data being extracted using LLMs. This can lower the cost of VAT tax compliance and may finally allow a substantial reduction in the VAT registration threshold that, by international standards, is very high in the UK.
● Reorientation of the Public Sector Workforce: As AI assumes routine back-office administrative tasks, takes over much of the conversational labor intensive handling tasks, there is potential for workforce reorientation toward more value-adding and citizen-facing roles. This shift enhances the overall service experience and maintains a human touch where desired. The organization of the civil service may need to be reoriented to provide more frontline services, particularly given the digital divide. Public sector workers can engage more with third-sector or private sector counterparts to carry out pilots or conduct research, shifting focus from administering to becoming an active integral part of the innovation ecosystem.
6. Improving Data Collection and Dissemination of Official Statistics
AI can significantly enhance the quality and speed of data collection and dissemination, which is essential for informed decision-making and responsive governance.
● Enhanced Data Collection Methods: AI-assisted enumerators and intelligent data collection tools can reduce the costs and burdens associated with traditional survey data collection. Semi-structured interviews and AI-driven surveys can gather data more efficiently and effectively at lower cost. Further, common tasks in national statistics such as mapping data across different classification systems or terminologies (e.g. for industry classification of firms or product classifications) can be easily automated with large language models increasing the accuracy, replicability and timeliness of data.
● Real-Time Data Processing: Integration of administrative data into official statistics pipelines, facilitated by AI, can increase the timeliness and relevance of disseminated information. During the pandemic, initiatives like the ONS’s Business Insights and Conditions Survey (BICS) provided vital real-time measures of economic health. The continued cost of collection of such data can drastically decrease, the sampling frame or spine can be increased.
● Improved Data Accuracy and Relevance: AI can help or make it easier for non technical staff to curate data or identify, for example, outliers and help with the data preparation which is often one of the biggest factors that induces delays in the publishing of official statistics. AI can also be used for interactive engagement with datasets for more challenging tasks such as pattern recognition or the detection of trends, leading to more accurate and actionable statistics. This capability supports proactive policy development and better resource allocation.
7. Importance of High-Quality Data for AI Adoption
● High-quality data is a prerequisite for successful AI deployment and to maximise AIs full potential. Experience with public sector data reveals significant challenges:
○ Data Quality Issues: Address data is often outdated or inaccurate. In some cases, data is intentionally obfuscated to preserve privacy, which can impair effective administration or undermine public sector effectiveness. This poses a huge challenge and a risk to the UK realising the many upsides that AI can offer through data integration. Some concrete examples consist of challenges in mapping legal ownership, geographic location of economic activity or business activity etc, due to the obfuscation of information on different data spines (e.g. company address via companies house vis-a-vis registered office address vis-a-vis accountants address vis-a-vis business address vis-a-vis bank address vis-a-vis statement address vis-a-vis office address for PAYE registration purposes etc.) and the use of ubiquitous and manifold different identifiers that are not mergeable.
○ The lack of mergable data reduces the upside potential of AI as naturally as a data owner, e.g. a company owner or an individual, being able to query the data that public sector entities hold about oneself can be incredibly valuable. For example, an integrated data spine could allow citizens to update their residential address upon moving with all address references being automatically updated or at least, this update process being streamlined. Lack of data integration at present results in huge amount of waste and provides a risk for identity theft of address to individual or company is important to ensure continued compliance
○ For example, in selective licensing schemes, the beneficial owner’s address might not be disclosed, or provided addresses may not be validated. Semi-autonomous AI agents could, at regular intervals, perform routine tasks such as database maintenance and record validation with users, in the absence of a decentralised data layer ensuring data on record is up-to-date.
○ Lack of Unique Identifiers: The absence of a strong public identity layer (the UK does not have a system of national ID) complicates the linking of datasets, making data integration more difficult. Without unique identifiers, merging data from different sources becomes a complex task and there may be structural data quality concerns.
● Data Integration Challenges: AI’s potential to boost public sector productivity hinges on reliable data and the ability to merge data across silos while preserving privacy. Technological tools for data integration exist but are underutilized, often due to the technical skill level required for implementation.
● Trust and Privacy Concerns: Low levels of trust in public sector may lead to data being purposefully obfuscated. Building trust in data handling processes is essential to ensure data quality without compromising privacy.
8. Barriers to AI Adoption
Despite the significant potential benefits, several barriers hinder AI adoption in the public sector:
● Trust Deficit
○ Privacy Concerns: There is apprehension regarding how data might be exploited, leading to a lack of trust in data handlers. Concerns include data being used for unintended purposes, such as algorithmic pricing or pattern detection that could disadvantage individuals or groups.
○ Data Breaches and Mishandling: Past incidents of data breaches and the mishandling of confidential information have eroded public trust. Building trust in AI and data management practices is crucial for safe and ethical AI deployment.
● Multi-layered skills gap
○ Challenge may consist of lack of deep-layered understanding of how AI works. The extent to which a user can leverage AI is significantly informed by the intellectual breadth and depth of the user. The UK’s high degree of specialisation in (higher) education from an early age onwards may be problematic here as it may lack the broad set of potential users that can see applications beyond their specific area of expertise or domain.
○ Technical Skills Limitations: A lack of deep technical skills across many tiers and branches of the public sector limits the ability to adopt and effectively use AI tools.
○ Need for Training and Paradigm Shift: Many users are unfamiliar with leveraging AI to its full potential. Comprehensive training and support are necessary to help public sector staff engage with AI technology, including an understanding of privacy issues inherent in training and using AI. The UK may want to lean on its higher education sector here to boost the upskilling of its public sector workforce. The fact that AI skills are in short supply and highly valued may require the public sector to adopt flexible compensation schemes to ensure it can retain access to a talent pool that may easily migrate or move to better opportunities.
● Resistance to Change: There may be institutional inertia resisting organizational changes required for AI adoption. While many roles could be enhanced or automated, willingness to upskill staff and adapt workflows is essential for embracing this transformational technology.
○ Inertia may be due to vested interests. One natural avenue is traditional software suppliers. These typically see their product evolve with their client needs resulting bespoke software solutions that have evolved alongside the specific user base. This creates mechanical barriers for adoption of new solutions if, for example, legacy systems are only understood by few users within organisations. This may be even more severe if legacy systems use software licensing with closed code bases.
○ Need for Workforce Reorientation: AI adoption may necessitate a reorganization of public sector roles, with a shift towards more frontline services and increased engagement with third-sector or private counterparts.
● Risks and resistance – the biggest productivity gains are to be had in many areas of the economy that revolve around managing the flow of data or information. Opposition and resistance to adoption of AI may well come from much of the traditional service sector. This could put stress on occupational license systems as these qualifications may effectively lose their economic value. Specific sectors at risk are: accountancy; audit; tax filing; legal services; data entry or record keeping; many services adjacent to data entry or validation often linked to legal services related to real estate transactions. Many of these services will become automatable or become obsolete requiring firms operating in these domains to innovate their business models. For example, small accounting firms may need to move into business intelligence and business development domains to help their clients retrieve the value from their business data. This may require a more proactive engagement of such firms with their clients possibly even aiding their digitisation. This may well have positive side effects reducing market concentration in some domains and could be a boon for small and medium sized business.
9. Benefits of AI Adoption: Better Integration Between Proprietary Software in Local Government
● Interoperability Barriers: The closed nature of IT systems used by many public sector entities may have been a main barrier hindering deep technology integration in the public service. Software providers have strong incentives to create interoperability barriers as part of their traditional license-based monetization strategies. These barriers would commonly be referred to as “moats”. Limits of interoperability may cultivate reliance on niche skills or secure recurring revenue through licensing and subscriptions. These practices undermine the potential for AI adoption more broadly.
● Streamlining Data and Workflows: AI can make it much cheaper to connect disparate systems, facilitating the development of Application Programming Interfaces (APIs) that enable faster information flow. By automating integration between proprietary systems or lowering the cost of doing so, AI can significantly enhance operational efficiency.
● Reducing Reliance on Proprietary Solutions: Adoption of AI and open-source tools can reduce dependence on proprietary software, lowering costs and increasing flexibility.
10. Conclusion and Recommendations
The strategic deployment of AI in the public sector holds immense potential to revolutionize governmental operations. By easing skills constraints, enhancing data integration and management, increasing the accessibility of public services, and improving data collection and dissemination, AI can deliver significant improvements in efficiency, service delivery, and citizen engagement.
Recommendations:
● Invest in Training and Skills Development: Implement comprehensive training programs to equip public sector staff with the necessary skills to leverage AI tools effectively and responsibly.
● Lean on research and development partnerships between higher education, private- and third sector partners to develop sandpit-style testing applications
● Enhance Data Quality and Integration: Prioritize data quality initiatives and develop robust data integration frameworks, including the use of unique identifiers to facilitate seamless data merging while preserving privacy.
● Adopt Secure AI Solutions: Deploy self-hosted AI systems within government IT infrastructure to harness AI capabilities without compromising data security or privacy.
● Foster Trust through Transparency: Build trust by establishing clear guidelines and transparent practices for data handling and AI deployment, addressing privacy concerns proactively.
● Encourage Organizational Change: Promote a culture of innovation and adaptability within public sector organizations to overcome institutional inertia and embrace the transformative potential of AI.
● Leverage Open-Source Tools: Utilize open-source AI and software solutions to reduce dependency on proprietary systems, enhancing interoperability and cost-effectiveness.
● Embracing AI thoughtfully and proactively will enable the public sector to meet future challenges effectively, delivering better outcomes for citizens while upholding transparency, accountability, and ethical standards.
APPENDIX: Case Studies
Case Study 1: Frictions in collaboration with public sector entities
A practical example highlighting the challenges of collaboration between public, private, and third-sector entities involves a research team working with a local authority to facilitate retrofitting projects aimed at accelerating the UK’s transition to net-zero emissions. The goal was to identify barriers to retrofitting in the private rental sector and assess if innovative finance solutions could assist landlords finding it difficult to secure traditional mortgage financing. Crucially, the project needed to comply with GDPR regulations to ensure ethical handling of personal data.
Challenges Faced:
● Data Query and Consent Process: The initiative required querying the council’s selective licensing database to identify landlords with properties below certain energy efficiency standards. This entailed tasks such as removing duplicate records, flagging problematic or incomplete entries, extracting contact details, and contacting individuals with bespoke consent forms. While these tasks are routine for academics using tools like R or Python, the sensitive nature of the data meant they had to be performed by a case officer within the local government that lacked adequate training and skills.
● Technical Limitations: Council staff lacked the permissions to use tools for efficient data handling on their local council devices citing unfounded IT security concerns. They were unable to utilize programming languages like R or Python, relying instead on less efficient methods.
● Delays and Inefficiencies: The research team had to resort to developing Excel macros to handle basic data manipulation. Building these macros remotely led to numerous revisions and significant delays. What could have been a straightforward task completed in a few hours extended into a two-month ordeal due to technical constraints and inefficiencies.
● Potential Solutions that AI could support:
○ If local council staff were allowed to use GPT or would have access to a local LLM installation with a PrivateGPT in the councils own networks, the co-development of the code that performs the above tasks could have been done in a 1-2 hour remote session with the council staff without any data ever needing to leave the council IT systems ensuring compliance with GDPR and IT security.
○ Alternatives to local LLM installations would be the hosting of a local repository for R and python packages that is vetted for use on government IT systems, such a repository could have made it possible for the research team to simply share functional code with the local council staff.
● Outcome Potential:
○ Significant Time Savings: Leveraging AI could reduce processing times from months to mere hours, accelerating project timelines.
○ Improved Data Reliability: AI-enhanced systems minimize human errors, increasing the reliability and efficiency of data handling processes.
Case Study 2: Local AI Deployment on Government IT Systems
Most government entities presently have no AI deployment or use policy in place. This is a natural standpoint due to potential and known risk of handling private data in cloud-based systems. The outright ban or lack of guidance causes uncertainty and may lead to risky behavior with private accounts being potentially used for official work.
Context and Challenges:
● Restrictions on AI Tools: Government staff are often prohibited from using cloud-based AI tools like ChatGPT due to risks of private data being processed without proper consent, leading to potential privacy violations.
● Need for AI Functionality: Despite restrictions, there is a need for AI capabilities to enhance productivity and efficiency in data handling and other administrative tasks.
Local AI Deployment Solutions:
● Self-Hosted Large Language Models (LLMs): Government entities can deploy self-hosted AI models on their own IT infrastructure providing accessibility very similar to what is possible with ChatGTP and other commercial LLM providers through inbuilt APIs. It is further possible to deploy such platforms without the required large compute as efficient mid-scale LLMs can be efficiently handled on regular consumer hardware. By utilizing open-source large language models, agencies can create AI tools that function similarly to ChatGPT but operate entirely within secure government networks.
● Integration with Existing Applications: These local AI deployments, for example, using llamafile, Ollama, llama.cpp can be integrated into existing applications for which public sector workers may have access to already such as Copilot or various office products. This way, a readily available AI chat assistant could be embedded within data management software, offering real-time support without compromising data security.
● Cost Efficiency: Self-hosted AI solutions reduce the need for expensive licensing agreements associated with commercial AI platforms. They offer substantial functionality at a fraction of the cost while maintaining full control over data privacy.
Final thoughts: Structural reconfiguration of the economy
● Adoption of AI has the potential of redefining whole sectors in particular in the service sector domain and in particular around the many sectors in the economy whose role is primarily to mediate relationships between firms, individuals and the state. Examples of such business activities are the filing of tax returns, VAT returns, and other legal requirements can become nearly automated with the integration of open banking, invoice-level VAT supported by a digital tax system. This will lead to drastically lower cost of entrepreneurship that benefits from limited liability, it can dramatically reduce taxation gaps and may lead to expanded fiscal space. The expansion in entrepreneurial possibilities could give rise to a healthy and growing small and medium sized company boom. .
● Yet, this also implies that many of the traditional roles that licensed service sector occupations would typically play in the economy will change fundamentally. Accountancy service providers, tax advisors etc., would see drastically lowered revenues due to lower cost of service provision in a digital economy. Invariably the foregone revenue streams may need to be compensated for through the expansion of the services that these entities may offer to their client. These services may well extend into the business advisory domain analysing increasingly digital data from their companies to study patterns or to identify areas for improvement in the cost structure and the offering of firms.
● Broad adoption of AI in the public sector could or should recognise the provision of software as well as data hosting services for the maintenance of digital records for companies as part of the public sector plumbing that supports SME development and growth through the provision of an efficient and cheap infrastructure supporting firms meeting their legal requirements.
December 2024