Using AI to Draft and Evaluate Regulation

Using AI to draft and evaluate regulation

Overview

We have used artificial intelligence to draft two regulatory instruments and to evaluate those outputs against government AI principles. We believe these processes could be among the first of their kind in New Zealand and perhaps globally.

We're ready to do the same thing again with any number of other laws, regulations, policies or other important documents. There are plenty of reasons to be skeptical about whether AI is capable of doing this work well. We also know that no one wants to read AI slop, let alone rely on it in important situations. For that reason, it's important that we explain the processes we've followed and why those processes mean it's worth your time to engage with the outputs we've produced.

This post is a high-level overview. If you'd like further detail on what we've done, or want to inspect some of the working files yourself, you can find more detail in here.

What have we done?

We've applied AI to regulatory documents in three distinct ways:

1. API Standard (Generation)

We converted the NZ API Guidelines (a set of comprehensive guidelines for how public service agencies develop APIs) into a technical standard. The generation process took approximately 2 hours once the data was prepared, producing a 14,657-word standard with 280 citations to source material.

What is an API? APIs (Application Programming Interfaces) enable cross-government data sharing in a secure and reliable way. Think of them like a waiter in a restaurant - you order from a menu, and the waiter brings your choices back in a predictable way. When APIs work well, people can build applications that make use of government data securely and reliably.

2. Identification Management Standards (Generation)

We consolidated 30 identification management documents into a single resource. These documents shape how people and organisations demonstrate who they are across the public service and private sector. The project transformed 9,374 content nodes into a coherent 7,140-line document while preserving all 109 core standards controls unchanged.

Key metrics:

  • Input: 30 documents
  • Output: 1 consolidated document
  • Active work time: ~11 hours across 3 days
  • Core controls preserved: 109/109 (100%)
  • DocRef citations: 415+

3. AI Guidance (Evaluation and Compliance)

We used a third MCP server containing government AI guidance in two ways:

Content Evaluation: During the identification management project, we used the guidance to validate AI-generated recommendations against government principles. This validated 7 of our 8 restructuring recommendations and confirmed alignment with all 6 core AI principles.

Process Compliance Analysis: In December 2025, we conducted a comprehensive compliance analysis evaluating how all the work documented in this Transparency Hub aligns with the NZ Public Service Generative AI Guidance. The analysis found strong overall compliance across all five OECD-aligned principles, with the Hub's approach to transparency and documentation exceeding typical requirements. Read the full compliance analysis.

What makes us think the AI-generated output is any good?

We're starting from a position of skepticism

Our default position when it comes to using AI to draft regulation is that the output will be of poor quality and unreliable. Writing regulation well is an expert task. Good regulation often uses less words rather than more. It requires:

  • awareness of context,
  • understanding of what's come before,
  • attention to matters of nuance,
  • and the exercise of expert judgement.

All of these things do not come by default from an AI system trained for general purposes on data from all over the world. From this starting position, it's up to people using AI to write regulation to persuade other people why what they're doing is useful.

The AI is not the point

When we say that we used AI, it's important to understand that the "AI" part of the system was really just one narrow slice. The systems and processes we adopted were actually about working with the text of the relevant regulations as data. This meant we could take transparent and explainable steps to transform that data to make it more useful and usable to the AI system.

The most important work was about constraining the available space for the AI model to operate. This meant we could target its focus, clarify its task, and break the overall process into steps that people can see and examine for themselves.

What was the process we followed?

Here are the key steps in the process, applicable to any regulatory document set:

  1. Convert to structured data: We turned regulations from text on a website to precise structured data using DocRef. Working with clean structured data meant we could analyse and transform it using software and track changes between versions.

  2. Understand the domain: We spent time understanding how the regulations were or weren't solving their purpose - reading the text and speaking with people they apply to. This gave us a conceptual understanding of the space the regulations were meant to fill.

  3. Add annotations and tags: We manually added notes and tags to key parts - noting mandatory requirements, best practices, expectations, and key definitions. These annotations were added to the dataset that shaped the model's comprehension.

  4. Model document structure: We tracked cross-references between paragraphs, hierarchies between sections and subsections, and links between separate parts of documents. This gave the model a way of understanding structure and relationships.

  5. Add semantic search: We added "vector embeddings" to individual document elements, enabling search by meaning rather than just keywords. This meant parts of documents that might otherwise be missed were more likely to be identified.

  6. Deploy as MCP server: We made the data and tools available to the AI as an "MCP server" - a standardized way for AI systems to access external data sources. The AI could then perform systematic research, save results, and generate outputs.

  7. Preserve citations throughout: DocRef enables precise citations to specific parts of documents (paragraphs, sub-paragraphs, table cells). These links travel with the data at every step. If you look at the outputs, you'll see DocRef links scattered throughout - click one to see exactly which source the AI relied upon.

  8. Evaluate against principles: Finally, we added an eighth step: using a separate MCP server containing government AI guidance to validate the process we followed against the public service generative AI guidance.

Metrics and evidence

API Standard Project

  • Duration: ~2 hours (generation phase)
  • Source material: 5,612 document nodes
  • Searches executed: 47 semantic searches
  • Output: 14,657 words, 280 citations
  • Peak context usage: 76% of 200,000 token limit

Identification Management Project

  • Duration: ~11 hours across 3 days
  • Source material: 9,374 document nodes across 30 documents
  • Output: 7,140-line consolidated document
  • Core controls: 109/109 preserved unchanged
  • Citations: 415+
  • Verification: 19/19 success criteria met (100%)

AI Guidance Compliance Analysis

  • Overall assessment: Strong compliance across all 5 OECD-aligned principles
  • Transparency & Explainability: Excellent - exceeds publication requirements
  • Accountability: Excellent - clear governance with documented human oversight
  • Source: 1,393 document nodes across 23 AI guidance documents
  • Full compliance analysis available

The opportunity going forward

How is this output better than one that is manually produced?

Drafting written materials for a regulatory context takes huge amounts of time and mental energy to:

  • collate relevant material,
  • analyse it comprehensively,
  • record conclusions and share them with others,
  • agree on an overall structure,
  • write the document,
  • receive feedback,
  • and incorporate edits into subsequent drafts.

The improvements we've added make all of those stages much more efficient. Specialists who might have spent time on manual labour can now devote their time to more efficient review of the material and output. The people in Government who have the requisite blend of technical, policy and written skills to produce such documents are extremely rare and can be much better deployed on higher value tasks.

We estimate that manually drafting the API Standard with the same granularity of citation would have taken multiple people at least months. The identification management consolidation - transforming 30 fragmented documents into one coherent resource - would likely never have been attempted at all given resource constraints.

Reusable infrastructure

The datasets and tooling created to enable this system are all reusable. The same system can power:

  • A search tool for finding relevant requirements
  • A publishing system for structured documents
  • A question-and-answer chatbot with pinpoint citations
  • AI-assisted drafting of new regulatory instruments
  • "Rules as code" compliance assets

What do you think?

All of this is only valuable if it produces a good quality output in a way that saves time and energy. The only way to assess output quality is through expert human review - so we need to hear your views. Here are some important things to remember:

  • We'd be surprised if any complex regulatory document is perfect the first time, but that's not the standard for success.
  • The standard for success is whether good quality work can be done faster, maximising the value of human experts who act as custodians of that process.
  • Another key factor is to consider what work might need to be done, but would never realistically have been done anyway because of time and financial constraints.

Next steps

We're now taking this work further as well as applying it to new regulatory domains.

  • We're consulting on the API Standard and then transitioning to the production of assets to support people to comply with it.
  • The identification management standards consolidation is ready for expert review and feedback.
  • We're exploring what other regulatory documents should be added to DocRef or used to supplement the core regulatory datasets.
  • We're looking for more opportunities to test what this system is capable of in both digital and non-digital policy contexts.

Contact and feedback

You can contact us by email on contact@syncopatelab.com.

If you would like to tell us about a project you're working on, you can access our project enquiry form here or send us an email.

Click here to sign up for our early testing programme or be notified when you can sign up for a DocRef account.

[^1]: "The government of New Zealand uses regulation to protect the community from harm and to improve the standard of living of its people. Regulation is about influencing people's behaviour to improve outcomes for all New Zealanders. This involves laws, rules and other ways to influence what people do. The rules, organisations and their practices – the whole regulatory system – work together to shape people's behaviour and interactions and improve the lives, work and businesses of all New Zealanders." Ministry for Regulation.