Practitioners building an AI governance function rarely begin with a clean slate and six months to plan. More often, they inherit AI sprawl, receive an audit request with a deadline, and work with headcount never sized for a new compliance area. This guide focuses on a function that can operate under those conditions, rather than an ideal program with two years of runway.
The order of work matters. Writing policy before identifying the systems being governed often produces a theoretical program. Starting with discovery gives the governance function a factual basis before policy documents are drafted.
Phase 1: Set Scope and Gain Executive Support
Before discovery or policy work, obtain a written charter from an executive sponsor. It should authorize the function's work. This is essential when discovery finds a business-unit tool without prior oversight or controls mapping shows that a vendor contract needs renegotiation.
The charter can fit on one page. It should name the function's owner, define which reviews and documentation requests it may make, specify escalation when a business unit does not cooperate, and state the target framework or certification goal. An executive-signed charter is more useful than a thirty-page policy with no owner or backing.
Choose the target framework at this stage. ISO 42001 certification, NIST AI RMF alignment, EU AI Act compliance, or a combination will shape later documentation. Designing for one framework and retrofitting another takes more work than setting the target at the outset.
Phase 2: Discover the Baseline Inventory
A defensible scope statement requires knowing what falls within scope. The baseline AI inventory supports risk assessment, controls mapping, evidence collection, and the scope statement required by ISO 42001 Clause 4.
Connect discovery to sources that record actual usage: identity provider authentication logs, SaaS catalog data, network telemetry, and API gateway activity. Surveys and self-reporting can supplement discovery but should not be the primary method. Signal-based scans run alongside self-reported inventories consistently find tools the self-reported lists omit.
Phase 2 should produce a dated inventory of active AI tools grouped as SaaS, API integration, internally developed model, or vendor-embedded feature. Each entry needs enough context for the initial Phase 3 classification. Accuracy matters more than polish. The aim is a truthful starting point.
Phase 3: Classify Risk and Set Controls Priorities
AI tools do not all create the same governance workload. An internal writing assistant belongs to a different risk category from a system used for employment screening, loan decisioning, or clinical decision support. Phase 3 classification determines which controls require immediate attention.
Assess each tool across three dimensions: processed data, including personal, sensitive business, or regulated data; decisions influenced, especially high-stakes outcomes; and supplier risk, including provider data handling and model training practices. A high score on any dimension moves the tool into the Phase 4 priority queue. Lower-risk tools can receive a lighter review.
For organizations with EU exposure, apply the EU AI Act risk tiers here. High-risk applications under Annex III require the Act's full documentation regime. Finding those tools before the pre-audit sprint costs less than identifying them then.
Phase 4: Map Priority Tools to Controls
For every Phase 3 priority tool, map applicable framework requirements, available evidence, and missing evidence. Under ISO 42001, this includes Clause 8 operational requirements and applicable Annex A controls. Under NIST AI RMF, map to relevant subcategories in Govern, Map, Measure, and Manage.
Mapping reveals documentation gaps and operational gaps. A documentation gap means the control exists but lacks a record. An operational gap means the control is absent and requires remediation or documented risk acceptance. Track both categories.
Most writing and collection work occurs in Phase 4: risk assessments, operating procedures, supplier assessments, training records, and incident response procedures. Capacity limits become clear here. A two-person security team cannot map controls for forty AI tools in one quarter. Prioritizing higher-risk tools and assigning a credible remediation schedule to the rest is more defensible than producing thin documentation for everything.
Phase 5: Create the Evidence Process
A document created once does not demonstrate an ongoing program. An ISO 42001 audit or EU AI Act regulatory review may request records showing continuous control operation, documented management reviews, and identification and treatment of nonconformities.
Phase 5 establishes the recurring processes behind those records. Include a management review calendar and decision template, a monitoring alert workflow that records new tools and controls gaps, a nonconformity log with root cause and corrective action steps, and an evidence assembly process that turns operational records into auditor-ready packages.
Stakeholder Support: Practical Lessons
The main obstacle to starting AI governance is usually organizational, not technical. Business units may not have requested oversight, procurement may view review as delay, and developers may see governance as bureaucracy. Each can halt progress.
A more workable approach presents governance as risk reduction rather than added process. If a vendor AI feature creates a data-handling issue and the business unit lacks review records, governance addresses that problem. If established evaluation criteria speed security review for a new purchase, procurement also benefits. Presenting governance as a compliance cost center instead turns each interaction into an overhead negotiation.
Organizational politics will remain. The sequence above, beginning with discovery, creates useful results sooner and gives the function credibility to expand. A team that can say "we found 23 AI tools you had no record of, here is the risk, and here is what we did" has more authority than one presenting a framework document never applied to a real organizational tool.