AI Governance for Small Businesses: Setting Rules for AI Tools at Work
A practical, five-step AI governance plan for small businesses: find the AI already in use, set data rules, write a one-page policy, and lock it down.
It usually comes up by accident. Someone in purchasing mentions they "had ChatGPT clean up" a supplier email, or a quality engineer pastes a customer drawing spec into a free AI tool to summarize it. Nobody meant any harm. But when I ask who approved that tool, what happens to the data, and whether it was a personal or company account, the room goes quiet. That moment is why AI governance for small businesses matters, and why it doesn't have to be a big, expensive project.
AI governance sounds like something only banks and tech giants need. In practice, for a 30-person shop or a 300-person manufacturer, it means a few clear rules: which AI tools are allowed, what information can go into them, who owns the accounts, and how you check that the rules are being followed.
My partner Mukesh has spent 25 years in cybersecurity, and I've spent 15 years running networks. We both see the same pattern: AI adoption is happening from the bottom up, whether leadership has a plan or not. This guide walks through the plan, step by step, in a way a busy IT manager can actually finish.
In this guide
- What AI governance means for a small business
- Step 1: Take inventory of the AI already in use
- Step 2: Sort your data into green, yellow and red
- Step 3: Write a one-page AI acceptable use policy
- Step 4: Lock down accounts and vet AI vendors
- Step 5: Train people and review every quarter
- Where AI rules are heading: laws, customers and insurers
What AI governance means for a small business
AI governance is simply the set of decisions and controls that keep AI use safe, legal and useful for your company. It is not a 60-page framework or a new committee. For most small businesses it fits on a few pages and takes a few weeks to set up.
Think of it the same way you think about email or file sharing. You already decide which systems people use, who gets access, and what can't leave the building. AI tools need the same treatment, with a few new wrinkles.
Why unmanaged AI is a real risk
The risks are practical, not science fiction:
- Data leaving your control. Free and consumer versions of AI tools may keep your prompts and, depending on the settings, use them to improve their models. Business tiers usually offer stronger terms, but you have to check.
- Shadow accounts. When employees sign up with personal emails, you can't see what was shared, and you can't shut it off when they leave.
- Wrong answers that look right. AI tools can produce confident, polished text that is simply incorrect. A bad number in a quote or a made-up spec in a customer reply is your company's mistake, not the tool's.
- Oversharing inside your own systems. AI assistants built into platforms like Microsoft 365 generally show users anything their permissions already allow. If your file permissions are messy, the assistant makes that mess much easier to find.
- Contract and compliance exposure. If you handle controlled defense data, health records or customer data covered by an NDA, pasting it into the wrong tool could break a promise you've made.
Step 1: Take inventory of the AI already in use
You can't govern what you can't see. Before writing any policy, find out what's already happening. In my experience, the answer is almost always "more than leadership thinks."
How to find AI use quickly
- Ask people directly. Send a short, no-blame survey: which AI tools do you use, for what tasks, and with which account? Make it clear nobody is in trouble.
- Check your firewall and DNS logs. Most business firewalls and DNS filters can report traffic to common AI services. Look for chat assistants, writing tools, transcription services and image generators.
- Review your SaaS admin consoles. Microsoft 365, Google Workspace, your CRM and your ERP may already have AI features switched on. Check what's enabled and for whom.
- Look at browser extensions. AI-powered extensions for grammar, summarizing and note-taking can read everything on a page. Your endpoint management tool can often list installed extensions.
- Ask about meeting bots. AI note-takers that join and record calls are easy to miss.
What to record
Keep a simple spreadsheet with one row per tool: name, what it's used for, who uses it, personal or company account, free or paid tier, and what kind of data goes into it. This becomes your AI register, and you'll update it every quarter.
Step 2: Sort your data into green, yellow and red
The most useful rule in any AI policy is about data, not tools. Employees don't need to memorize vendor terms. They need to know what they can and can't paste in.
A simple three-color system works well for small teams:
- Green (okay for approved AI tools): public information, marketing copy, general how-to questions, published product info, generic code snippets with no secrets.
- Yellow (approved business AI tools only, with care): internal procedures, non-sensitive internal emails, draft documents, anonymized data.
- Red (never in AI tools unless specifically approved for it): customer data, employee personal information, financial records, passwords and keys, health information, controlled unclassified information (CUI), export-controlled technical data, and anything covered by an NDA.
A note for defense suppliers
If you hold CUI or are working toward CMMC, treat AI tools like any other cloud service that might touch that data. A tool that hasn't been approved and documented as part of your CUI environment generally should not see CUI at all. If you're still sorting out which level applies to you, start with our guide to CMMC Level 1 vs. Level 2, and confirm the details with your assessor.
Step 3: Write a one-page AI acceptable use policy
Long policies don't get read. A one-page AI acceptable use policy that people actually understand beats a polished document nobody opens. You can attach it to your existing acceptable use or IT policy.
What to include
- Purpose. One sentence: we use AI to work faster, and we protect our customers' and company's information while doing it.
- Approved tools. List them by name, and say that anything not on the list needs IT approval first.
- Account rules. Company accounts only, created by IT, with multi-factor authentication. No personal accounts for work.
- Data rules. Your green, yellow and red categories, in plain words with examples.
- Human review. A person checks AI output before it goes to a customer, into a quote, into code that ships, or into any official record.
- Disclosure. When work is mostly AI-generated, say so where it matters, such as customer deliverables or anything with legal weight.
- No AI for certain decisions. Hiring, firing, pay and credit decisions should not be made by AI alone. Some state laws are moving in this direction, so it's a sensible line to draw now.
- How to request a new tool. Who to email and what information to include.
- Who owns this policy. Name a person, and set a review date.
Write it the way you'd explain it to a new hire on day one, and have everyone acknowledge it like your other IT policies.
Step 4: Lock down accounts and vet AI vendors
A policy without technical controls is just a suggestion. The good news is that most of these controls use tools you already have.
Technical controls to put in place
- Single sign-on and MFA. Connect approved AI tools to your Microsoft or Google identity so accounts are created and removed with the employee. Basic hygiene like MFA is the first item in our 10 steps to protect your company.
- Turn off training on your data. In business tiers, confirm in the admin settings that your prompts and files are not used to train models. Recheck after major updates.
- Fix file permissions before enabling built-in assistants. Before you roll out an assistant inside Microsoft 365 or Google Workspace, review who can see which SharePoint sites, Teams and shared drives. Clean up "Everyone" links and old guest access first.
- Block or warn on unapproved tools. Use your firewall, DNS filter or secure web gateway to block unapproved AI sites, or show a warning page that points to the approved option.
- Use data loss prevention if you have it. Many Microsoft 365 business plans include DLP features that can flag things like credit card or Social Security numbers. Start in "audit" mode so you learn before you block.
- Manage browser extensions. Allow only approved extensions on company devices.
Questions to ask any AI vendor
Before you approve a tool, or switch on a new AI feature in software you already pay for, get answers to these:
- Is our data used to train your models? Can we turn that off, and is it off by default on our plan?
- Where is our data stored, and how long is it kept?
- Do you support single sign-on and MFA?
- Can we export or delete our data when we leave?
- Do you have a recent independent security report, such as a SOC 2 Type II?
- Which third parties or sub-processors handle our data?
- Is this tool appropriate for regulated data like CUI or health information, and will you put that in writing?
If a vendor can't answer clearly, that's your answer. Choosing technology carefully is a theme we come back to often; our post on how to choose the right technology for your business covers the broader buying process.
Step 5: Train people and review every quarter
Most AI mistakes come from people who didn't know the rules, not people trying to break them. A short, practical training session goes a long way.
What a 30-minute training should cover
- Which tools are approved and how to get access.
- The green, yellow and red data rules, with real examples from your business.
- Why AI answers need to be checked, with one example of a confident but wrong answer.
- How AI is making phishing and voice scams more convincing, and how to verify unusual payment or password requests.
- Who to ask when they're unsure.
Your quarterly review checklist
Put a recurring 60-minute meeting on the calendar and work through this list:
- Update the AI register with new tools and features.
- Review firewall or DNS reports for unapproved AI sites.
- Remove accounts for people who left or changed roles.
- Check vendor settings for changes after updates.
- Review any incidents or near misses.
- Confirm the policy still matches how people actually work.
A 30-day plan to get started
If you want a starting point, here's a realistic schedule:
- Week 1: Send the survey and pull firewall or DNS reports. Build the AI register.
- Week 2: Define green, yellow and red data. Pick approved tools.
- Week 3: Write the one-page policy. Set up SSO, MFA and training opt-outs.
- Week 4: Run the training, collect acknowledgments, and schedule the first quarterly review.
Where AI rules are heading: laws, customers and insurers
The rules around AI are moving quickly, and they're not settled. Here's where things stand as of mid-November 2026, at a high level. This is not legal advice; please confirm anything that affects your business with your attorney.
State laws are uneven and changing
There is still no single, comprehensive federal AI law for private businesses, and federal efforts to limit state AI laws are an active and unresolved fight. States are moving on their own. Colorado, for example, replaced its 2024 AI law in May 2026 with a narrower law focused on transparency for automated decision tools, now scheduled to take effect January 1, 2027, with enforcement depending on rulemaking.
Here in Virginia, a broad high-risk AI bill (HB 2094) passed the General Assembly in 2025 but was vetoed, and as far as I can tell no comprehensive AI law is in force here today. Other bills are expected to come back in future sessions, so it's worth keeping an eye on. If you sell into or hire in other states, check those states too.
Customers and insurers are asking
The bigger near-term pressure often comes from customers and insurers, not regulators. Larger customers are adding AI questions to supplier security questionnaires. Some cyber insurance applications now ask about AI use and controls. Having a written policy and an AI register makes those questions much easier to answer honestly.
Frequently asked questions
Do small businesses really need an AI policy?
Yes, if anyone in the company uses AI tools, and in most companies someone does. The policy doesn't need to be long. A single page that covers approved tools, data rules and human review handles most of the risk.
Is it safe to use ChatGPT or other AI tools for work?
It can be, if you use a business account with the right settings and keep sensitive data out. The bigger risks come from free or personal accounts where you don't control the data terms or the account itself. Check the current terms of whatever plan you use, since they change.
What should employees never put into AI tools?
Customer data, employee personal information, passwords, financial records, health information, CUI and anything covered by an NDA. A simple rule of thumb: if you'd be uncomfortable seeing it in a news story, don't paste it into an AI tool that hasn't been specifically approved for it.
How do I find out which AI tools my employees are using?
Start with a no-blame survey, then check your firewall or DNS filter reports, your SaaS admin consoles and installed browser extensions. Together, those usually give you a clear picture within a week or two.
Are there AI laws in Virginia that apply to my business?
As of November 2026, Virginia does not appear to have a comprehensive AI law for private businesses in force; a broad bill was vetoed in 2025. However, existing privacy, employment and industry rules still apply when you use AI, and other states may have their own laws. Confirm your situation with your attorney.
Get a second set of eyes on your AI and security setup
AI governance is one piece of a bigger picture. If your file permissions, MFA, firewall rules and account cleanup are already in good shape, AI rules are easy to add. If they aren't, AI tends to expose the gaps.
If you'd like help, we offer a free cybersecurity assessment. We'll look at how your accounts, data and network are set up, flag where AI tools could create exposure, and give you a short, prioritized list of fixes. No pressure and no long sales pitch. Request your free assessment here.
Related reading
- Cybersecurity for Roanoke Small Businesses: 10 Steps to Protect Your Company
- CMMC Level 1 vs. Level 2: Which One Does Your Manufacturing Shop Need?
- IT Consulting: How to Choose the Right Technology for Your Business
Harsha is a co-founder of SoftwarePac Technologies in Roanoke, VA, and a network engineer with 15 years of experience.




