INVOKE THIS SKILL when auditing an AI agent or LLM app for regulatory compliance. Covers EU AI Act, GPAI Code of Practice, GDPR, NIST AI RMF, Colorado AI Act, HIPAA, and ISO 42001. Scans the codebase for compliance gaps, cross-references Arize instrumentation for audit trail coverage, and produces a

Arize Compliance Audit Skill Use this skill when the user wants to audit their AI agent or LLM application for regulatory compliance . The skill scans the codebase for compliance gaps, cross references Arize instrumentation for audit trail coverage, and produces a tailored checklist with optional remediation. Triggers: "audit my app for compliance", "EU AI Act requirements", "NIST AI RMF checklist", "GDPR for AI", "is my AI app compliant", "compliance checklist", "regulatory audit", "ISO 42001", "AI management system", "AIMS certification". Disclaimer Before doing anything else, present this disclaimer verbatim to the user: ⚠️ Legal disclaimer This audit is for guidance only and does not constitute legal advice or a complete compliance assessment. It identifies common technical patterns and gaps based on publicly available regulatory frameworks, but cannot assess your organisation's specific legal obligations, contractual commitments, data processing agreements, or operational processes. Do not rely on this output as a substitute for qualified legal counsel. Regulatory compliance is a complex, jurisdiction specific, and fact dependent determination. Always engage a qualified attorney or compliance specialist for binding assessments. Core principles Prefer inspection over mutation — understand the codebase before suggesting changes. Be practical, not legal — produce developer actionable items, not legal opinions. Tailor to jurisdiction and use case — a chatbot has different obligations than a hiring tool. Do not dump the entire regulatory framework. Cross reference instrumentation — compliance requires audit trails; check whether Arize tracing captures what regulators expect. Offer remediation, always confirm — after presenting the checklist, offer to implement specific fixes, but never modify code without explicit user confirmation. Keep output concise and production focused — do not generate extra documentation or summary files unless requested. Never embed literal credential values — always reference environment variables. Never ask the user to paste secrets into the coding agent chat. Phase 0: Framework selection and use case Before scanning code, determine which compliance frameworks apply. Step 1 — Framework selection Use the AskUserQuestion tool to ask the user which frameworks apply. Do not infer or auto select — always ask explicitly. Ask: Based on the selection: 1 selected — EU AI Act, GPAI Code of Practice, GDPR apply. See [references/eu ai act gpai.md](references/eu ai act gpai.md). 2 selected — NIST AI RMF, Colorado AI Act, NYC LL144, HIPAA may apply. See [references/us ai compliance.md](references/us ai compliance.md). 3 selected — ISO 42001 AIMS controls apply. See [references/iso 42001.md](references/iso 42001.md). Note: ISO 42001 is an organisational management system — the audit will cover technically auditable controls only; purely organisational clauses (leadership review, internal audits) are flagged separately. Multiple selected — all selected frameworks apply; the audit covers the union of requirements, with cross references where frameworks overlap. Step 2 — Determine use case category Use the AskUserQuestion tool to ask: What does your AI application do? General chatbot / assistant — Limited risk (EU), general obligations (US) Hiring / HR — High risk (EU Art. 6, Annex III); Colorado AI Act applies; NYC LL144 applies if NYC Healthcare — High risk (EU); HIPAA applies if processing PHI Credit / financial — High risk (EU); Colorado AI Act applies Education — High risk (EU) Content generation — Limited risk (EU Art. 50 transparency); general obligations (US) GPAI model provider — GPAI Code of Practice applies (EU) Step 3 — Determine risk tier Based on the use case and selected frameworks: EU selected : Classify as Unacceptable / High / Limited / Minimal per [references/eu ai act gpai.md](references/eu ai act gpai.md) US selected : Classify as High risk (consequential decisions per Colorado AI Act) or General ISO 42001 selected : Risk tier is not a formal classification in ISO 42001, but note whether the system is high stakes (which elevates the priority of impact assessment and bias controls) Phase 0 output Present a brief summary: Then proceed directly to Phase 1. Phase 1: Codebase audit (read only) Do not write any code or create any files during this phase. Systematically scan the codebase for evidence of compliance and gaps across seven domains. For each domain, run the listed searches and record findings. A. Transparency and disclosure What to look for: User facing strings disclosing AI involvement: search for terms like AI , artificial intelligence , automated , bot , machine learning , generated by , powered by in UI templates, API responses, and user facing code Content labelling: markers on AI generated output (text, images, audio) Terms of service, privacy policy references in the codebase Signals of concern: Absence of any AI disclosure in user facing code, especially if the application generates content or makes recommendations. B. Data protection and privacy What to look for: PII field names in code: email , phone , ssn , social security , date of birth , address , name in prompts, context, or retrieved documents PII in trace span attributes: check if input.value or output.value could contain personal data sent to Arize without redaction Consent mechanisms: consent , opt in , opt out , gdpr , ccpa references DPIA or privacy assessment references Data retention and deletion handlers Data subject rights: right to access , right to erasure , data subject request , data protection officer C. Security What to look for: Prompt injection defences: input validation, guardrail libraries ( guardrails ai , nemo guardrails , rebuff , lakera ), content filtering, system prompt protection Data loss prevention: output scanning before returning to users, sensitive data detection Tool/function calling controls: permission boundaries, allowlists, sandboxing for tool execution Rate limiting and authentication on AI endpoints Hardcoded secrets: api key , secret , password , token literals in source files (not env var references) D. Testing and evaluation What to look for: Bias and fairness testing: references to demographic parity, impact ratios, fairness metrics Red teaming or adversarial test suites: prompt injection tests, jailbreak tests Evaluation frameworks: Arize evaluators, custom eval scripts, pytest based evals, experiment infrastructure A/B testing or model comparison infrastructure E. Documentation What to look for: Model cards: MODEL CARD.md , model card.json , model card.yaml , or similar System architecture documentation Change logs or version tracking for prompts and model updates Incident response documentation F. Monitoring and observability What to look for: Arize tracing setup: arize otel , register() , TracerProvider , opentelemetry , openinference imports If tracing exists, check coverage: All LLM calls traced (not just some) Session IDs for conversation continuity User IDs for data subject request support Error tracking and exception spans Alerting and drift detection configuration Trace retention configuration G. Vendor management What to look for: Third party AI API usage: OpenAI, Anthropic, Google, Azure, Bedrock, Cohere imports or client instantiation Model versioning: are specific model versions pinned (e.g., gpt 4 0613 ) or using latest / unversioned identifiers Fallback and failover logic between providers Phase 1 output Present a two part report: Part 1 — Summary table Domain Evidence found Gaps identified Rating A. Transparency {findings} {gaps} Compliant / Partial / Non compliant / N/A B. Data protection {findings} {gaps} ... C. Security {findings} {gaps} ... D. Testing {findings} {gaps} ... E. Documentation {findings} {gaps} ... F. Monitoring {findings} {gaps} ... G. Vendor management {findings} {gaps} ... Part 2 — Gap detail (required for every Non compliant or Partial rating) For each domain rated Non compliant or Partial, write a dedicated subsection that includes: 1. The exact code path — file path(s), line number(s), and the relevant code snippet showing where the gap exists. Do not paraphrase; quote the actual code. 2. Why it matters in this specific app — explain the concrete risk in the context of this codebase (e.g. which tools could be abused, which data flows are exposed, what an attacker or regulator would find). 3. What is missing — a precise description of the control or code that should exist but does not (e.g. "a span attribute processor that hashes user email before the OTLP exporter fires", not just "add PII redaction"). Minimum one subsection per Non compliant/Partial domain. Do not omit this section — it is the primary value of the audit for engineering teams. Then proceed directly to Phase 2. Phase 2: Compliance checklist Using the Phase 1 findings and the template in [references/compliance checklist template.md](references/compliance checklist template.md), generate a tailored compliance checklist . Rules for checklist generation 1. Only include relevant sections. If the user is US only, skip GDPR specific items. If not healthcare, skip HIPAA. If not hiring in NYC, skip LL144. 2. Mark items from Phase 1. Items where evidence was found: mark as Compliant . Items with gaps: mark as Non compliant with a concrete remediation suggestion. 3. Prioritise correctly. Critical = enforcement risk or system prohibition. High = required by regulation. Medium = recommended by framework. Low = best practice. 4. Be specific in remediation. Instead of "implement input validation", say "add a guardrail library like guardrails ai to validate LLM inputs and outputs against your content policy". 5. Include the instrumentation cross reference table from the template. If Arize tracing is not set up, flag this as a Critical gap — audit trails are required by EU Art. 12 and NIST MAN 2.1. Final report Present a single consolidated report with four sections: Section 1 — Audit scope (Phase 0 summary) Frameworks selected, use case, risk tier, applicable regulations Section 2 — Codebase findings (Phase 1 summary table) The domain table (A–G) with evidence, gaps, and ratings Section 3 — Gap detail (Phase 1 expanded) One subsection per Non compliant or Partial domain, each containing: exact file paths and line numbers, quoted code snippets, app specific risk explanation, and a precise description of what is missing. This section is mandatory — never omit it. Section 4 — Compliance checklist (Phase 2) The tailored checklist with status and remediation suggestions, instrumentation cross reference table, priority summary, and recommended next steps When the user asks for a report file, write a single markdown file to /tmp/<app name compliance audit <YYYY MM DD .md containing all four sections. After presenting the report, offer Phase 3 remediation. Phase 3: Remediation (optional) After presenting the checklist, offer to implement specific fixes. Always use the AskUserQuestion tool to confirm before making any changes. Remediation categories Add dependencies — offer to install: Guardrail libraries for input/output validation (e.g., guardrails ai , nemo guardrails ) PII detection/redaction packages (e.g., presidio analyzer , scrubadub ) Content safety classifiers Insert code — offer to add: AI disclosure strings in user facing output (templates, API responses) PII redaction filters on span attributes before export to Arize

arize-compliance-audit