Practice Project 20

Responsible AI Audit

Audit predictive algorithms and language models for subgroup bias, fairness metrics, hallucination risks, and regulatory compliance postures.

Domain / Environment
Ethical AI / Conda VM
Difficulty
Intermediate (3/5)
Course Module
Responsible & Ethical AI
Deliverables
Audit script & Compliance report logs
1. Responsible AI Audit Workflow

The diagram below displays the Responsible AI audit pipeline. The model outputs are evaluated across different demographic groups to compute disparate impact ratios, while LLM outputs are checked for hallucinations and compared against GDPR regulations.

Predictions Group A (Protected) Group B (Reference) Selection rates Assess 2. Bias Metrics Disparate Impact Demographic Parity 3. LLM Audit Hallucination check Similarity match Truth extraction 4. GDPR Check Data Privacy Opt-out rights Audited Log
2. Part 1: Step-by-Step Action Items & Key Execution Steps
STEP 1

Activate Python Virtual Environment

Point the terminal execution environment to the course conda sandbox environment.

$ conda activate ds_ai_ml
This targets active python libraries to the isolated virtual sandbox.
STEP 2

Create Project Folders inside Linux VM

Create a dedicated folder for the project files inside your guest VM home folder directory.

$ mkdir -p ~/Projects/ethical_ai && cd ~/Projects/ethical_ai
This sets up the working directory layout for the Responsible AI code files.
STEP 3

Install dependencies via Pip

Install pandas and scikit-learn inside the active conda session.

$ pip install pandas scikit-learn numpy
This installs the tabular and mathematical libraries needed to run the bias audit.
STEP 4

Create audit script file in VS Code

Launch VS Code and create the Responsible AI audit script file.

Launch VS Code via terminal "code ." -> New File -> Type: ethical_audit.py -> Paste Python code -> Save file
This registers the demographic subgroup analysis, disparate impact calculator, and GDPR mapping logic in `ethical_audit.py`.
STEP 5

Run the compliance audit

Execute the script to audit the model and output bias metrics to the console.

$ python ethical_audit.py
This runs the script, prints disparate impact ratios, flags subgroup differences, and logs compliance checklists.
3. Ethical Audit Execution Flow

The flowchart below outlines the ethical audit execution flow. It details the steps from extracting protected subgroups and computing selection rates to calculating disparate impact ratios and reviewing GDPR compliance.

1. Group Splits Extract protected and reference subgroups df[df['Age'] < 30] 2. Compute Rates Calculate selection rates for groups selection_rate() 3. Impact Ratio Divide protected rate by reference rate ratio = rateA / rateB 4. 80% Rule Check Flag ratios below 0.80 threshold assert ratio >= 0.8 5. Log Report Output audit report to console file log print("Audited")
4. Part 2: Complete Deliverable Assets & Production Templates

To run the compliance audit, we need the Python script file. Below is a line-by-line explanation of the code, followed by the combined template.

Step-by-Step Code Construction

Lines 1 - 5

Import pandas and numpy

Include system packages, pandas dataframes, and numerical matrices in the script.

import pandas as pd import numpy as np
These imports pull standard pandas dataframes and numpy mathematical utilities.
Lines 6 - 20

Calculate Selection Rates

Group model predictions and calculate selection rates for both protected and reference subgroups.

def selection_rate(df, group_col, group_val, target_col): subset = df[df[group_col] == group_val] positives = subset[subset[target_col] == 1] return len(positives) / len(subset)
This filters predictions by subgroup and calculates the selection rate (the ratio of positive predictions to total samples).
Lines 21 - 32

Apply 80% Rule (Disparate Impact Ratio)

Divide the protected group's selection rate by the reference group's rate. Ratios below 0.80 flag potential bias issues.

ratio = selection_rate_protected / selection_rate_reference if ratio < 0.8: print("WARNING: Disparate Impact detected! Ratio:", ratio) else: print("PASSED: Disparate Impact check passed. Ratio:", ratio)
This divides the selection rates to compute the disparate impact ratio and checks it against the 80% threshold.

Production templates

1. Python script (Save as ~/Projects/ethical_ai/ethical_audit.py):

# ethical_audit.py - Run fairness audits and GDPR compliance checks import pandas as pd import numpy as np def calculate_selection_rate(df, group_col, group_val, pred_col): sub = df[df[group_col] == group_val] pos = sub[sub[pred_col] == 1] if len(sub) == 0: return 0.0 return len(pos) / len(sub) def main(): print("=== Part 1: Group Fairness & Subgroup Bias Audit ===") # Create mock prediction data (e.g. churn predictions) data = { "UserID": range(1, 11), "AgeGroup": ["Young", "Senior", "Young", "Senior", "Senior", "Young", "Senior", "Young", "Young", "Senior"], "Prediction": [1, 1, 0, 1, 1, 0, 1, 1, 0, 1] # 1=Churn risk flag } df = pd.DataFrame(data) # Define protected and reference groups protected_group = "Young" reference_group = "Senior" rate_prot = calculate_selection_rate(df, "AgeGroup", protected_group, "Prediction") rate_ref = calculate_selection_rate(df, "AgeGroup", reference_group, "Prediction") print(f" - Selection Rate (Protected - {protected_group}): {rate_prot:.4f}") print(f" - Selection Rate (Reference - {reference_group}): {rate_ref:.4f}") # Calculate Disparate Impact Ratio di_ratio = rate_prot / rate_ref if rate_ref != 0 else 0.0 print(f" - Disparate Impact Ratio: {di_ratio:.4f}") if di_ratio < 0.8: print("WARNING: The model violates the 80% Rule for disparate impact! (Potential bias)") else: print("PASSED: Disparate Impact check passed. (No significant bias detected)") print("\n=== Part 2: Language Model Hallucination Risk Assessment ===") # Evaluate generated answer against reference context reference_context = "PyTorch is developed by Meta's AI Research lab." generated_hallucination = "PyTorch was created by Google DeepMind in London." print(f" - Reference Context: \"{reference_context}\"") print(f" - LLM Generated Answer: \"{generated_hallucination}\"") print(" - Evaluation: Fails factual verification. Check references and add grounding rules.") print("\n=== Part 3: GDPR Compliance Posture Review ===") print("Checking compliance mapping:") gdpr_checklist = { "Article 5(1)(c) - Data Minimization": "PASSED - Only AgeGroup and UserID are retained.", "Article 15 - Right of Access": "PASSED - User history and records can be queried.", "Article 17 - Right to Erasure ('Right to be Forgotten')": "PASSED - Drop commands clear records from active tables." } for principle, status in gdpr_checklist.items(): print(f" - {principle:45} | {status}") print("\n=== Responsible AI Compliance Audit Completed! ===") if __name__ == "__main__": main()
5. Deliverables Summary

Verify that the following configurations and outputs exist inside your project workspace.

Created Files / Templates

  • ~/Projects/ethical_ai/ethical_audit.py - Compliance audit script file.

Verification Artifacts / Execution Proof

  • Selection rates calculated across subgroups.
  • Disparate impact ratios and 80% rule checks logged.
  • GDPR article mappings checked and verified.
6. Closing Explanation: Why We Did This & What It Accomplishes

Architectural Intent & Operational Impact

Why We Did This

What This Accomplishes

Data Science & AI/ML Training Program | Practice Project 20 | Responsible AI