Practice Project 1

Mathematical & Statistical Foundations for AI/ML Project

Compute descriptive statistics, execute IQR outlier detection algorithms, perform hypothesis testing, and implement a custom gradient descent optimizer loop on a retail dataset inside a VM.

Domain / Environment

Retail Analytics / Conda VM

Difficulty

Intermediate (2/5)

Course Module

Math & Stats for AI/ML

Deliverables

Jupyter Notebook / Analysis Report

1. System Architecture & Process Workflow

The diagram below displays the data science math and statistics pipeline. Input records are cleaned of statistical outliers using Interquartile Range filters. The cleaned datasets are passed to SciPy for hypothesis t-tests while a custom NumPy loop runs a gradient descent optimizer to minimize loss metrics.

2. Part 1: Step-by-Step Action Items & Key Execution Steps

STEP 1

Activate Python Virtual Environment

Point the terminal execution environment to the course conda sandbox environment.

$ conda activate ds_ai_ml

This action maps terminal interpreter paths to the course packages, making library packages importable.

STEP 2

Create Project Folders inside Linux VM

Create a dedicated folder for the project files inside your guest VM home folder directory.

$ mkdir -p ~/Projects/math_foundations && cd ~/Projects/math_foundations

This creates the project workspace folder and shifts the active directory context of the terminal into it.

STEP 3

Generate Mock Transaction Dataset

Create a CSV text database containing transactional logs, customer age records, and group tags.

$ nano transactions.csv

This opens nano editor. Paste the CSV data template from Part 2, press **Ctrl + O** and **Enter** to save, and **Ctrl + X** to exit.

STEP 4

Install SciPy Math Modules via Pip

Install SciPy scientific packages to enable hypothesis t-test calculations.

$ pip install scipy

This command downloads SciPy, which provides statistical distributions and calculations.

STEP 5

Open VS Code inside Project Folder

Launch Visual Studio Code inside your active project directory.

$ code .

This boots the editor and mounts the project directory in the left file explorer bar automatically.

STEP 6

Create verification script file in VS Code

Use VS Code GUI tools to create a new python file inside the project workspace.

Hover over "MATH_FOUNDATIONS" in VS Code left sidebar -> Click "New File" icon -> Type: verify_math.py -> Press Enter

This instantiates a new empty script document `verify_math.py` inside the active workspace directory.

STEP 7

Load Python Math Diagnostic Logic

Write the statistical functions and gradient loop code inside the empty python file.

Click on Code Editor window tab "verify_math.py" -> Copy code from Part 2 -> Paste it into the editor window -> Press Ctrl + S to save

This populates the file with script code and saves it to local disk directories.

STEP 8

Execute python script via terminal pane

Run the validation script using the python engine to verify statistical operations work.

$ python verify_math.py

This runs the calculations, outputting mean stats, IQR outliers, t-test p-values, gradient loss values, and saving a loss curve plot.

STEP 9

View Saved Graphics Output

Open and check the saved gradient descent loss chart using the default Linux desktop photo viewer.

$ xdg-open loss_minimization.png

This command loads the image viewer application on the VM desktop to show the generated plot chart.

3. Operational Pipeline Architecture

The flowchart below outlines the math analysis pipeline. It details the steps from raw database parsing and IQR outlier cleaning to calculating descriptive stats, running t-tests, executing the custom gradient loop, and saving output reports.

4. Part 2: Complete Deliverable Assets & Production Templates

To run math diagnostics, we need the raw database file and python logic script. Below is a step-by-step code breakdown, followed by the combined template files.

Step-by-Step Code Construction

Lines 1 - 5

Import Framework Modules

Include system packages, Pandas/NumPy, and SciPy stats modules in the code script.

import pandas as pd import numpy as np import matplotlib.pyplot as plt from scipy import stats

These imports check system paths and load mathematical computing packages, graphing tools, and statistics modules.

Lines 6 - 15

Compute Descriptive Statistics

Load the CSV dataset and print descriptive statistics like mean, median, and variance.

df = pd.read_csv("transactions.csv") print("Mean Spend:", df["Spend"].mean()) print("Median Spend:", df["Spend"].median()) print("Variance Spend:", df["Spend"].var()) print("Std Deviation Spend:", df["Spend"].std())

These statement loads the data table, calculates standard descriptive statistics, and prints them to the terminal.

Lines 16 - 25

Detect Outliers via IQR Algorithm

Implement the Interquartile Range algorithm to flag and filter database outlier records.

q1 = df["Spend"].quantile(0.25) q3 = df["Spend"].quantile(0.75) iqr = q3 - q1 lower_bound = q1 - 1.5 * iqr upper_bound = q3 + 1.5 * iqr outliers = df[(df["Spend"] < lower_bound) | (df["Spend"] > upper_bound)] print(f"Num Outliers: {len(outliers)}") df_cleaned = df[~df.index.isin(outliers.index)]

This calculates the lower and upper bounds of the data and filters out any records that fall outside this range.

Lines 26 - 32

Perform Hypothesis Test (t-test)

Run SciPy t-tests to evaluate differences between group mean values.

group_a = df_cleaned[df_cleaned["Group"] == "A"]["Spend"] group_b = df_cleaned[df_cleaned["Group"] == "B"]["Spend"] t_stat, p_val = stats.ttest_ind(group_a, group_b) print(f"T-statistic: {t_stat:.4f} | P-value: {p_val:.4f}")

This runs an independent two-sample t-test, calculating p-values to evaluate if the differences in spending between the two groups are statistically significant.

Lines 33 - 48

Run Gradient Descent Loop

Implement a custom gradient descent loop inside Python to minimize a mean squared error loss function.

X_mat = df_cleaned["Age"].values y_val = df_cleaned["Spend"].values w = 0.0 # Initial weight lr = 0.0001 # Learning rate losses = [] for epoch in range(1, 101): prediction = X_mat * w loss = np.mean((prediction - y_val) ** 2) gradient = 2 * np.mean(X_mat * (prediction - y_val)) w = w - lr * gradient losses.append(loss)

This uses a custom loop to calculate predicted values, compute gradients using partial derivatives, update model weights, and track loss reduction.

Production templates

1. Sample Dataset (Save as ~/Projects/math_foundations/transactions.csv):

Transaction_ID,Age,Spend,Group 1,25,120.50,A 2,34,350.00,B 3,45,210.20,A 4,22,89.90,A 5,50,450.00,B 6,29,150.00,A 7,38,380.00,B 8,65,1200.00,B 9,41,290.50,A 10,31,310.00,B 11,28,140.00,A 12,47,260.00,B 13,55,95.00,A 14,36,410.00,B 15,62,1300.00,A

2. Python script (Save as ~/Projects/math_foundations/verify_math.py):

# verify_math.py - Statistical analysis and gradient descent loop import pandas as pd import numpy as np import matplotlib.pyplot as plt from scipy import stats def main(): print("=== Step 1: Load and profile raw data ===") df = pd.read_csv("transactions.csv") print(f"Dataset Loaded. Total records: {len(df)}") # Descriptive Statistics print("\n[Descriptive Statistics - Column: Spend]") print(f" - Mean: {df['Spend'].mean():.2f}") print(f" - Median: {df['Spend'].median():.2f}") print(f" - Variance: {df['Spend'].var():.2f}") print(f" - Std Deviation: {df['Spend'].std():.2f}") # 2. Outlier Detection via IQR print("\n=== Step 2: Outlier Detection via IQR ===") q1 = df["Spend"].quantile(0.25) q3 = df["Spend"].quantile(0.75) iqr = q3 - q1 lower_bound = q1 - 1.5 * iqr upper_bound = q3 + 1.5 * iqr outliers = df[(df["Spend"] < lower_bound) | (df["Spend"] > upper_bound)] print(f"IQR Thresholds: [{lower_bound:.2f}, {upper_bound:.2f}]") print(f"Number of Outliers detected: {len(outliers)}") for idx, row in outliers.iterrows(): print(f" - ID {row['Transaction_ID']}: Spend={row['Spend']} (Outlier)") df_cleaned = df[~df.index.isin(outliers.index)].copy() print(f"Data cleaned. Remaining records: {len(df_cleaned)}") # 3. Hypothesis testing (Independent t-test) print("\n=== Step 3: Hypothesis Testing ===") group_a = df_cleaned[df_cleaned["Group"] == "A"]["Spend"] group_b = df_cleaned[df_cleaned["Group"] == "B"]["Spend"] t_stat, p_val = stats.ttest_ind(group_a, group_b) print(f"Independent T-Test results:") print(f" - T-statistic: {t_stat:.4f}") print(f" - P-value: {p_val:.4f}") if p_val < 0.05: print("Result: Statistically significant. Reject the null hypothesis.") else: print("Result: Not statistically significant. Fail to reject the null hypothesis.") # 4. Custom Gradient Descent Loop print("\n=== Step 4: Custom Gradient Descent Loop ===") # We model: Spend = w * Age X_mat = df_cleaned["Age"].values y_val = df_cleaned["Spend"].values w = 0.0 lr = 0.0001 losses = [] print("Starting weights optimization...") for epoch in range(1, 101): prediction = X_mat * w loss = np.mean((prediction - y_val) ** 2) gradient = 2 * np.mean(X_mat * (prediction - y_val)) w = w - lr * gradient losses.append(loss) if epoch % 20 == 0: print(f" - Epoch {epoch}: Loss={loss:.4f}, weight={w:.4f}") print(f"Optimized weight (w): {w:.4f}") # 5. Graph rendering and save plt.figure(figsize=(8, 4)) plt.plot(range(1, 101), losses, color="#c084fc", linewidth=2) plt.title("Gradient Descent Loss Convergence Curve", color="#f8fafc") plt.xlabel("Epoch", color="#94a3b8") plt.ylabel("Mean Squared Error Loss", color="#94a3b8") plt.grid(True, linestyle="--", alpha=0.5, color="#334155") plot_filename = "loss_minimization.png" plt.savefig(plot_filename, facecolor="#0f172a", edgecolor="none") print(f"\nConvergence plot chart saved: {plot_filename}") print("=== Math Foundations Project Successfully Complete! ===") if __name__ == "__main__": main()

5. Deliverables Summary

Verify that the following configurations and outputs exist inside your project workspace.

Created Files / Templates

~/Projects/math_foundations/transactions.csv - Raw transaction database file.
~/Projects/math_foundations/verify_math.py - Verification python script file.
~/Projects/math_foundations/loss_minimization.png - Saved convergence curve image.

Verification Artifacts / Execution Proof

Console outputs showing descriptive stats calculations.
Outlier logging statements matching ID 8 and ID 15 in logs.
T-test p-value showing null hypothesis rejection details.
Gradient descent logs displaying decreasing loss values.

6. Closing Explanation: Why We Did This & What It Accomplishes

Architectural Intent & Operational Impact

Why We Did This

Outliers can skew statistical metrics, so removing them using IQR ensures analysis accuracy.
Hypothesis testing evaluates differences between dataset groups, helping validate marketing metrics.
Writing a custom gradient descent loop builds intuition for optimization algorithms.

What This Accomplishes

Cleans the dataset of outliers and profiles columns, preparing data for modeling.
Runs t-tests to evaluate differences between A/B groups.
Optimizes model weights to minimize prediction error.