Practice Project 7

Supervised Learning Model Comparison Project

Train and evaluate multiple regression and classification algorithms on a bank loan dataset, run hyperparameter tuning via GridSearchCV, and compare model metrics.

Domain / Environment

Retail Banking / Conda VM

Difficulty

Intermediate (3/5)

Course Module

Supervised Machine Learning

Deliverables

Evaluation Model logs & Comparison metrics report

1. System Architecture & Process Workflow

The diagram below displays the supervised learning workflow. The bank loan dataset is split into training and testing sets. Numerical columns are scaled and categorical features encoded. Data is passed to a classification pipeline (Logistic Regression, Random Forest, SVC, and KNN) and a regression pipeline, with grid search tuning active.

2. Part 1: Step-by-Step Action Items & Key Execution Steps

STEP 1

Activate Python Virtual Environment

Point the terminal execution environment to the course conda sandbox environment.

$ conda activate ds_ai_ml

This action maps the terminal binary packages to the isolated conda environment paths.

STEP 2

Create Project Folders inside Linux VM

Create a dedicated folder for the project files inside your guest VM home folder directory.

$ mkdir -p ~/Projects/model_comparison && cd ~/Projects/model_comparison

This compiles project files in one isolated directory structure.

STEP 3

Save Credit Risk Loan Dataset

Create a CSV dataset containing bank credit records, customer age, income, debt, and credit scores.

$ nano loan_records.csv

This opens nano editor. Paste the CSV data template from Part 2, press **Ctrl + O** and **Enter** to save, and **Ctrl + X** to exit.

STEP 4

Install Machine Learning Packages via Pip

Install Scikit-Learn libraries inside the active conda session.

$ pip install scikit-learn pandas numpy

This command downloads scikit-learn frameworks, loading linear regression models, decision trees, ensemble frameworks, and hyperparameter tuning grid-search modules.

STEP 5

Create classifier script file in VS Code

Launch VS Code and create a new script file inside the project workspace folder.

Launch VS Code via terminal "code ." -> Right-click in explorer tree -> click New File -> Type: model_comparison.py -> Press Enter

This registers an empty file `model_comparison.py` inside the active directory editor workspace.

STEP 6

Load Python modeling comparison logic

Paste the training, tuning, and evaluation code into the newly created python script file.

Click model_comparison.py -> Paste python code from Part 2 below -> Save file via Ctrl + S

This writes modeling and metrics calculations to the script file.

STEP 7

Execute model comparison script

Run the validation script using the python engine to train and compare the models.

$ python model_comparison.py

This trains 4 classifiers and 2 regressors, runs GridSearchCV to tune hyperparameters, and prints comparative evaluation metrics to the console.

3. Operational Pipeline Architecture

The flowchart below outlines the supervised modeling pipeline. It details dataset loading, splitting, running classification and regression pipelines, tuning parameters via GridSearchCV, and outputting benchmark reports.

4. Part 2: Complete Deliverable Assets & Production Templates

To run the model comparison, we need the raw loan CSV dataset and the Python script. Below is a line-by-line explanation of the code, followed by the combined template files.

Step-by-Step Code Construction

Lines 1 - 7

Import Modeling Packages

Include system OS, Pandas/NumPy, classifier models, and metrics functions in the script.

import pandas as pd import numpy as np from sklearn.model_selection import train_test_split, GridSearchCV from sklearn.linear_model import LogisticRegression, LinearRegression from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor from sklearn.metrics import accuracy_score, f1_score, r2_score, mean_squared_error

These imports pull scikit-learn models, validation grids, and performance metric evaluators.

Lines 8 - 18

Load and Split Datasets

Read the CSV file and partition records into feature matrices and target vectors.

df = pd.read_csv("loan_records.csv") X = df[["Credit_Score", "Income", "Debt_To_Income"]] y_class = df["Approved"] y_reg = df["Interest_Rate"] X_train, X_test, y_train_class, y_test_class = train_test_split(X, y_class, test_size=0.2, random_state=42) _, _, y_train_reg, y_test_reg = train_test_split(X, y_reg, test_size=0.2, random_state=42)

This parses the file and splits records into training and testing subsets for classification and regression tasks.

Lines 19 - 30

Train and Evaluate Classifiers

Fit Logistic Regression and Random Forest models, and compare accuracy and F1 scores.

# Logistic Regression lr_model = LogisticRegression() lr_model.fit(X_train, y_train_class) lr_preds = lr_model.predict(X_test) print("Logistic Reg Accuracy:", accuracy_score(y_test_class, lr_preds)) # Random Forest rf_model = RandomForestClassifier(random_state=42) rf_model.fit(X_train, y_train_class) rf_preds = rf_model.predict(X_test) print("Random Forest F1-score:", f1_score(y_test_class, rf_preds))

This trains the models, runs predictions on the test set, and calculates performance metrics.

Lines 31 - 42

Run GridSearchCV Hyperparameter Tuning

Define hyperparameter search spaces and run grid search with cross-validation on the Random Forest model.

param_grid = {'n_estimators': [10, 50, 100], 'max_depth': [3, 5, None]} grid_search = GridSearchCV(RandomForestClassifier(random_state=42), param_grid, cv=3, scoring='f1') grid_search.fit(X_train, y_train_class) print("Best parameters found:", grid_search.best_params_) tuned_preds = grid_search.predict(X_test) print("Tuned Model F1-score:", f1_score(y_test_class, tuned_preds))

This searches a grid of parameters, fits models with 3-fold cross validation, and prints the best parameter values.

Production templates

1. Sample Dataset (Save as ~/Projects/model_comparison/loan_records.csv):

Credit_Score,Income,Debt_To_Income,Approved,Interest_Rate 720,85000,0.25,1,5.50 610,45000,0.40,0,12.00 790,120000,0.15,1,4.25 580,35000,0.45,0,15.00 680,65000,0.30,1,7.25 740,95000,0.20,1,5.00 630,50000,0.35,0,10.50 710,75000,0.28,1,6.00 590,40000,0.50,0,14.50 820,140000,0.10,1,3.75 660,58000,0.33,1,8.50 690,78000,0.22,1,6.50 750,105000,0.18,1,4.75 600,42000,0.38,0,11.50 670,62000,0.29,1,7.75 730,90000,0.24,1,5.25 620,48000,0.42,0,12.50 800,130000,0.12,1,4.00 640,52000,0.36,0,9.75 700,80000,0.26,1,5.85

2. Python script (Save as ~/Projects/model_comparison/model_comparison.py):

# model_comparison.py - Train, tune, and evaluate models import pandas as pd import numpy as np from sklearn.model_selection import train_test_split, GridSearchCV from sklearn.linear_model import LogisticRegression, LinearRegression from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor from sklearn.svm import SVC from sklearn.neighbors import KNeighborsClassifier from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, r2_score, mean_squared_error def main(): print("=== Loading and Partitioning Bank Loan Dataset ===") df = pd.read_csv("loan_records.csv") X = df[["Credit_Score", "Income", "Debt_To_Income"]] y_class = df["Approved"] y_reg = df["Interest_Rate"] # Classification split X_train, X_test, y_train_class, y_test_class = train_test_split(X, y_class, test_size=0.2, random_state=42) # Regression split _, _, y_train_reg, y_test_reg = train_test_split(X, y_reg, test_size=0.2, random_state=42) print(f"Train set: {len(X_train)} samples, Test set: {len(X_test)} samples") print("\n=== Part 1: Classification Models Comparisons ===") classifiers = { "Logistic Regression": LogisticRegression(), "Decision Tree": DecisionTreeClassifier(random_state=42), "Support Vector Machine": SVC(probability=True, random_state=42), "K-Nearest Neighbors": KNeighborsClassifier(n_neighbors=3) } for name, clf in classifiers.items(): clf.fit(X_train, y_train_class) preds = clf.predict(X_test) acc = accuracy_score(y_test_class, preds) f1 = f1_score(y_test_class, preds) print(f" - {name:25} | Accuracy: {acc:.4f} | F1-Score: {f1:.4f}") print("\n=== Part 2: Hyperparameter Tuning via GridSearchCV ===") rf_param_grid = { 'n_estimators': [10, 50, 100], 'max_depth': [3, 5, None] } print("Tuning Random Forest Classifier parameters...") grid_search = GridSearchCV(RandomForestClassifier(random_state=42), rf_param_grid, cv=3, scoring='f1') grid_search.fit(X_train, y_train_class) print(f" - Best Parameter values: {grid_search.best_params_}") tuned_model = grid_search.best_estimator_ tuned_preds = tuned_model.predict(X_test) print(f" - Tuned RF F1-Score: {f1_score(y_test_class, tuned_preds):.4f}") print("\n=== Part 3: Regression Models Comparisons ===") regressors = { "Linear Regression": LinearRegression(), "Decision Tree Regressor": DecisionTreeRegressor(random_state=42) } for name, reg in regressors.items(): reg.fit(X_train, y_train_reg) preds = reg.predict(X_test) r2 = r2_score(y_test_reg, preds) rmse = np.sqrt(mean_squared_error(y_test_reg, preds)) print(f" - {name:25} | R²: {r2:.4f} | RMSE: {rmse:.4f}") print("\n=== Supervised Learning Project Successfully Complete! ===") if __name__ == "__main__": main()

5. Deliverables Summary

Verify that the following configurations and outputs exist inside your project workspace.

Created Files / Templates

~/Projects/model_comparison/loan_records.csv - Credit risk dataset.
~/Projects/model_comparison/model_comparison.py - Model evaluation script file.

Verification Artifacts / Execution Proof

Accuracy and F1 score print logs for the 4 classifiers.
GridSearch results showing the optimal hyperparameter values.
R² and RMSE regression metrics printed to the console window.

6. Closing Explanation: Why We Did This & What It Accomplishes

Architectural Intent & Operational Impact

Why We Did This

Different algorithms perform best on different data shapes, so comparing multiple models helps ensure good selection.
Tuning hyperparameters via cross validation prevents overfitting and optimizes generalization.
Evaluating regression models with R² and classification with F-1 scores provides balanced performance measures.

What This Accomplishes

Splits credit risk data into training and test datasets.
Trains 4 classifiers and 2 regressors to baseline performance.
Tunes model hyperparameters and selects the best model.