Practice Project 14

Applied Python for AI/ML Mini-Project

Build an end-to-end data preprocessing pipeline using scikit-learn, transform tabular columns, and export a serialized pipeline file to bridge into the deep learning coursework.

Domain / Environment

Tabular Data / Conda VM

Difficulty

Beginner-Friendly (2/5)

Course Module

Python for AI & ML

Deliverables

Data Preprocessing script & Serialized pipeline (.joblib)

1. Preprocessing Pipeline Architecture

The diagram below displays the tabular data preprocessing pipeline. Raw columns are passed to a scikit-learn `ColumnTransformer`. Numeric features go through an imputer and scaler, while categorical features go through an encoder, producing a model-ready feature matrix.

2. Part 1: Step-by-Step Action Items & Key Execution Steps

STEP 1

Activate Python Virtual Environment

Point the terminal execution environment to the course conda sandbox environment.

$ conda activate ds_ai_ml

This targets active python libraries to the isolated virtual sandbox.

STEP 2

Create Project Folders inside Linux VM

Create a dedicated folder for the project files inside your guest VM home folder directory.

$ mkdir -p ~/Projects/python_preproc && cd ~/Projects/python_preproc

This sets up the working directory layout for the preprocessing code.

STEP 3

Save Raw Tabular Dataset

Create a CSV dataset containing customer profiles with numerical and categorical values.

$ nano user_profiles.csv

This opens nano editor. Paste the CSV data template from Part 2, press **Ctrl + O** and **Enter** to save, and **Ctrl + X** to exit.

STEP 4

Install scikit-learn via Pip

Install the required preprocessing packages inside the active conda session.

$ pip install scikit-learn pandas numpy joblib

This command downloads scikit-learn, which provides preprocessing transformers and pipeline objects.

STEP 5

Create pipeline script file in VS Code

Launch VS Code and create the preprocessing pipeline script file.

Launch VS Code via terminal "code ." -> New File -> Type: pipeline_builder.py -> Paste Python code -> Save file

This registers the preprocessing logic in `pipeline_builder.py` to transform data columns.

STEP 6

Execute and Verify the Preprocessing Pipeline

Run the script to preprocess the dataset and serialize the pipeline object.

$ python pipeline_builder.py

This runs the script, prints transformed values, and saves the fitted pipeline object to disk.

3. Preprocessing Execution Flow

The flowchart below outlines the data preprocessing execution flow. It details the steps from raw data ingestion and column partitioning to pipeline fits and serializing pipeline files.

4. Part 2: Complete Deliverable Assets & Production Templates

To run the preprocessing pipeline, we need the user profiles CSV dataset and the Python analysis script. Below is a line-by-line explanation of the code, followed by the combined template files.

Step-by-Step Code Construction

Lines 1 - 7

Import tabular and preprocessing modules

Include system packages, Pandas/NumPy, scaling, encoding, and pipeline modules in the script.

import pandas as pd import numpy as np from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler, OneHotEncoder from sklearn.impute import SimpleImputer from sklearn.compose import ColumnTransformer from sklearn.pipeline import Pipeline import joblib

These imports pull standard Pandas arrays, scikit-learn transformers, pipeline wrappers, and joblib serialization.

Lines 8 - 25

Define Preprocessing Transformers

Construct sub-pipelines for numerical scaling and categorical encoding, and merge them using ColumnTransformer.

num_pipeline = Pipeline([ ('imputer', SimpleImputer(strategy='median')), ('scaler', StandardScaler()) ]) cat_pipeline = Pipeline([ ('imputer', SimpleImputer(strategy='most_frequent')), ('encoder', OneHotEncoder(handle_unknown='ignore')) ]) preprocessor = ColumnTransformer([ ('num', num_pipeline, ['Age', 'Salary']), ('cat', cat_pipeline, ['City']) ])

This sets up transformers to impute missing numeric values and scale them, and encode categorical columns.

Lines 26 - 32

Fit the Preprocessor and Transform

Load the CSV dataset and run the preprocessor to transform features.

df = pd.read_csv("user_profiles.csv") X = df[['Age', 'Salary', 'City']] y = df['Target'] X_preprocessed = preprocessor.fit_transform(X)

This fits the preprocessing pipelines on features, standardizing numeric fields and encoding categorical columns.

Lines 33 - 42

Split and Serialize the Pipeline

Partition the preprocessed dataset into train, validation, and test splits, and save the preprocessor object to disk.

X_train, X_temp, y_train, y_temp = train_test_split(X_preprocessed, y, test_size=0.3, random_state=42) X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42) joblib.dump(preprocessor, "preprocessor.joblib")

This partitions the transformed feature matrix (70% train, 15% val, 15% test) and saves the preprocessor object to `preprocessor.joblib`.

Production templates

1. Sample Dataset (Save as ~/Projects/python_preproc/user_profiles.csv):

Age,Salary,City,Target 25,50000.00,New York,0 30,75000.00,London,1 ,60000.00,Paris,0 35,80000.00,New York,1 40,,London,0 45,110000.00,Paris,1 22,45000.00,New York,0 28,70000.00,London,1 32,65000.00,,0 38,90000.00,Paris,1

2. Python script (Save as ~/Projects/python_preproc/pipeline_builder.py):

# pipeline_builder.py - Data Preprocessing Pipeline import pandas as pd import numpy as np from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler, OneHotEncoder from sklearn.impute import SimpleImputer from sklearn.compose import ColumnTransformer from sklearn.pipeline import Pipeline import joblib def main(): print("=== Loading User Profiles Tabular Dataset ===") df = pd.read_csv("user_profiles.csv") print("Initial shape:", df.shape) print("\nMissing values check:") print(df.isnull().sum()) X = df[['Age', 'Salary', 'City']] y = df['Target'] # 1. Define sub-pipelines for columns print("\n=== Constructing Preprocessing Pipeline ===") num_pipeline = Pipeline([ ('imputer', SimpleImputer(strategy='median')), ('scaler', StandardScaler()) ]) cat_pipeline = Pipeline([ ('imputer', SimpleImputer(strategy='most_frequent')), ('encoder', OneHotEncoder(handle_unknown='ignore', sparse_output=False)) ]) # 2. Merge pipelines using ColumnTransformer preprocessor = ColumnTransformer([ ('num', num_pipeline, ['Age', 'Salary']), ('cat', cat_pipeline, ['City']) ]) # 3. Fit preprocessor and transform features X_preprocessed = preprocessor.fit_transform(X) print("Transformed features matrix shape:", X_preprocessed.shape) print("Preprocessing complete. Sample row:", X_preprocessed[0]) # 4. Split dataset (70% train, 15% val, 15% test) print("\n=== Splitting Preprocessed Data ===") X_train, X_temp, y_train, y_temp = train_test_split(X_preprocessed, y, test_size=0.3, random_state=42) X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42) print(f" - Training Split: {X_train.shape[0]} rows") print(f" - Validation Split: {X_val.shape[0]} rows") print(f" - Testing Split: {X_test.shape[0]} rows") # 5. Serialize pipeline object pipeline_filename = "preprocessor.joblib" joblib.dump(preprocessor, pipeline_filename) print(f"\nPreprocessor pipeline serialized and saved to: {pipeline_filename}") print("=== Preprocessing Pipeline successfully built! ===") if __name__ == "__main__": main()

5. Deliverables Summary

Verify that the following configurations and outputs exist inside your project workspace.

Created Files / Templates

~/Projects/python_preproc/user_profiles.csv - Raw user profile dataset.
~/Projects/python_preproc/pipeline_builder.py - Preprocessing builder script.
~/Projects/python_preproc/preprocessor.joblib - Serialized preprocessing model.

Verification Artifacts / Execution Proof

Correct replacement of missing fields (using medians for numerical columns).
OneHotEncoding categorical outputs printed to the console.
Correct data splitting (70% train, 15% val, 15% test) logged.

6. Closing Explanation: Why We Did This & What It Accomplishes

Architectural Intent & Operational Impact

Why We Did This

Scaling features prevents high-magnitude columns (like salary) from dominating model calculations.
Imputing missing values prevents errors during neural network training.
ColumnTransformer groups preprocessing steps into a single reusable object.

What This Accomplishes

Loads raw user profiles data.
Transforms numeric and categorical columns.
Splits data and serializes the pipeline for reuse.