Practice Project 18

Emerging AI Domains Exploration: Speech, Robotics & Reinforcement Learning

Build demonstrations across voice interfaces, simple robotic grid navigation, and tabular Q-learning to solve reinforcement learning environments.

Domain / Environment
Speech / Robotics / RL / Conda VM
Difficulty
Intermediate (3/5)
Course Module
Emerging AI Domains
Deliverables
Q-learning Grid script & Q-table verification outputs
1. Reinforcement Learning Agent Loop

The diagram below displays the Reinforcement Learning (RL) agent-environment loop. The Q-learning agent takes actions (up/down/left/right) in the grid-world environment. In return, the environment provides the new state and rewards, which are used to update the Q-table values.

Q-Learning Agent Q-Table Lookups Epsilon-Greedy Policy execution Action (a): Up/Down/Left/Right Grid Environment Start state: (0, 0) Trap state: (1, 1) -100 Goal state: (3, 3) +100 Feedback: State (s') & Reward (r)
2. Part 1: Step-by-Step Action Items & Key Execution Steps
STEP 1

Activate Python Virtual Environment

Point the terminal execution environment to the course conda sandbox environment.

$ conda activate ds_ai_ml
This targets active python libraries to the isolated virtual sandbox.
STEP 2

Create Project Folders inside Linux VM

Create a dedicated folder for the project files inside your guest VM home folder directory.

$ mkdir -p ~/Projects/emerging_ai && cd ~/Projects/emerging_ai
This sets up the working directory layout for the emerging AI code files.
STEP 3

Install TTS packages via Pip

Install the Google Text-to-Speech library inside the active conda session.

$ pip install gTTS numpy matplotlib
This installs the `gTTS` library to convert text statements into MP3 audio outputs.
STEP 4

Create Q-learning script file in VS Code

Launch VS Code and create the RL training script file.

Launch VS Code via terminal "code ." -> New File -> Type: q_learning_grid.py -> Paste Python code -> Save file
This registers the grid environment and tabular Q-learning algorithm in `q_learning_grid.py`.
STEP 5

Execute Reinforcement Learning training

Run the script to train the RL agent and output the learned Q-table.

$ python q_learning_grid.py
This runs Q-learning training episodes, prints the final learned Q-table, and saves a TTS audio file.
STEP 6

Verify generated TTS output

Play the generated MP3 file to confirm the text-to-speech conversion was successful.

$ aplay voice_output.mp3
This runs the default audio player inside your VM to play the output file. (If `aplay` is missing, you can skip audio playback checks).
3. Reinforcement Learning Execution Flow

The flowchart below outlines the reinforcement learning execution flow. It details the steps from setting hyperparameter values to updating Q-table values and running policy checks.

1. Initialize Initialize Q-table values to 0 np.zeros((16, 4)) 2. Select Action Choose action via epsilon-greedy rule random() < epsilon 3. Step Env Move coordinates and calculate reward state_next, reward 4. Update Q Update Q values using Bellman formula Bellman equation 5. Save Q-table Print final values to console log Console logging
4. Part 2: Complete Deliverable Assets & Production Templates

To run the demonstrations, we need the Python script file. Below is a line-by-line explanation of the code, followed by the combined template.

Step-by-Step Code Construction

Lines 1 - 6

Import tabular and TTS modules

Include system packages, Numpy arrays, Google Text-to-Speech libraries, and mathematical modules in the script.

import numpy as np from gtts import gTTS import os import random
These imports pull standard Numpy arrays, gTTS API wrappers, and random selection generators.
Lines 7 - 18

Define Grid World Environment

Define a 4x4 Grid World environment with starting, trap, and goal coordinates.

grid_size = 4 start = (0, 0) trap = (1, 1) goal = (3, 3) actions = ["UP", "DOWN", "LEFT", "RIGHT"]
This configures a 4x4 grid navigation coordinate layout, defining trap and target states.
Lines 19 - 35

Implement Tabular Q-Learning Loop

Fit the Q-table using the Bellman equation. Balance exploration and exploitation using the epsilon-greedy rule.

q_table = np.zeros((16, 4)) # Training loop for episode in range(500): state = 0 # Starting index while state != 15: # Goal index if random.uniform(0, 1) < epsilon: action = random.randint(0, 3) # Explore else: action = np.argmax(q_table[state]) # Exploit # Transition and calculate reward state_next, reward = step(state, action) q_table[state, action] = q_table[state, action] + alpha * (reward + gamma * np.max(q_table[state_next]) - q_table[state, action]) state = state_next
This updates the Q-table iteratively over 500 episodes using Bellman calculations.
Lines 36 - 45

Generate TTS Audio File

Convert text reports into MP3 audio outputs using the gTTS API.

text = "Reinforcement learning agent successfully trained!" tts = gTTS(text=text, lang='en') tts.save("voice_output.mp3")
This uses the Google Text-to-Speech API to save the statement as an MP3 audio file.

Production templates

1. Python script (Save as ~/Projects/emerging_ai/q_learning_grid.py):

# q_learning_grid.py - Tabular Q-learning in Grid World import numpy as np from gtts import gTTS import os import random def state_to_idx(state): (r, c) = state return r * 4 + c def idx_to_state(idx): return (idx // 4, idx % 4) def step(state_idx, action_idx): (r, c) = idx_to_state(state_idx) if action_idx == 0: # UP r = max(0, r - 1) elif action_idx == 1: # DOWN r = min(3, r + 1) elif action_idx == 2: # LEFT c = max(0, c - 1) elif action_idx == 3: # RIGHT c = min(3, c + 1) next_state = (r, c) next_idx = state_to_idx(next_state) if next_state == (3, 3): # Goal reward = 100.0 elif next_state == (1, 1): # Trap reward = -100.0 else: reward = -1.0 # Step penalty return next_idx, reward def main(): print("=== Part 1: Tabular Q-Learning training ===") # Hyperparameters alpha = 0.1 # Learning rate gamma = 0.99 # Discount factor epsilon = 0.2 # Exploration rate q_table = np.zeros((16, 4)) # 16 states, 4 actions for episode in range(1000): state_idx = 0 # Start at (0,0) while state_idx != 15: # Goal state (3,3) index is 15 if random.uniform(0, 1) < epsilon: action_idx = random.randint(0, 3) # Explore else: action_idx = np.argmax(q_table[state_idx]) # Exploit next_idx, reward = step(state_idx, action_idx) # Bellman Optimality Equation update best_next = np.max(q_table[next_idx]) q_table[state_idx, action_idx] += alpha * (reward + gamma * best_next - q_table[state_idx, action_idx]) state_idx = next_idx print("Training completed over 1000 episodes.") print("\nFinal Learned Q-Table:") print(" State | UP | DOWN | LEFT | RIGHT") for i in range(16): state_coords = idx_to_state(i) print(f" {str(state_coords):7} | {q_table[i,0]:7.2f} | {q_table[i,1]:7.2f} | {q_table[i,2]:7.2f} | {q_table[i,3]:7.2f}") print("\n=== Part 2: Generate Audio TTS Report ===") report_text = "Q learning training complete. The reinforcement learning agent learned to navigate the grid while avoiding the trap at coordinate 1, 1." print(f"Report text: \"{report_text}\"") tts = gTTS(text=report_text, lang='en') tts_filename = "voice_output.mp3" tts.save(tts_filename) print(f"Audio file successfully exported: {tts_filename}") print("\n=== Emerging AI Domains Demos Successfully Complete! ===") if __name__ == "__main__": main()
5. Deliverables Summary

Verify that the following configurations and outputs exist inside your project workspace.

Created Files / Templates

  • ~/Projects/emerging_ai/q_learning_grid.py - Q-learning grid script file.
  • ~/Projects/emerging_ai/voice_output.mp3 - Exported audio TTS file.

Verification Artifacts / Execution Proof

  • Q-table convergence showing high values pointing towards index 15.
  • Trap state (1,1) showing negative Q-values.
  • MP3 audio output file generated successfully.
6. Closing Explanation: Why We Did This & What It Accomplishes

Architectural Intent & Operational Impact

Why We Did This

What This Accomplishes

Data Science & AI/ML Training Program | Practice Project 18 | Emerging AI Domains