Project 10 – Intelligent Cloud Operations Platform

1. CloudOps and AI Observability Architecture

Modern cloud platforms handle millions of requests and generate huge volumes of logs, metrics, and invoices. Manually keeping track of this information is impossible. We solve this by constructing an Intelligent Operations Center: a centralized system that gathers performance metrics, alerts engineers on service disruptions, utilizes a vector database RAG pipeline to answer troubleshooting queries, and audits resource costs for waste.

2. Step-by-Step Action Items

PHASE 1 · STEP 1 Deploy Prometheus and Grafana on the Host Environment

Before starting AI integration, we must deploy our metrics engine. You will configure Prometheus and Grafana as Docker containers running side-by-side with your local Kubernetes resources.

Create the metrics configuration directory:

mkdir -p ~/monitoring cd ~/monitoring

Write the Prometheus global configuration file (prometheus.yml):

cat > prometheus.yml << 'EOF' global: scrape_interval: 15s evaluation_interval: 15s rule_files: - "alert_rules.yml" scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'kubernetes-nodes' static_configs: - targets: ['172.17.0.1:9100'] # Docker Bridge IP for host system Node Exporter labels: node: 'abc-retail-control-plane' EOF

Prometheus uses a pull model, fetching logs/metrics at a predefined rate (here every 15s). Node Exporter runs on port 9100 on the host interface and converts physical OS state variables (RAM usage, CPU spikes, disk space) into key-value formats readable by Prometheus.

Create the Alert Rules configuration file (alert_rules.yml):

cat > alert_rules.yml << 'EOF' groups: - name: abc_retail_alert_rules rules: - alert: CriticalCPUUtilization expr: node_load1 > 2.0 for: 1m labels: severity: critical annotations: summary: "Host CPU utilization is critical" description: "System 1-minute load average is above 2.0." - alert: LowAvailableDisk expr: (node_filesystem_free_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15 for: 2m labels: severity: warning annotations: summary: "Disk space is running low" description: "Root partition space is below 15%." EOF

Spin up the entire monitoring stack using Docker Compose:

cat > docker-compose.monitoring.yml << 'EOF' version: '3.8' services: prometheus: image: prom/prometheus:v2.45.0 container_name: prometheus ports: - "9090:9090" volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml - ./alert_rules.yml:/etc/prometheus/alert_rules.yml - prometheus_data:/prometheus restart: unless-stopped grafana: image: grafana/grafana:10.0.0 container_name: grafana ports: - "3000:3000" volumes: - grafana_data:/var/lib/grafana environment: - GF_SECURITY_ADMIN_PASSWORD=AdminPass2024! restart: unless-stopped node-exporter: image: prom/node-exporter:v1.6.0 container_name: node-exporter pid: "host" volumes: - "/:/host:ro,rslave" ports: - "9100:9100" restart: unless-stopped volumes: prometheus_data: grafana_data: EOF docker compose -f docker-compose.monitoring.yml up -d

PHASE 2 · STEP 1 Implement FinOps Cost Auditing Program

Cloud bills can spin out of control quickly if unused resources remain active. You will build a Python program that scans the environment (using simulated data matching AWS/Azure APIs) to detect cost leaks like idle VMs, unused volumes, and wrong storage tiers.

Create a directory for FinOps scripts:

mkdir -p ~/finops cd ~/finops

Write the Cost Audit Tool (finops_audit.py):

finops_audit.py

import json # Mock JSON data representing active infrastructure assets mock_assets = { "instances": [ {"id": "i-019a7fd9a1b2", "type": "t3.xlarge", "cpu_util_avg": 2.4, "monthly_cost": 136.0, "status": "running", "owner": "dev-team"}, {"id": "i-08a8e1b2f90a", "type": "t3.medium", "cpu_util_avg": 78.5, "monthly_cost": 34.0, "status": "running", "owner": "prod-team"}, {"id": "i-09ab0192bcde", "type": "m5.2xlarge", "cpu_util_avg": 1.1, "monthly_cost": 288.0, "status": "running", "owner": "staging-team"}, {"id": "i-029cba810fd2", "type": "t3.micro", "cpu_util_avg": 0.0, "monthly_cost": 8.5, "status": "stopped", "owner": "sandbox-team"} ], "volumes": [ {"id": "vol-018f921ab0", "size_gb": 500, "type": "gp2", "attached": False, "monthly_cost": 50.0}, {"id": "vol-0a8b92ef30", "size_gb": 100, "type": "gp3", "attached": True, "monthly_cost": 8.0} ] } def analyze_costs(data): total_spend = 0 savings_opportunities = [] print("--- ABC RETAIL CLOUD RESOURCE AUDIT ---") # Analyze Instances for inst in data["instances"]: total_spend += inst["monthly_cost"] if inst["status"] == "running" and inst["cpu_util_avg"] < 5.0: potential_saving = inst["monthly_cost"] savings_opportunities.append({ "resource_id": inst["id"], "type": "Idle VM", "current_cost": inst["monthly_cost"], "recommendation": f"Terminate or downscale {inst['id']} (Avg CPU: {inst['cpu_util_avg']}%). Recommended size: t3.small.", "saving": potential_saving }) # Analyze Volumes for vol in data["volumes"]: total_spend += vol["monthly_cost"] if not vol["attached"]: savings_opportunities.append({ "resource_id": vol["id"], "type": "Unattached Storage", "current_cost": vol["monthly_cost"], "recommendation": f"Delete vol {vol['id']} (Unattached since 30 days).", "saving": vol["monthly_cost"] }) print(f"Current Estimated Monthly Spend: ${total_spend:.2f}") print(f"Identified Waste Resources: {len(savings_opportunities)}") print("\n--- OPTIMIZATION OPPORTUNITIES ---") total_savings = 0 for idx, opt in enumerate(savings_opportunities, 1): print(f"{idx}. [{opt['type']}] Resource: {opt['resource_id']}") print(f" Recommendation: {opt['recommendation']}") print(f" Potential Saving: ${opt['saving']:.2f}/month") total_savings += opt["saving"] print("---------------------------------------") print(f"Total Actionable Monthly Savings: ${total_savings:.2f}") print(f"New Monthly Target Spend: ${total_spend - total_savings:.2f}") if __name__ == "__main__": analyze_costs(mock_assets)

Execute the cost analysis script to get optimization recommendations:

python3 finops_audit.py

This FinOps auditing script identifies waste in cloud environments. It evaluates assets against rules like "average CPU utilization < 5%" or "storage volume unattached to any server," listing specific actions to save money. In enterprise cloud centers, tools like AWS Cost Explorer or Azure Advisor perform similar checks dynamically.

PHASE 2 · STEP 2 Build a GenAI Operations Assistant with RAG Architecture

When an outage occurs, finding the correct runbook or command takes time. You will construct a Retrieval-Augmented Generation (RAG) pipeline in Python. It indexes technical runbooks and uses vector-like search (TF-IDF/cosine similarity) to find the most relevant troubleshooting documents to feed into a large language model prompt.

Install Python dependencies for vector computation and formatting:

pip install scikit-learn numpy

Create the AI script (ai_ops_assistant.py):

ai_ops_assistant.py

import numpy as np from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine_similarity # Knowledge Base: Runbooks for ABC Retail Cloud Infrastructure runbooks = [ { "title": "Restarting Kubernetes Web Deployments", "content": "To restart a crashed web container deployment in Kubernetes, use: 'kubectl rollout restart deployment/abc-retail-web -n abc-retail'. To view status, use: 'kubectl rollout status deployment/abc-retail-web -n abc-retail'." }, { "title": "MariaDB Storage Space Recovery", "content": "If the database disk space fills up, connect via mysql client and run 'PURGE BINARY LOGS TO ...'. Ensure the PVC is expanded using 'kubectl edit pvc mariadb-data -n abc-retail' and setting the storage attribute to a higher limit." }, { "title": "HAProxy Load Balancer Troubleshooting", "content": "To check HAProxy service health, run 'sudo systemctl status haproxy'. If configuration is updated, validate using 'haproxy -c -f /etc/haproxy/haproxy.cfg' before reloading the daemon via 'sudo systemctl reload haproxy'." }, { "title": "Vault Secret Retrieval Errors", "content": "If the web app cannot read credentials, check token status with 'vault token lookup'. Ensure the shell variable 'export VAULT_ADDR=http://127.0.0.1:8200' is set and the container has network access to the Vault instance." } ] def search_knowledge_base(query): corpus = [r["content"] for r in runbooks] vectorizer = TfidfVectorizer().fit_transform(corpus) query_vector = TfidfVectorizer().fit(corpus).transform([query]) similarities = cosine_similarity(query_vector, vectorizer).flatten() best_match_idx = np.argmax(similarities) if similarities[best_match_idx] > 0.15: return runbooks[best_match_idx] return None def main(): print("=== ABC RETAIL CO-PILOT: AI OPERATIONS ASSISTANT ===") print("Grounding context enabled. Type 'exit' to quit.\n") while True: query = input("Ask CloudOps Assistant: ") if query.lower() == 'exit': break doc = search_knowledge_base(query) if doc: print("\n[AI Assistant grounded response based on Runbook: " + doc["title"] + "]") # In a live app, this context is appended to the LLM prompt. # We mock the LLM output using the retrieved, grounded facts: print(f"Based on ABC Retail runbooks, here is the resolution procedure:\n{doc['content']}\n") else: print("\n[AI Assistant Response]") print("Sorry, I could not find a relevant troubleshooting runbook in the grounded database. Please refer to system administrators.\n") if __name__ == "__main__": main()

Run the AI Operations assistant and type a query like "how to restart kubernetes web server" or "haproxy service failed":

python3 ai_ops_assistant.py

RAG (Retrieval-Augmented Generation) prevents AI "hallucinations" by searching a vector database of internal runbooks for matching keywords or semantic vectors, extracting the matching paragraph, and supplying it as context inside the LLM prompt. This guarantees that recommendations align with ABC Retail's actual operating standards rather than generic web search responses. Type 'exit' when finished.

PHASE 3 · STEP 1 Simulate and Troubleshoot Production Outages

To validate your operations center, you will intentionally trigger system faults (simulating production issues) and use Prometheus, Grafana, and your AI assistant to resolve them.

Simulate Incident 1: Kubernetes Pod Outage. Cause a crash on your web deployment:

# Scale deployment down to zero kubectl scale deployment/abc-retail-web --replicas=0 -n abc-retail

This simulates a total web pod deletion or infrastructure failure. The web server immediately goes down, and HTTP requests fail.

Observe the outage:
• Access http://[VM-IP]:9090 (Prometheus UI) and search for the status metric: up{job="abc-retail-web"}. It should show 0 (Down).
• Access your Grafana SLA dashboard. The "Uptime %" gauge will drop.

Solve the incident using the AI assistant:
• Run python3 ai_ops_assistant.py.
• Ask: "how to restart kubernetes web server".
• The assistant returns the command: kubectl rollout restart deployment/abc-retail-web -n abc-retail.
• Run that command to repair the cluster:

kubectl scale deployment/abc-retail-web --replicas=2 -n abc-retail kubectl rollout status deployment/abc-retail-web -n abc-retail

Restoring the replica count brings the web pods back online. The Prometheus scrape metrics will recover to "1" on the next interval, and the Grafana status indicators will turn green again.

3. Deliverables Summary

📄 Files to Submit

monitoring/prometheus.yml
monitoring/alert_rules.yml
monitoring/docker-compose.monitoring.yml
finops/finops_audit.py (cost script)
finops/cloud_cost_analysis.md (written report)
ai_ops_assistant.py (RAG script)
Cloud Operations Handbook (Markdown)
Screenshot: Grafana SLA Dashboard with metrics
Screenshot: AI Assistant terminal interaction
Screenshot: Prometheus targets showing active checks

✅ Verification Checklist

Prometheus container is actively running on port 9090
Grafana container is actively running on port 3000
Node Exporter is exposing metrics on port 9100
Prometheus successfully scrapes host node statistics
Alert rules file is loaded and parsed correctly
FinOps script calculates potential cost optimization accurately
AI assistant retrieves correct runbook matching query
RAG pipeline returns appropriate context block
Incident simulation completed and documented
Handbooks cover all 10 required operations sections

4. Why We Did This & What It Accomplishes

Strategic Intent & Operational Impact

Why We Did This

Operating large systems requires visibility. A platform without metrics is a black box — if a failure happens, you won't know why, or how to prevent it. Implementing Prometheus and Grafana establishes a baseline for System Reliability Engineering (SRE).
Generative AI RAG pipelines allow operations teams to respond to incidents faster. When a pager goes off at 3:00 AM, the engineer doesn't have to search hundreds of pages of documentation — the AI searches the vector store and extracts the exact remediation steps instantly.
FinOps is a vital cloud engineering discipline. Moving to the cloud makes it easy to spin up expensive servers, but easy to forget to turn them off. Continuous cost optimization ensures that performance does not come at the expense of fiscal responsibility.

What This Accomplishes

ABC Retail now operates an **observability and intelligent orchestration suite** matching enterprise scale: centralized metrics collection, active alert escalation, AI-assisted runbook lookup, and automatic cost leak identification.
You have demonstrated skill across the full operational spectrum — not just deploying infrastructure but managing it sustainably using AI, FinOps, and SRE best practices. This matches the day-to-day responsibilities expected in modern CloudOps engineering.