PROJECT 7

Observability Stack Deployment

Deploy local Node Exporters to generate metrics, configure Prometheus TSDB to scrape targets, and build interactive Grafana analytical dashboards.

Environment
Ubuntu / Prometheus / Grafana
Difficulty
Advanced
Course Module
Chapter 11: Observability
Deliverables
prom.yml & Dashboard Proof
1. System Architecture & Workflow

The monitoring topology below displays the scraping loop, showing how host resource metrics are collected, stored in Prometheus, and visualized in Grafana.

Linux Guest VM Node Exporter Port: 9100 Exposes CPU, Memory, Disk as Prometheus metrics Prometheus TSDB Scrape Engine (pull) Config: prometheus.yml Port: 9090 Grafana Server Visual Dashboards Port: 3000 Queries Prometheus to display graphs Scrapes Query
2. Part 1: Step-by-Step Action Items & Key Execution Steps
STEP 1

Install and Configure Node Exporter

Download and run Node Exporter on the target host to collect server metrics like CPU and memory utilization.

$ wget https://github.com/prometheus/node_exporter/releases/download/v1.6.0/node_exporter-1.6.0.linux-amd64.tar.gz
This command downloads the compiled Node Exporter binary archive from official GitHub releases.
$ tar -xvf node_exporter-1.6.0.linux-amd64.tar.gz && sudo mv node_exporter-1.6.0.linux-amd64/node_exporter /usr/local/bin/
This command extracts the zip archive contents and installs the Node Exporter binary to a global execution directory.
$ node_exporter --version
This command displays the binary version information, confirming that Node Exporter is ready to collect system metrics.
STEP 2

Install the Prometheus Time Series Database Engine

Download the Prometheus server archive, extract files, install execution binaries, and write target definitions.

$ wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
This command downloads the stable release archive of the Prometheus Time Series Database.
$ tar -xvf prometheus-2.45.0.linux-amd64.tar.gz && sudo mv prometheus-2.45.0.linux-amd64/prometheus /usr/local/bin/
This command extracts the archive folders and installs the Prometheus server daemon binary to a global bin directory.
$ sudo mkdir -p /etc/prometheus && sudo mv prometheus-2.45.0.linux-amd64/prometheus.yml /etc/prometheus/
This command creates a configuration folder and relocates the default configuration template to configure scrape intervals and targets.
STEP 3

Install the Grafana Dashboards Server

Register Grafana's official GPG verification keys, write package lists, install the software, and enable the systemd service.

$ sudo mkdir -p /etc/apt/keyrings && wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
This command downloads Grafana's GPG keys and imports them, enabling apt to verify the integrity of downloaded packages.
$ echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
This command registers Grafana's stable repository in the system's package listings, allowing apt to query updates.
$ sudo apt update && sudo apt install grafana -y
This command updates repository indexes and installs the Grafana dashboard server application.
$ sudo systemctl start grafana-server && sudo systemctl enable grafana-server
This command starts the Grafana background daemon process and configures the service to launch automatically during system startup.
STEP 4

Verify Scrape Targets and Configure Data Sources

Check the Prometheus targets interface, open the Grafana dashboard, and add your local Prometheus database as a data source.

$ curl -s http://localhost:9090/api/v1/targets | grep -o '"health":"up"'
This command queries Prometheus's targets API to verify that the local Node Exporter instance is being scraped successfully.

1. Open your browser and navigate to http://localhost:3000 (login with username admin and password admin). Change the default password.

2. Go to Connections -> Data Sources, click Add Data Source, select Prometheus, set the URL to http://localhost:9090, and click Save & Test.

3. Go to Dashboards -> Import, enter dashboard ID 1860 (Node Exporter Full Dashboard), and click Load to import a preconfigured dashboard.

3. Operational Pipeline Architecture

The workflow below traces the scrape-to-alert loop, showing how host activity triggers metric updates, Prometheus scrapes targets, and Grafana updates dashboards.

1. Collect Metrics Node Exporter reads system parameters GET /metrics 2. Scrape TSDB Prometheus pulls and records metrics Scrape interval: 15s 3. Visualize Grafana queries TSDB to refresh charts PromQL queries 4. Monitoring Dashboard renders real-time updates Dashboard 1860
4. Part 2: Complete Deliverable Assets & Production Templates

To monitor your system, we will configure Prometheus and set up Node Exporter as a background systemd service. Below is a step-by-step breakdown of how these configuration files are constructed, followed by the final combined files.

Step-by-Step Script Construction

Step 1

Define Prometheus Global Timing Settings

Setup the primary server scraping frequency configurations.

global: scrape_interval: 15s evaluation_interval: 15s
This sets Prometheus to scrape metric endpoints and evaluate alerting rules every 15 seconds.
Step 2

Configure Scrape Targets for Exporters

Define connection details for local monitoring services.

scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'node_exporter' static_configs: - targets: ['localhost:9100']
This tells Prometheus to scrape performance data from itself on port 9090 and from Node Exporter on port 9100.
Step 3

Configure the Node Exporter systemd Unit Section

Declare background daemon metadata and network startup requirements.

[Unit] Description=Prometheus Node Exporter Wants=network-online.target After=network-online.target
This defines the systemd service name and specifies that systemd must wait for network interfaces to be online before launching Node Exporter.
Step 4

Configure the Service Execution path and Install Directives

Define the execution user and configure standard multi-user system targets.

[Service] User=devops Group=devops Type=simple ExecStart=/usr/local/bin/node_exporter [Install] WantedBy=multi-user.target
This run configuration starts the Node Exporter binary located in /usr/local/bin as the user devops, and registers the service to start automatically under standard multi-user boot targets.

Combined Complete Configuration Files

Create these files in their respective folders inside the VM:

1. prometheus.yml

Save this configuration file as /etc/prometheus/prometheus.yml:

global: scrape_interval: 15s # Set the default scrape interval to every 15 seconds evaluation_interval: 15s # Evaluate rules every 15 seconds scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'node_exporter' static_configs: - targets: ['localhost:9100']

2. node_exporter.service

Save this systemd service definition as /etc/systemd/system/node_exporter.service:

[Unit] Description=Prometheus Node Exporter Wants=network-online.target After=network-online.target [Service] User=devops Group=devops Type=simple ExecStart=/usr/local/bin/node_exporter [Install] WantedBy=multi-user.target
5. Deliverables Summary

Verify that the following configurations and outputs exist inside your project workspace.

Created Files / Templates

  • /etc/prometheus/prometheus.yml - Scrape specifications configuration file.
  • /etc/systemd/system/node_exporter.service - Systemd unit service configuration file.

Verification Artifacts / Execution Proof

  • Output of systemctl status node_exporter showing the service status as active (running).
  • Successful target status (health state "UP") on the Prometheus console at http://localhost:9090/targets.
  • Screenshot of the imported Node Exporter dashboard rendering system performance statistics in Grafana.
6. Closing Explanation: Why We Did This & What It Accomplishes

Architectural Intent & Operational Impact

Why We Did This

What This Accomplishes

DevOps Professional Training Program | Project 7 | Observability Stack Deployments