Project 16: NLP & LLM Application Project

The diagram below displays the Retrieval-Augmented Generation (RAG) architecture. Raw documents are chunked and converted into vector embeddings. When a user asks a query, a semantic search retrieves relevant chunks, which are passed as context to prompt the LLM to generate the answer.

STEP 1

Activate Python Virtual Environment

Point the terminal execution environment to the course conda sandbox environment.

$ conda activate ds_ai_ml

This targets active python libraries to the isolated virtual sandbox.

STEP 2

Create Project Folders inside Linux VM

Create a dedicated folder for the project files inside your guest VM home folder directory.

$ mkdir -p ~/Projects/nlp_app && cd ~/Projects/nlp_app

This sets up the working directory layout for the NLP code.

STEP 3

Install Hugging Face packages via Pip

Install PyTorch, transformers, and scikit-learn inside the active conda session.

$ pip install transformers torch scikit-learn pandas numpy

This installs Hugging Face, PyTorch, and scikit-learn to run text classification and transformers.

STEP 4

Create NLP script file in VS Code

Launch VS Code and create the NLP pipeline script file.

Launch VS Code via terminal "code ." -> New File -> Type: nlp_pipeline.py -> Paste Python code -> Save file

This registers the TF-IDF vectorizer, Naive Bayes classifier, Hugging Face sentiment analysis, and localized RAG search flow in `nlp_pipeline.py`.

STEP 5

Run and Verify the NLP pipeline

Execute the script to train the classifier and run RAG search queries.

$ python nlp_pipeline.py

This runs TF-IDF vectorization, fits the Naive Bayes model, runs the Hugging Face sentiment check, and executes RAG queries.

The flowchart below outlines the NLP pipeline execution flow. It details the steps from text cleaning and TF-IDF calculation to Hugging Face sentiment analysis and RAG prompt generation.

To run the NLP pipeline, we need the Python script file. Below is a line-by-line explanation of the code, followed by the combined template.

Step-by-Step Code Construction

Lines 1 - 7

Import NLP and ML modules

Include system packages, TF-IDF, Naive Bayes models, metrics, and Hugging Face pipeline interfaces in the script.

from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.naive_bayes import MultinomialNB from sklearn.metrics import classification_report from transformers import pipeline import numpy as np

These imports pull standard scikit-learn transformers, Naive Bayes models, and Hugging Face pipeline classes.

Lines 8 - 25

Train Naive Bayes Sentiment Classifier

Define a training dataset, compute TF-IDF features, fit a Naive Bayes model, and run sentiment predictions.

texts = ["I love this product", "This is terrible and bad", "Outstanding quality", "Disappointed with service"] labels = [1, 0, 1, 0] # 1=Positive, 0=Negative vectorizer = TfidfVectorizer() X = vectorizer.fit_transform(texts) clf = MultinomialNB() clf.fit(X, labels)

This builds a custom dataset, converts texts to TF-IDF metrics, and trains a Multinomial Naive Bayes model.

Lines 26 - 32

Load Hugging Face Pipeline

Load the sentiment analysis pipeline using Hugging Face to compare predictions.

try: classifier = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english") res = classifier("This course is absolutely amazing!") print(res) except Exception as e: print("HF pipeline failed to load:", e)

This tries to download and run a DistilBERT sentiment analysis model, printing predictions to the console.

Lines 33 - 48

Implement Localized RAG Semantic Search

Define document chunks, calculate TF-IDF similarity to find the most relevant chunk, and format the context prompt.

doc_chunks = ["Python is an interpreted high-level language.", "FastAPI is a modern web framework.", "PyTorch is an open source machine learning library."] query = "What is PyTorch?" query_vec = vectorizer.transform([query]) doc_vecs = vectorizer.transform(doc_chunks) similarities = (doc_vecs * query_vec.T).toarray() best_idx = np.argmax(similarities) prompt = f"Context: {doc_chunks[best_idx]}\n\nQuestion: {query}\nAnswer:"

This runs semantic query searches against document chunks, selects the best matching context, and formats the RAG prompt.

Production templates

# nlp_pipeline.py - Classical NLP and LLM RAG pipelines from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.naive_bayes import MultinomialNB from sklearn.metrics import classification_report from transformers import pipeline import numpy as np def main(): print("=== Part 1: Training Classical Naive Bayes Classifier ===") # Sample training data train_texts = [ "I love this course, the content is outstanding", "This is absolute garbage and a waste of time", "Excellent explanations and detailed step-by-step guides", "Terrible presentation, useless examples, and bad support", "Great class, highly recommended for beginners", "Disappointed with the quality and slow response times" ] train_labels = [1, 0, 1, 0, 1, 0] # 1=Positive, 0=Negative vectorizer = TfidfVectorizer(stop_words='english') X_train = vectorizer.fit_transform(train_texts) clf = MultinomialNB() clf.fit(X_train, train_labels) # Test classifier test_texts = [ "This class is great, I love the guides", "Bad experience, terrible quality" ] X_test = vectorizer.transform(test_texts) preds = clf.predict(X_test) for text, pred in zip(test_texts, preds): sentiment = "Positive" if pred == 1 else "Negative" print(f" - Text: \"{text}\" | Predicted Sentiment: {sentiment}") print("\n=== Part 2: Loading Hugging Face Pipeline ===") try: print("Loading DistilBERT model. Please wait...") sentiment_pipeline = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english") res = sentiment_pipeline("I am incredibly happy with how this project turned out!") print(f" - Hugging Face Pipeline output: {res}") except Exception as e: print(f" - HF Pipeline failed to run: {e}") print("\n=== Part 3: Running Localized RAG Semantic Search Flow ===") # Document knowledge base doc_chunks = [ "Python is an interpreted, high-level, general-purpose programming language created by Guido van Rossum.", "FastAPI is a modern, fast, web framework for building APIs with Python 3.8+ based on standard Python type hints.", "PyTorch is an open-source machine learning library based on the Torch library, used for applications such as computer vision and NLP.", "Docker is a set of platform-as-a-service products that use OS-level virtualization to deliver software in packages called containers." ] query = "What is FastAPI?" print(f"Query: \"{query}\"") # Calculate cosine similarity using TF-IDF vectors rag_vectorizer = TfidfVectorizer(stop_words='english') X_docs = rag_vectorizer.fit_transform(doc_chunks) X_query = rag_vectorizer.transform([query]) # Dot product of normalized vectors yields cosine similarities similarities = (X_docs * X_query.T).toarray().flatten() best_match_idx = np.argmax(similarities) best_score = similarities[best_match_idx] print(f" - Best Match Index: {best_match_idx} | Similarity Score: {best_score:.4f}") print(f" - Retext Context chunk: \"{doc_chunks[best_match_idx]}\"") # Construct Prompt context prompt_context = f"""Context: {doc_chunks[best_match_idx]} Question: {query} Answer:""" print("\nGenerated Prompt for LLM:") print(prompt_context) print("\n=== NLP & LLM Project Successfully Complete! ===") if __name__ == "__main__": main()

Verify that the following configurations and outputs exist inside your project workspace.

Created Files / Templates

~/Projects/nlp_app/nlp_pipeline.py - NLP and RAG pipeline script file.

Verification Artifacts / Execution Proof

Sentiment predictions logged for test sentences.
Hugging Face pipeline returning labels and scores.
RAG prompts generated with relevant retrieved context.

Architectural Intent & Operational Impact

Why We Did This

TF-IDF features reflect term importance, making them useful for fast text classification.
Cosine similarity matches query vectors to document vectors, enabling semantic search.
RAG pipelines retrieve relevant context to ground LLM generation, reducing hallucination risks.

What This Accomplishes

Trains a Naive Bayes sentiment classifier.
Loads a Hugging Face pipeline for sentiment analysis.
Implements semantic search to construct prompt contexts.