Install the NLTK and spaCy core syntactic text processors, configure the Hugging Face Transformers pipeline engine, authenticate read access tokens, and verify deep-learning sentiment models inside a VM.
Environment
Conda Env / Linux VM
Difficulty
Intermediate (2/5)
Course Module
Natural Language Processing
Deliverables
HF Token Auth & Sentiment Pipeline Logs
1. System Architecture & Process Workflow
The diagram below displays the natural language processing compilation flow. Raw unstructured text is tokenized and tagged by spaCy or NLTK locally before being processed by the Hugging Face pipeline engine, which parses inputs through a pretrained Transformer model cached on disk to return sentiment label probabilities.
2. Part 1: Step-by-Step Action Items & Key Execution Steps
STEP 1
Activate Python Virtual Environment
Point the terminal execution environment to the course conda sandbox environment created in Setup Project 1.
$ conda activate ds_ai_ml
This action redirects paths and links the active shell to our isolated python binary libraries.
STEP 2
Install NLP and Deep Learning Packages via pip
Install PyTorch framework alongside NLTK, spaCy, and the Hugging Face Transformers module library.
This downloads the sentence punctuation index tables and list databases of common irrelevant text values.
STEP 5
Create a Hugging Face Account
Open Firefox web browser in your virtual workspace and sign up for a Hugging Face account.
Launch Firefox -> Navigate to https://huggingface.co -> Click "Sign Up" -> Complete Registration -> Verify your Email
This provides access permissions to download pre-trained LLMs and Transformers from the model hub registry.
STEP 6
Generate Read access tokens
Create a read-only token in your profile settings to query repositories via CLI sessions.
Click Profile Avatar (top-right) -> Click "Settings" -> Click "Access Tokens" in sidebar -> Click "New token" -> Set Name: course_token -> Set Role: Read -> Click "Generate a token" -> Click Copy button
This registers an API key credential (starts with `hf_...`), allowing you to fetch model weights programmatically.
STEP 7
Authenticate terminal session in Hugging Face
Launch the CLI login module in the terminal and write your generated access token key.
$ huggingface-cli login
When prompted, paste your access token (the characters will be hidden in terminal for security) and press **Enter**. Type **n** if asked to add Git credentials, and press **Enter** to complete authentication.
STEP 8
Save Python NLP Diagnostic Script
Open a document editor inside VS Code or the terminal and write your verification script code.
$ nano ~/verify_nlp.py
This opens nano editor inside terminal. Paste the script from Part 2 below, press **Ctrl + O** and **Enter** to save, and **Ctrl + X** to exit.
STEP 9
Execute verification script
Run the validation script using the python engine to verify NLP operations work.
$ python ~/verify_nlp.py
This runs the script. The terminal will download a DistilBERT model during first run and output classification labels.
3. Operational Pipeline Architecture
The flowchart below outlines the NLP environment initialization pipeline. It traces the steps from package installation and downloading language corpora to generating API keys, executing CLI authentication, and running the verification script.
4. Part 2: Complete Deliverable Assets & Production Templates
To verify the NLP stack, we will write a Python script that tokenizes text using NLTK, runs POS tagger and NER using spaCy, and classifies sentiment using Hugging Face. Below is a line-by-line explanation of the code, followed by the combined script.
Step-by-Step Code Construction
Lines 1 - 5
Import NLP and Transformer Packages
Include NLTK, spaCy, and Hugging Face Pipeline APIs in the code script.
import nltk
import spacy
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from transformers import pipeline
These imports import sentence tokenizers, stopword lists, dictionary models, and Hugging Face pipeline builders.
Lines 6 - 11
Tokenize Input String via NLTK
Load input text and tokenize it into list values using NLTK functions.
input_text = "Google DeepMind team designed a powerful AI coding agent named Antigravity."
words = word_tokenize(input_text)
stop_words = set(stopwords.words('english'))
filtered_words = [w for w in words if w.lower() not in stop_words]
This tokenizes the sentence, loads English stopwords, and filters out common grammar particles (like 'a') from the output.
Lines 12 - 16
Extract Named Entities via spaCy
Load the spaCy language model and retrieve entity groups from the string.
nlp = spacy.load("en_core_web_sm")
doc = nlp(input_text)
for ent in doc.ents:
print(f"Entity: {ent.text} | Label: {ent.label_}")
This parses the text through the small English dictionary model, printing named tags (like PERSON, ORG) found in the sentence.
Lines 17 - 21
Load and Run Sentiment Analysis Pipeline
Instantiate a pre-trained DistilBERT model pipeline to calculate sentiment values.
classifier = pipeline("sentiment-analysis")
result = classifier("Antigravity is an amazing AI agent that makes coding fun!")
print(f"Sentiment Label: {result[0]['label']}")
print(f"Sentiment Score: {result[0]['score']}")
This loads the default text classifier model, passes a test sentence, and prints label outputs and probability scores.
Combined NLP Diagnostic Script
Save the consolidated blocks above as ~/verify_nlp.py and execute it inside the active ds_ai_ml environment:
# verify_nlp.py - NLP and Hugging Face Verification Scriptimport nltk
import spacy
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from transformers import pipeline
defmain():
print("=== NLP & Hugging Face Diagnostics ===")
test_sentence = "Google DeepMind team designed a powerful AI coding agent named Antigravity."
print(f"Original Text: {test_sentence}")
# 1. NLTK Tokenization and Stopwords Filtering
tokens = word_tokenize(test_sentence)
stop_words = set(stopwords.words('english'))
filtered_tokens = [w for w in tokens if w.lower() not in stop_words]
print("\n[NLTK Tokenization Results]")
print(f"Tokens: {tokens[:10]}...")
print(f"Filtered: {filtered_tokens[:10]}...")
# 2. spaCy Named Entity Recognition
print("\n[spaCy Named Entity Recognition Results]")
nlp = spacy.load("en_core_web_sm")
doc = nlp(test_sentence)
for ent in doc.ents:
print(f" - Entity: {ent.text} | Label: {ent.label_}")
# 3. Hugging Face Sentiment Analysis Pipeline
print("\n[Hugging Face Sentiment Pipeline Results]")
# Automatically downloads default model: distilbert-base-uncased-finetuned-sst-2-english
classifier = pipeline("sentiment-analysis")
happy_sentence = "Antigravity is an amazing AI agent that makes coding fun!"
result = classifier(happy_sentence)[0]
print(f"Input: {happy_sentence}")
print(f" - Label: {result['label']}")
print(f" - Score: {result['score']:.4f}")
print("\n=== NLP Environment Successfully Verified! ===")
if __name__ == "__main__":
main()
5. Deliverables Summary
Verify that the following configurations and outputs exist inside your project workspace.