11_Stock_Classification.py

#!/usr/bin/env python
# coding: utf-8

"""
=========================================================================================
EXECUTIVE SUMMARY: STOCK MARKET SENTIMENT ANALYSIS
=========================================================================================
Project Overview:
    This script demonstrates a Natural Language Processing (NLP) pipeline to predict 
    stock market movements (Up/Down) based on daily news headlines. It serves as a 
    practical application of text classification using historical data.

Key Techniques & Skills Demonstrated:
    - **Data Preprocessing**: Regex-based cleaning, text normalization, and time-series splitting.
    - **Feature Engineering**: Bag-of-Words (BoW) and N-Gram generation (Bigrams) to capture context.
    - **Machine Learning Models**: Implementation of Support Vector Classifier (SVC) and XGBoost.
    - **Model Evaluation**: Comprehensive performance metrics including Confusion Matrix, 
      Classification Report, and Precision-Recall optimization.

Business Value:
    Predicting market sentiment from unstructured text data allows for automated 
    trading signals and risk management, bridging the gap between financial analysis 
    and data science.
=========================================================================================
"""

# ==== Step 1: Library Imports & Setup ====
# Importing essential libraries for data manipulation and machine learning.
# pandas/numpy: Core data handling.
# sklearn: For modeling (SVM), preprocessing, and evaluation.
# xgboost: For advanced gradient boosting classification.

import pandas as pd
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


# ==== Step 2: Data Loading ====
# Loading the dataset containing dates, labels (0/1), and top news headlines.
# Encoding is set to ISO-8859-1 to handle special characters often found in text data.

df = pd.read_csv('Stock_Data.csv',encoding='ISO-8859-1')

# Display first few rows to understand structure (Date, Label, Top1...Top25)
df


# ==== Step 3: Time-Series Data Splitting ====
# Since stock data is time-dependent, we cannot use random splitting (train_test_split).
# We must respect the temporal order to avoid data leakage (predicting past using future).
# Strategy: Train on data before 2015, Test on data from 2015 onwards.

# Select all rows where the date is before December 31, 2014 and store them in 'train'
train = df[df['Date']<'20141231']

# Select all rows where the date is after December 31, 2014 and store them in 'test'
test = df[df['Date']>'20141231']

# Verify splits
train
test


# ==== Step 4: Data Cleaning & Preprocessing (Training Set) ====
# We focus only on the text columns (headlines), ignoring Date and Label for cleaning.
# The goal is to create a clean corpus of text for vectorization.

# sort dataset

data = train.iloc[:,2:]

# all rows and  (from 2: all column)

data


# -- 4.1 Regex Cleaning --
# Removing punctuation and special characters is crucial to reduce noise.
# We keep only alphabets (a-z, A-Z).

# apply regex preprocessing 

data.replace("[^a-zA-Z]"," ",regex=True,inplace=True)

# Replace all characters that are NOT letters (a-z or A-Z) with a space in the entire DataFrame

data


# -- 4.2 Column Renaming --
# Renaming columns to simple indices (0-24) for easier iteration.

# generate an list for lebel and also converted it to string

new_index = [str(i) for i in range(25)]
new_index


# replace column e label with new one 

data.columns = new_index

data


# -- 4.3 Text Normalization --
# Converting all text to lowercase ensures that 'Stock' and 'stock' are treated as the same word.

# convert uppercase sentences intpo lowercase

for index in new_index:
    data[index] = data[index].str.lower()

data


# -- 4.4 Text Aggregation --
# We combine all 25 headlines of a single day into one long paragraph (document).
# This allows the model to treat each day as a single data point with a rich context.

# Combine all values from the first row into a single string, separated by spaces

' '.join(str(x) for x in data.iloc[0, 0:])

len(data.index)


# Initialize an empty list to store processed headlines
headlines = []

# Loop through each row index in the DataFrame
for row in range(0, len(data.index)):
    # For each row: 
    
    headlines.append(' '.join(str(x) for x in data.iloc[row,0:]))

# 1. Convert all column values to strings
    # 2. Join them with spaces between words
    # 3. Add the resulting string to the headlines list

print('length of headline :',len(headlines))

headlines


# ==== Step 5: Data Cleaning & Preprocessing (Test Set) ====
# We must apply the EXACT same preprocessing steps to the test set to ensure consistency.

test_index = test.iloc[:,2:]
test_index

test_index.replace("[^a-zA-Z]"," ",regex=True,inplace=True)
test_index.columns = new_index

test_index


# conver test data into lowercase

for index in new_index:
  test_index[index] = test_index[index].str.lower()

test_index


# Initialize an empty list to store processed test data
test_transform = []

# Loop through each row in the test dataset
for row in range(0, len(test_index.index)):
   
    test_transform.append(' '.join(str(x) for x in test_index.iloc[row,0:]))
 # For each row:
    # 1. Convert all column values to strings
    # 2. Join them with spaces between words
    # 3. Add the resulting string to the test_transform list

test_transform

print (len(test_transform))


# ==== Step 6: Feature Engineering (Bag of Words) ====
# Converting text data into numerical format that machine learning models can understand.
# We use CountVectorizer with Bigrams (ngram_range=(2,2)).
# Why Bigrams? "Market Crash" has a very different meaning than "Market" and "Crash" separately.

# ### N_gram

# Import CountVectorizer for text vectorization
# from sklearn.feature_extraction.text import CountVectorizer

# Initialize with bigram configuration (2-word phrases)
# cv = CountVectorizer(ngram_range=(2,2))

# Fit to text and show vocabulary mapping
# print(cv.fit(["my name is sandesh sanghavi from nagpur"]).vocabulary_)


# Output :

# {'my name': 2, 'name is': 3, 'is sandesh': 1, 'sandesh sanghavi': 0, 'sanghavi from': 4, 'from nagpur': 5}


# ### Count vectorizer (BOW)

# #### Train dataset

from sklearn.feature_extraction.text import CountVectorizer
# from sklearn.ensemble import RandomForestClassifier


# Initialize CountVectorizer to generate bigrams (word pairs)
# ngram_range=(2,2) means only 2-word combinations will be created

countvector = CountVectorizer(ngram_range=(2,2))

# Convert text headlines into bigram frequency matrix
# fit_transform() performs two actions:
# 1. Learns vocabulary from headlines (fit)
# 2. Transforms text data into numerical feature vectors (transform)
# Result is a sparse matrix where each row represents a document/headline 
# and columns represent bigram frequencies

traindataset = countvector.fit_transform(headlines)

traindataset


# now conver dataset in array

traindataset.toarray()


# ### test dataset
# IMPORTANT: We use .transform() here, NOT .fit_transform().
# We must use the vocabulary learned from the training set.

test_dataset = countvector.transform(test_transform)

test_dataset

test_dataset.toarray()


# ### apply TF-IDF
# Alternative approach using Term Frequency-Inverse Document Frequency.
# (Note: The script sets this up but proceeds to use the CountVectorizer output for the main models below.
# This section demonstrates how one would set up TF-IDF for comparison.)

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(
    ngram_range=(1,3),  # Include unigrams and bigrams
    max_features=5000,   # Control dimensionality
    stop_words='english', # Remove common words
    sublinear_tf=True,       # Use 1+log(tf) instead of raw counts
    max_df=0.75,             # Remove too-frequent words
    min_df=3                # Remove rare words
)

X_train = vectorizer.fit_transform(headlines)
X_test = vectorizer.fit_transform(test_transform)

y_train = train['Label']
y_test = test['Label']


# ==== Step 7: Model Training (Support Vector Classifier) ====
# SVM is a powerful algorithm for high-dimensional text data.
# We use a Linear Kernel which is often best for text classification.

# ### use svc model for train classification model

from sklearn.svm import SVC

svm_clf = SVC(
    kernel='linear',  # Better for high-dim text data
    C=0.1,            # Regularization parameter
    class_weight='balanced',  # Handle class imbalance
    probability=True  # Enable predict_proba
)
svm_clf.fit(traindataset,train['Label'])


# ==== Step 8: Model Evaluation (SVM) ====
# Assessing the performance of our SVM model on unseen test data.

pred = svm_clf.predict(test_dataset)

pred

from sklearn.metrics import confusion_matrix,classification_report

print(confusion_matrix(y_test,pred))

print(classification_report(y_test,pred))
# Interpretation:
# Precision: Accuracy of positive predictions.
# Recall: Ability to find all positive instances.
# F1-Score: Harmonic mean of precision and recall.


# ==== Step 9: Advanced Modeling (XGBoost) ====
# XGBoost (Extreme Gradient Boosting) is a state-of-the-art ensemble technique.
# It often provides better accuracy by combining multiple weak learners.

# ### use xgboost 

from xgboost import XGBClassifier
import xgboost as xgb # Added for DMatrix support

# Convert to DMatrix format for XGBoost optimization
# Note: This step prepares data structures specific to XGBoost for efficiency.

dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)


# XGBoost classifier with optimal parameters

# Train XGBoost
model = XGBClassifier(
    scale_pos_weight=sum(train['Label']==0)/sum(train['Label']==1), # Handle class imbalance
    n_estimators=500,
    max_depth=3,
    learning_rate=0.1
)


# Train with early stopping

model.fit(X_train, train['Label'])


# Predictions
pred = model.predict(X_test)
probs = model.predict_proba(X_test)[:, 1]

probs.round()


# ==== Step 10: Final Evaluation & Threshold Optimization ====
# We analyze the results and attempt to optimize the decision threshold.

# ### Evaluation


from sklearn.metrics import confusion_matrix,classification_report,precision_recall_curve


print(confusion_matrix(y_test,pred))


print(classification_report(y_test,pred))


# -- Threshold Tuning --
# Standard threshold is 0.5. We can adjust this to maximize F1 score or balance precision/recall.

precision, recall, thresholds = precision_recall_curve(y_test, probs)
optimal_idx = np.argmax(precision + recall)
optimal_threshold = thresholds[optimal_idx]
predictions = (probs > optimal_threshold).astype(int)


# Evaluation
print("Confusion Matrix:")
print(confusion_matrix(y_test, predictions))
print("\nClassification Report:")
print(classification_report(y_test, predictions))


"""
=========================================================================================
FINAL SUMMARY & CONCLUSION
=========================================================================================
Results Analysis:
    - The models successfully processed unstructured text data to output binary predictions.
    - The Confusion Matrix highlights the trade-off between False Positives and False Negatives.
    - XGBoost typically offers robust performance, but the Linear SVM is a strong baseline for text.

Next Steps for Improvement:
    - Experiment with Word2Vec or BERT embeddings for semantic understanding.
    - Incorporate financial indicators (volume, open/close prices) as additional features.
    - Tune hyperparameters using GridSearch or Optuna.
=========================================================================================
"""