Complete Jupyter Notebook -

• Introduction
  • How it was evaluated
  • The problem statement
• Data & Approach
  • Reshaping to long format
  • Feature engineering — teaching the model what water looks like
  • Handling the missing months honestly
  • A radar-only model for cloud-obscured months
  • Preventing leakage in the split
  • Aggregating months to a per-pixel decision
• Models
• Results
  • Validation (per-pixel, on the held-out grouped split)
  • Blind competition test (ragged, missing months data)
  • What I confirmed along the way
• Conclusion

Introduction

Aquaculture ponds are used to farm fish and shrimp in controlled water environments. They are easier to manage and more productive than open-water fishing, but to monitor them effectively we first need to know where they are. Satellite imagery offers a reliable way to map these ponds across large areas, supporting water management, environmental monitoring, and policy decisions, especially in regions where ground-level data is hard to come by.

This project was my entry to the Zindi GeoAI Aquaculture Pond Identification Challenge. The goal was to build a machine learning model that predicts whether a given location is an aquaculture pond or another type of land cover, using features derived from satellite data. Each location represents a 10m × 10m patch of ground, so the model needs to be precise.

The challenge came with a deliberate challenge: the model is trained on data from one time period and tested on data from a different one. The solution has to stay accurate as conditions change across seasons and years, not just perform well on familiar data, but perform well on data with missing values. These constraint shaped every decision I made.

How it was evaluated

The leaderboard used a weighted, multi-metric score:

• F1-Score (60%) — a balanced measure of precision and recall, important because aquaculture ponds are a small fraction of all locations.
• ROC-AUC (40%) — how well the model ranks pond locations above non-pond locations across all thresholds, giving a stable read regardless of class imbalance.

The model had to output two things: a binary target (pond or other) and a probability of being a pond. Setting a custom probability threshold was forbidden — the binary target had to come from the default 0.5 cutoff. That rule matters: it means the model can’t be tuned to game F1 by shifting the decision boundary; the probabilities themselves have to be well-placed around 0.5.

The problem statement

Two properties of the data defined the whole problem:

• The test set is blind — no labels, and I only see my score through the leaderboard.
• The training data is complete — no missing values in the bands.
• The test data has missing values, especially in the Sentinel-2 optical bands, which are blocked by cloud cover. The Sentinel-1 radar bands are far less affected, because radar penetrates cloud.

Training on clean, complete 12-month time series, but I predict on messy data from a different time period.

Data & Approach

Each pixel is described by a full year of monthly observations across twelve bands — two radar (VH, VV from Sentinel-1) and ten optical (blue, green, nir, nira, re1, re2, re3, red, swir1, swir2 from Sentinel-2). In the raw file, those months live side by side as wide columns (VH_01, VH_02, …, VH_12).

The Data:

Reshaping to long format

My first decision was to collapse the months into a long format, one row per pixel-month instead of one row per pixel. This does two things at once: it folds the seasonal signal into the training data (the model sees each month as its own observation), and it gives me a natural way to deal with missing months later — I can simply drop the rows that are missing, rather than imputing 240+ columns.

bands = ["VH", "VV", "blue", "green", "nir", "nira",
         "re1", "re2", "re3", "red", "swir1", "swir2"]

all_data = origin_data.reset_index()          # a unique id to pivot on
long_data = pd.wide_to_long(all_data, stubnames=bands,
                            i="index", j="month", sep="_", suffix=r"\d+")
long_data = long_data.reset_index()

The optical reflectance bands are stored as integers scaled by 10,000, I rescale them back to the 0–1 reflectance range before computing any index that is sensitive to scale.

optical = ["blue", "green", "nir", "nira", "re1", "re2", "re3", "red", "swir1", "swir2"]
long_data[optical] = long_data[optical] / 10000

Feature engineering — teaching the model what water looks like

A pond is, reflective body of water. I engineered features that make that signature explicit rather than expecting the model to detect water from raw bands.

• NDWI (green − nir)/(green + nir) — the classic water index; water reflects green and absorbs NIR.
• MNDWI (green − swir1)/(green + swir1) — a modified version using SWIR, which separates water from built-up land more cleanly.
• VH − VV — the radar backscatter difference. Because these are in decibels (log space), the ratio becomes a subtraction, not a division. Smooth water surfaces scatter radar differently than rough land.
• AWEI (two variants) — the Automated Water Extraction Index, in no-shadow and shadow-suppressing forms, designed to push the water/land contrast even harder than NDWI.
• EVI — the Enhanced Vegetation Index, computed after rescaling to reflectance because its +1 term assumes a 0–1 range.
• Otsu water flags — for each month I compute an Otsu threshold on the radar bands and flag pixels below it as “water”. Otsu finds the natural split between two populations, and these radar histograms are bimodal (water vs land), so the threshold cleanly separates them.

long_data['ndwi']  = (long_data["green"] - long_data["nir"])   / (long_data["green"] + long_data["nir"])
long_data['mndwi'] = (long_data["green"] - long_data["swir1"]) / (long_data["green"] + long_data["swir1"])
long_data['vh_vv'] = long_data['VH'] - long_data['VV']          # dB → subtraction, not ratio

long_data["awei_nsh"] = 4*(long_data["green"] - long_data["swir1"]) - (0.25*long_data["nir"] + 2.75*long_data["swir2"])
long_data["awei_sh"]  = long_data["blue"] + 2.5*long_data["green"] - 1.5*(long_data["nir"] + long_data["swir1"]) - 0.25*long_data["swir2"]
long_data["evi"]      = 2.5 * (long_data["nir"] - long_data["red"]) / (long_data["nir"] + 6*long_data["red"] - 7.5*long_data["blue"] + 1)

# per-month Otsu threshold on radar → binary water flag
from skimage.filters import threshold_otsu
thr = long_data.groupby('month')['VH'].apply(lambda s: threshold_otsu(s.dropna().values))
long_data['vh_water'] = (long_data['VH'] < long_data['month'].astype(int).map(thr)).astype(int)

Handling the missing months honestly

This is where the long format pays off. In the blind test set, cloudy months arrive as fully-NaN rows. Rather than imputing those months — fabricating optical values the satellite never saw, I simply drop them and predict only on the months that have real data.

cols = [c for c in test_final_data.columns if c not in ('index', 'month', 'ID')]
model_input = test_final_data.dropna(how="all", subset=cols)   # drop fully-cloudy months

The justification is that my model learned “what a pond looks like” from twelve-month data, and filling gaps with borrowed values produces test pixels whose feature distribution differs from anything in training. Dropping-and-aggregating keeps every prediction grounded in something the satellite actually measured..

A radar-only model for cloud-obscured months

Some missing rows are a special case: the optical bands are missing but the radar bands are present, because radar sees through cloud. Throwing these away wastes real signal and precious data. So alongside the full model I trained a radar-only model on just the always-present features (VH, VV, vh_vv, and the radar water flags), and routed each row to the right model.

radar_cols = ['month', 'VH', 'VV', 'vh_vv', 'vv_water', 'vh_water']

# a row is "radar-only" when all optical bands are NaN but radar is present
is_radar_only = X[optical_cols].isna().all(axis=1) & X[radar_cols].notna().all(axis=1)

proba.loc[full.index] = model.predict_proba(full)[:, 1]                   # full model
proba.loc[ro.index]   = radar_model.predict_proba(ro[radar_cols])[:, 1]   # radar-only model

This means a cloudy month still contributes a prediction based on what radar can see, instead of being silently dropped.

Preventing leakage in the split

Because in the long format a single pixel spans twelve rows(months), an ordinary random tain_test_split would scatter one pixel’s months across both train and test and since a pixel’s months are highly correlated, the model would effectively see the test pixels during training. I used a grouped, stratified split so every pixel lands wholly on one side.

from sklearn.model_selection import StratifiedGroupKFold

sgkf = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)
train_idx, test_idx = next(sgkf.split(X_10, long_y, groups=long_data['ID']))

# verify no pixel appears on both sides
assert len(set(long_data.iloc[train_idx]['ID']) & set(long_data.iloc[test_idx]['ID'])) == 0

Aggregating months to a per-pixel decision

The label is per pixel, but the model predicts per month. To collapse a pixel’s monthly predictions into one, I take the maximum probability across its available months (in blind test pixels had -7 months available). The reasoning for taking max month probability is l ask “does this pixel look strongly like a pond in its best available month?” and it works whether a pixel has five months or twelve.

pixel = X_new.groupby('ID').agg(proba=('proba', 'max'), label=('label', 'first'))
pred  = (pixel['proba'] >= 0.5).astype(int)

Models

I trained four classifiers, all evaluated per-pixel after max aggregation, and all on the same leakage-safe grouped split:

• XGBoost — gradient-boosted trees; scale-invariant, native NaN handling, my primary workhorse.
• Gradient Boosting (scikit-learn) — a regularised second boosting model for diversity.
• SVM (SVC) — inside a pipeline with StandardScaler and a cyclical sin/cos encoding of month, since a distance-based model needs scaled features and a circular notion of the calendar.
• TabPFN — a pretrained tabular transformer that needs no tuning and is well-suited to small tabular data like this.

The SVM pipeline is worth showing, because it’s the one model where preprocessing genuinely matters:

def month_cyclical(x):
    x = x.astype(float)
    radians = 2 * np.pi * x / 12
    return np.hstack([np.sin(radians), np.cos(radians)])   # Dec and Jan sit next to each other

preprocessor = ColumnTransformer(
    transformers=[('scale', StandardScaler(), scale_cols),
                  ('month', FunctionTransformer(month_cyclical), ['month'])],
    remainder='passthrough'
)
model = make_pipeline(preprocessor, SVC(random_state=42, probability=True))
model.fit(X_train, y_train)

Results

Validation (per-pixel, on the held-out grouped split)

On clean twelve-month validation data, every model was excellent:

Blind competition test (ragged, missing months data)

The blind test told the real story and the gap between validation and test is the whole lesson of this project:

My best models generalised from ~0.99 validation to ~0.87–0.89 on the blind set, because the grouped split and drop-don’t-impute strategy meant my validation was never inflated by leakage or fabricated data. My Final rank: 253 / 1314.

What I confirmed along the way

I also ran wide-format models with imputed months (including MICE / predictive-mean-matching) and they consistently scored below the long-format drop-and-aggregate approach. Those models dropped my accuracy from ~0.85 to ~0.75 because imputed months diluted every pixel with predicted-pond signals. That negative result is what gave me confidence the long+max design was the right call.

Conclusion

The models generalised well but not spectacularly, and the reason is instructive. My feature engineering captured where water is (spectral water indices, radar backscatter, Otsu flags) far better than when and how it cycle, the seasonality of ponds. The pond’s temporal signature, the fill-and-drain rhythm across the year is exactly what a clean twelve-month training set expresses richly and a test set with missing months doesnt. Also the max approach is likely to be affected by alot of False positives, coz it pick one month which said water and without seasonality it affect hte overall accuracy.

Two directions would close the gap to the leaderboard leaders:

1. Model the temporal signature in a form that survives missing months — persistence (fraction of months a pixel reads as water), cycle amplitude, or a fitted seasonal harmonic — rather than fragile summary statistics that mean different things when computed from five months versus twelve.
2. Model the missing values in the test set explicitly as part of the signal, rather than only routing around them.

The competition rewarded honesty about the train/test mismatch, and my solution’s clean generalisation reflects that. The remaining distance to the top is a temporal-modelling problem, not a preprocessing one a satisfying place to have landed, because it points clearly at what to build next.

• Introduction
• Methods
  • Data Import
  • Data Overview
  • Objective 1; Day to day sales trends
  • Objective 4; Product association
  • Objective 6; Difference between Black Friday and Cyber Monday sales
  • Business Recommendations

Introduction

My client is a partnership B2B company looking to leverage data-driven decision-making tools to help businesses understand and capitalize on market trends and drive growth in product sales. In this project, l analyzed sales for Thanksgiving and Black Friday week of 2021 from Monday 22 November –to Monday 29 November.

The main objectives were:

1. What product sales trends were observed during the week?
2. What are the top-selling products and categories during Thanksgiving week?
3. What are the highest commissioned products?
4. Which products and product categories are most frequently purchased together?
5. How were product sales trends similar or different between Black Friday and Cyber Monday?
6. What business recommendations can you provide based on the above analysis?
7. How would the product classification model be deployed and productionalised to support client-facing product features?

To illustrate the power of sales data analysis l presented my findings in a live R shiny interactive dashboard highlighting the objectives on each tab.

Click to view Sales Presentation -

Methods

For this project, I utilized Python to summarize, wrangle data, and perform statistical analysis.Click to view data wrangling and analysis in Python - l exported the data into R, I compiled and visualized the insights, presenting them as an organized interactive R shiny dashboard to enhance the user experience for stakeholders Click to view R Shiny dashboard and visualisation script -

Data Import

A Naïve Bayes text classifier was used to categorize product items using product descriptions, see more here

#PYTHON CODE BLOCK
# Predict categories for unlabeled data
# Create a pipeline with TF-IDF vectorizer and Naive bayes hadnle textual data and imblanced datasets
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer

from sklearn.metrics import classification_report, accuracy_score
from sklearn.pipeline import Pipeline

X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=0.2, random_state=42)## no need to sub-sample teh data

pipeline_2 = Pipeline([
    ('tfidf', TfidfVectorizer(max_df=0.50, 
                              min_df=1, max_features=500000, 
                              ngram_range=(1, 2), 
                              stop_words='english', 
                               use_idf=True, 
                              smooth_idf=True, 
                              sublinear_tf=True)),   # Adjust max_features based on your dataset size, ngrams captures bi-words
    #('svd', TruncatedSVD(n_components=1000)), # Adjust n_components as needed
   ('nb', MultinomialNB(alpha=0.001))  # Naive Bayes classifier
])

# Train the model
pipeline_2.fit(X_train, y_train)

product_sales['target_label'] = pipeline_2.predict(product_sales['item_name'])# classify product sales using product description

Data Overview

The key variables will come from one table containin product sales.

#PYTHON CODE BLOCK
import pandas as pd
product_sales.head(6)

Objective 1; Day to day sales trends

#PYTHON CODE BLOCK
# since we are looking at a week of data the lowet granular value is day and for sales intepretation we need the name of day
#convert date time column
from datetime import datetime
product_sales_new['sqldate'] =pd.to_datetime(product_sales_new['sqldate'], format='%Y-%m-%d')
product_sales_new['day'] = product_sales_new['sqldate'].dt.strftime('%d')
product_sales_new['day_name'] = product_sales_new['sqldate'].dt.strftime("%A")#day name

pd.set_option('display.float_format', lambda x: '%.0f' % x)#supress scientific notation

day_to_day = product_sales_new.groupby(['sqldate','day','day_name'])['saleamt'].sum().reset_index()# day to day data

Visualise wrangled in R

# R CODE BLOCK
plot_ly(day_to_day, 
        x = ~sqldate, 
        y = ~saleamt, 
        type = 'scatter', 
        mode = 'lines+markers', 
        hoverinfo = 'text',
        hovertemplate = '<b>%{x}</b>: %{y:.2s} USD<extra></extra>') %>%
  layout(
    hovermode = "x",
    xaxis = list(
      title = list(text = 'Thanksgiving Week November 2021', standoff = 20),
      tickformat = '%A %d %b',
      dtick = 'D1',
      showline = TRUE,
      linecolor = 'black'
    ),
    yaxis = list(
      title = list(text = 'Daily Sales Revenue (USD)', standoff = 20),
      showline = TRUE,
      linecolor = 'black'
    )
  )

View more in dashboard -

Objective 2, 3, and 5 required dats wrangling using Python and visulalisation and integration into siny dashboard in R as illustrated above.

Objective 4; Product association

To determine frequently purchased products, I used a market basket association rule technique with support, confidence, and lift as metrics to quantify the strength of relationships between item sets.

• Support: The support of an item set is the probability of its occurrence out of the total number of transactions. It quantifies the share of transactions containing an item set. For item set A:

• Confidence: For association ( A \rightarrow B ), confidence is the probability of A and B occurring together out of all transactions where A has already occurred:

Given the dataset’s size (200,000 transactions and items), I created a sparse matrix to speed up processing and scaled the data to use the Apriori and FP-Growth algorithms, but the matrix size was too large. I opted to manually calculate support, confidence, and lift for item sets within the data. View more in dashboard -

Objective 6; Difference between Black Friday and Cyber Monday sales

Looking at graphs; Cyber Monday had more commissioned items than Black Friday, Conclusion: Cyber Monday is focused on online sales, which reach a larger audience with extended shopping time; where affiliates can be used to boost online traffic compared to Black Friday which focusses more on in-store shopping

Sales almost doubled for both Black Friday and Cyber Monday, especially in the the United States but Cyber Monday had more sales, Conclusion 1: Retailers often use Black Friday to attract a large crowd with in-store promotions. These promotions could be extended to Cyber Monday,offer exclusive online deals, sometimes with additional discounts which drive up sales Conclusion 2: Canada conversely had less sales on Cyber monday than Black Friday, which could mean that customers still prefer traditional instore deals than online shopping

Business Recommendations

Black Friday and Cyber Monday Sales Spike:

• Increase marketing and promotional sales on these days.
• Ensure that inventory levels are sufficient to meet increased demand, especially for US sales where customer spending almost doubles.
• For Cyber Monday, invest in enhancing the online shopping experience. Ensure that websites can handle increased traffic smoothly, which improves user experience and can increase the conversion rate.

Top Selling Products and Categories:

• Theragun Elite Black, gaming consoles, and Jordan Shoes are top sellers. To attract customers, push to feature these prominently in marketing campaigns.
• Bowflex C6 Bike and Bowflex Cardio Machine Mat are often bought together. We could offer complementary bundle discounts for these products and other similar products.

Data Driven Decision:

• Models are only as good as the data used. In this case, the product_taxanomy still needs more cleaning on the correct cataloguing of items before they can be trained on a model. This will improve the implementation of recommendation tools and product association rules technique to identify items frequently bought-together items and create bundle deals or cross-selling opportunities.

Complete Jupyter Notebook -

• Introduction
  • Data Overview
• Methods & Data Cleaning
  • Text Normalisation
• Model Training & Pipeline
  • TF-IDF Score:
  • Term Frequency (TF):
  • Inverse Document Frequency (IDF):
  • Model Implementation
• Results
  • Final Model
  • Evaluation and Prediction
• PART 2

Introduction

In this project, I developed a deployable machine learning product classification model for a B2B e-commerce company struggling with incomplete category information in their product catalog. The objective was to enhance the accuracy and efficiency of sales product categorization, ultimately improving the user experience on their website. The task involved building a text classification model to predict missing category data based on product descriptions. A significant challenge was dealing with category data in multiple formats, some of which did not align with the Google product taxonomy format. I started by analyzing the dataset to identify and extract rows that conformed to the Google taxonomy, applying natural language processing techniques to clean the text data and enrich the training set. The resulting model was trained to accurately classify new products into the correct categories, improving data consistency and streamlining product management.

Data Overview

The key variables to categorise products will come from one table contain incosistantly categorised products that will be cleaned to created a final dataset for modelling.

import pandas as pd
product_taxanomy = pd.read_csv('ProductDataChallenge_02_ProductTaxonomy.csv')

product_taxanomy.head(6)

Methods & Data Cleaning

The client wants to standadise product categorisation using google taxanomy format. Taxanomy data was downloaded here Firstly l filtered out products not in google taxanomy format and then remove duplicates.

# Clean product taxonomy file match it to google merchant catalogue 
google_catalogue = pd.read_csv('taxonomy-with-ids-template.csv')
# Combine columns into one with ">" separator, stopping at NaN
def combine_levels(row):
    combined = []
    for item in row[1:]:  # Skip the first column
        if pd.isna(item):
            break
        combined.append(str(item))
    return ' > '.join(combined)

google_catalogue['original_format_category'] = google_catalogue.apply(combine_levels, axis=1)
google_catalogue.rename(columns={"1": "id"}, inplace=True)

#combine with product_taxanomy and remove nans for mis classification
new_taxanomy =product_taxanomy.merge(google_catalogue[['id','original_format_category']], on='original_format_category', how='left')

#remove categories not in google catalogue format
new_taxanomy.dropna(subset=["id"],inplace=True)

l removed duplicate products because l assumed that each product belongs to one category, this could affect the prediction of unseen products

# Check for dupllicates
duplicates_2 = new_taxanomy.groupby('item_name').size()
duplicates_2 =duplicates_2[duplicates_2 > 1].sort_values(ascending=False).reset_index()

##remove duplicated items
new_taxanomy = new_taxanomy[~new_taxanomy['item_name'].isin(duplicates_2['item_name'])]

#double check
zz =new_taxanomy.groupby('item_name').size()
zz[zz > 1].sort_values(ascending=False).reset_index()# double check

Text Normalisation

l normalized text before inputting it into a predictive model to ensure consistency and remove noise. This helps the model focus on meaningful patterns, improving accuracy and performance. l preprocessed the data by removing non-alphanumeric characters, stripping whitespace, and converting text to lowercase.

import requests
import pandas as pd
import re
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
import nltk

# Download necessary NLTK resources
nltk.download('punkt')
nltk.download('stopwords')

# Function to clean text
def clean_text(text):
    text = re.sub(r'[^a-zA-Z0-9\s]', '', text) # Remove non-alphanumeric characters
    text =text.strip() #remove white space
    text = text.lower() # Convert text to lowercase
    return text


new_taxanomy['new_name'] = new_taxanomy['item_name'].apply(clean_text)

Model Training & Pipeline

l created a two-level preprocessing and modeling pipeline, employing TfidfVectorizer() for transforming text data into term-frequency inverse document frequency (TF-IDF) vectors and MultinomialNB() for the Naive Bayes classification. Additionally, l utilized GridSearchCV to perform hyperparameter tuning for optimal model parameters. This pipeline ensured efficient text normalization and robust classification performance.

TF-IDF (Term Frequency-Inverse Document Frequency) is a powerful technique in Natural Language Processing used to convert text data into a statistically interpratable numerical feature vector that reflects the significance of a word within a document relative to a collection of documents, known as a corpus. It assigns a numerical value to each word (TF-IDF score) that quantifies the importance of each term (word) in a document (string of product item name description) by considering its frequency in the document and its rarity across multiple documents. The will then be used by machine learning model to categorise the products and hten convert the numerica feature vector back into text

TF-IDF Score:

$\text{TF-IDF}(t, d) = \text{TF}(t, d) \times \text{IDF}(t)$

Term Frequency (TF):

Definition: Measures the frequency of a term in a document.

Formula:

Purpose: Captures the importance of a term within a specific document.

Inverse Document Frequency (IDF):

Definition: Measures the rarity of a term across all documents in the corpus.

Formula:

Purpose: Assigns higher weights to terms that are rare across the corpus, reducing the impact of common terms.

Because of the size of the data…

#count number of unique values
print(f'Number of product categories: {new_taxanomy["target_label"].nunique()} ' ) 
print(f'Number of product items: {new_taxanomy["new_name"].nunique()} ' ) 

Number of product categories: 2871 
Number of product items: 577600 

l used a Multinomial Naive Bayes for product classification because it is both robust (it can handle multidimensional text data) and efficient (it can handle large data and is not computationally intensive). After applying TfidfVectorizer(), which converts text data into numerical TF-IDF vectors, the Multinomial Naive Bayes algorithm becomes more accurate in distinguishing between different product categories based on textual features.

Model Implementation

#THIS CODE TOOK THREE HOURS TO RUN DO NOT RE-RUN
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV, train_test_split

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=0.2, random_state=42)## no need to sub-sample teh data
# Define the pipeline
pipeline_2 = Pipeline([
    ('tfidf', TfidfVectorizer()),
    ('nb', MultinomialNB(alpha=0.0001))
])

# Define the parameter grid for grid search
param_grid = {
    'tfidf__max_features': [200000, 350000, 500000],
    'tfidf__ngram_range': [(1, 1), (1, 2), (1, 3)],
    'nb__alpha': [0.0001, 0.001, 0.01, 0.1]
}

# Initialize GridSearchCV
grid_search = GridSearchCV(pipeline_2, param_grid, cv=3, n_jobs=1, verbose=2)

# Fit the model
grid_search.fit(X_train, y_train)

# Best parameters and best score
print("Best parameters found: ", grid_search.best_params_)
print("Best cross-validated score: ", grid_search.best_score_)

# Evaluate on the test set
accuracy = grid_search.score(X_test, y_test)
print("Test set accuracy: ", accuracy)

Best parameters found:  {'nb__alpha': 0.001, 'tfidf__max_features': 500000, 'tfidf__ngram_range': (1, 2)}
Best cross-validated score:  0.8641511636958045
Test set accuracy:  0.8747221573562354

Results

Final Model

l employed the model with the highest cross-validation and test accuracy

#Best model parameter
pipeline_2 = Pipeline([
    ('tfidf', TfidfVectorizer(max_df=0.75, 
                              min_df=1, max_features=500000, 
                              ngram_range=(1, 2), stop_words='english', 
                               use_idf=True, 
                              smooth_idf=True, 
                              sublinear_tf=True)),
    ('nb', MultinomialNB(alpha=0.001))
])

Evaluation and Prediction

To evaluate our model, l used precision, recall, F1-score, and accuracy. l also calculated macro and micro averages to better understand the model’s performance with unbalanced data. Macro averages treat all classes equally, providing a balanced view across classes, while micro averages consider the frequency of each class, giving a more holistic measure of the overall performance.

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
import pandas as pd

# Predict on the test set
y_pred_2 = pipeline_2.predict(X_test)

# Assuming y_test and y_pred_2 are already defined
accuracy = accuracy_score(y_test, y_pred_2)
precision_macro = precision_score(y_test, y_pred_2, average='macro', zero_division=0)
recall_macro = recall_score(y_test, y_pred_2, average='macro', zero_division=0)
f1_macro = f1_score(y_test, y_pred_2, average='macro', zero_division=0)
precision_micro = precision_score(y_test, y_pred_2, average='micro', zero_division=0)
recall_micro = recall_score(y_test, y_pred_2, average='micro', zero_division=0)
f1_micro = f1_score(y_test, y_pred_2, average='micro', zero_division=0)

# Create a dictionary of the metrics
metrics_data = {
    'Accuracy': [accuracy, accuracy],  # Accuracy is the same for both macro and micro
    'Precision': [precision_macro, precision_micro],
    'Recall': [recall_macro, recall_micro],
    'F1 Score': [f1_macro, f1_micro]
}

# Specify row labels
row_labels = ['Macro Avg', 'Micro Avg']

# Convert dictionary to a DataFrame
metrics_df = pd.DataFrame(metrics_data, index=row_labels)

# Display the table
print(metrics_df)

           Accuracy  Precision  Recall   F1 Score
Macro Avg    0.88      0.62      0.54      0.56
Micro Avg    0.88      0.88      0.88      0.88

The low macro averages indicates that the model’s performance is not consistent across all classes. It suggests that some classes may have lower precision, recall, or F1 scores, potentially due to class imbalance or the model struggling with specific categories.

On the other end high micro average means that the model performs well overall when considering all classes and their frequencies. This reflects strong overall accuracy and suggests that the model correctly identifies a large proportion of instances, even if some individual classes are less well-predicted.

PART 2

Due to high traffic during Black Friday week, the product classification model was used to categorize products. Sales data was analyzed to reveal valuable trends in buying patterns, such as popular products, peak shopping times, and discrepancies between regional sales.

import pandas as pd
product_sales = pd.read_csv('ProductDataChallenge_01_ProductSales_thanksgiving_week.csv')

# Predict categories for unlabeled data
product_sales['target_label'] = pipeline_2.predict(product_sales['item_name'])

An interaction Shiny dashboard was created showcasing Product sales KPIs, trends and business recommendations. For more information click here

Complete Project repository:

• Business understanding & Problem framing
• Exploratory data analysis & data cleaning
  • Data Overview
  • Data Import
  • Descriptive statistics
• Feature engineering & Hypothesis framing
• Modelling and evaluation
  • Model Training
  • Model Evaluation
  • Model understanding
• Insights & Recommendations

Business understanding & Problem framing

Our client PowerCo - a major gas and electricity utility that supplies to small and medium sized enterprises is concerned about their customers leaving for better offers from other energy providers (customer churn). The energy market has had a lot of change in recent years nd there are now more options than ever for customers to choose from. They would like to diagnose the the reason why their customers are churning. One of the leading objective is - How sensitive are PowerCo customers to pricing and how does it influence the customer’s decision stay or leave PowerCo.

Data required;

1. Customer data - historical electricity and gas consumption, date customer joined (subscribed for services)
2. Churn data - indicating if customer churned or not (reclassified as 1 nad 0)
3. Historical and seasonal pricing data - the price the client charges its customers and how it changes with demand.

Work Plan;

In order to test the hypothesis of how pricing affects churn, we defined what data we need from PowerCo, and what variables best define price sensitivity, prepare the data and engineer features. A binary classificantion model was used and trained to extrapolate the extent to which price sensitivity influences churn.

Exploratory data analysis & data cleaning

Data Overview

Data Import

import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np

# Shows plots in jupyter notebook
%matplotlib inline

# Set plot style
sns.set(color_codes=True)

client_df = pd.read_csv('client_data.csv')
price_df = pd.read_csv('price_data.csv')

For modelling purposes we convert columns into date time

client_df['date_activ'] = pd.to_datetime( client_df['date_activ'])
client_df['date_end'] = pd.to_datetime( client_df['date_end'])
client_df['date_modif_prod'] = pd.to_datetime( client_df['date_modif_prod'])
client_df['date_renewal'] = pd.to_datetime( client_df['date_renewal'])
client_df.head(3)

It is useful to first understand the data that you’re dealing with along with the data types of each column and na values in those columns. The data types may dictate how you transform and engineer features.

# Check for column data types
print(client_df.info())
print(price_df.info())

#Check for missing values in columns
print(client_df.isna().sum())
print(price_df.isna().sum())

Descriptive statistics

To visualise the statistics of churned customers, we used function ‘plot_stacked_bars’. The code below shows that close to 10% of PowerCo customers churned in the last three months

churn = client_df[['id', 'churn']]
churn.columns = ['Companies', 'churn']
churn_total = churn.groupby(churn['churn']).count()
churn_percentage = churn_total / churn_total.sum() * 100
plot_stacked_bars(churn_percentage.transpose(), "Churning status", (5, 5), legend_="lower right")

Complete exploratory data analysis script -

Feature engineering & Hypothesis framing

Hypothesis 1: “The difference between off-peak prices in December and January the preceding year could be a significant feature when predicting customer churn”

Load cleaned data set after EDA

df = pd.read_csv('C:/Users/farie/Downloads/clean_data_after_eda.csv')
df["date_activ"] = pd.to_datetime(df["date_activ"], format='%Y-%m-%d')
df["date_end"] = pd.to_datetime(df["date_end"], format='%Y-%m-%d')
df["date_modif_prod"] = pd.to_datetime(df["date_modif_prod"], format='%Y-%m-%d')
df["date_renewal"] = pd.to_datetime(df["date_renewal"], format='%Y-%m-%d')

Calculate the difference between off-peak prices in December and preceding January

price_df = pd.read_csv('price_data.csv')
price_df["price_date"] = pd.to_datetime(price_df["price_date"], format='%Y-%m-%d')
price_df.head()

# Group off-peak prices by companies and month
monthly_price_by_id = price_df.groupby(['id', 'price_date']).agg({'price_off_peak_var': 'mean', 'price_off_peak_fix': 'mean'}).reset_index()

# Get january and december prices
jan_prices = monthly_price_by_id.groupby('id').first().reset_index()
dec_prices = monthly_price_by_id.groupby('id').last().reset_index()

# Calculate the difference
diff = pd.merge(dec_prices.rename(columns={'price_off_peak_var': 'dec_1', 'price_off_peak_fix': 'dec_2'}), jan_prices.drop(columns='price_date'), on='id')
diff['offpeak_diff_dec_january_energy'] = diff['dec_1'] - diff['price_off_peak_var']
diff['offpeak_diff_dec_january_power'] = diff['dec_2'] - diff['price_off_peak_fix']
diff = diff[['id', 'offpeak_diff_dec_january_energy','offpeak_diff_dec_january_power']]

#merger files
df = pd.merge(df, diff, on='id')
df.head()

Hypothesis 2: “Average and max price changes between periods (peak and low season) through out the entire year could be a significant feature when predicting customer churn”

mean_prices = price_df.groupby(['id']).agg({
    'price_off_peak_var': 'mean', 
    'price_peak_var': 'mean', 
    'price_mid_peak_var': 'mean',
    'price_off_peak_fix': 'mean',
    'price_peak_fix': 'mean',
    'price_mid_peak_fix': 'mean'    
}).reset_index()

# Calculate the mean difference between consecutive periods
mean_prices['off_peak_peak_var_mean_diff'] = mean_prices['price_off_peak_var'] - mean_prices['price_peak_var']
mean_prices['peak_mid_peak_var_mean_diff'] = mean_prices['price_peak_var'] - mean_prices['price_mid_peak_var']
mean_prices['off_peak_mid_peak_var_mean_diff'] = mean_prices['price_off_peak_var'] - mean_prices['price_mid_peak_var']
mean_prices['off_peak_peak_fix_mean_diff'] = mean_prices['price_off_peak_fix'] - mean_prices['price_peak_fix']
mean_prices['peak_mid_peak_fix_mean_diff'] = mean_prices['price_peak_fix'] - mean_prices['price_mid_peak_fix']
mean_prices['off_peak_mid_peak_fix_mean_diff'] = mean_prices['price_off_peak_fix'] - mean_prices['price_mid_peak_fix']

columns = [
    'id', 
    'off_peak_peak_var_mean_diff',
    'peak_mid_peak_var_mean_diff', 
    'off_peak_mid_peak_var_mean_diff',
    'off_peak_peak_fix_mean_diff', 
    'peak_mid_peak_fix_mean_diff', 
    'off_peak_mid_peak_fix_mean_diff'
]
df = pd.merge(df, mean_prices[columns], on='id')

Hypothesis 3: “How long a company has been a client of PowerCo (tenure) could be a significant feature when predicting customer churn”

• months_activ = Number of months active until reference date (Jan 2016)
• months_to_end = Number of months of the contract left until reference date (Jan 2016)
• months_modif_prod = Number of months since last modification until reference date (Jan 2016)
• months_renewal = Number of months since last renewal until reference date (Jan 2016)

def convert_months(reference_date, df, column):
    """
    Input a column with timedeltas and return months
    """
    time_delta = reference_date - df[column]
    months = (time_delta / np.timedelta64(1, 'M')).astype(int)
    return months

# Create reference date
reference_date = datetime(2016, 1, 1)

# Create columns
df['months_activ'] = convert_months(reference_date, df, 'date_activ')
df['months_to_end'] = -convert_months(reference_date, df, 'date_end')
df['months_modif_prod'] = convert_months(reference_date, df, 'date_modif_prod')
df['months_renewal'] = convert_months(reference_date, df, 'date_renewal')

Transform categorical data using one hot encoding (dummy variables)

df = pd.get_dummies(df, columns=['channel_sales'], prefix='channel')
df = pd.get_dummies(df, columns=['origin_up'], prefix='origin_up')

Transforming numerical data to remove the effect of skewed distribtion and also improve the speed at which predictive models are able to converge to its best solution. Note: We cannot apply log to a value of 0, so we will add a constant of 1 to all the values

# Apply log10 transformation
df["cons_12m"] = np.log10(df["cons_12m"] + 1)
df["cons_gas_12m"] = np.log10(df["cons_gas_12m"] + 1)
df["cons_last_month"] = np.log10(df["cons_last_month"] + 1)
df["forecast_cons_12m"] = np.log10(df["forecast_cons_12m"] + 1)
df["forecast_cons_year"] = np.log10(df["forecast_cons_year"] + 1)
df["forecast_meter_rent_12m"] = np.log10(df["forecast_meter_rent_12m"] + 1)
df["imp_cons"] = np.log10(df["imp_cons"] + 1)

Complete feature engineering script -

Modelling and evaluation

###Import packages for modelling

import warnings
warnings.filterwarnings("ignore", category=FutureWarning)

from sklearn import metrics
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

Splitting the dataset into training and test samples

# Make a copy of our data
train_df = df.copy()

# Separate target variable from independent variables
y = df['churn']
X = df.drop(columns=['id', 'churn'])
print(X.shape)
print(y.shape)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)

Model Training

• We use a Random Forest Classifier whixh is an ensemble algorithm
• We add a randomsearch CV which helps fine tune the optimal parameters
• To make random forest more suitable for learning from extremely imbalanced data (only 9% of churned customers in dataset) follows the idea of cost sensitive learning. Since the RF classifier tends to be biased towards the majority class, we placed a heavier penalty on misclassifying the minority class (class_weight = ‘balanced’)

rf = RandomForestClassifier(random_state = 42)#run model

from pprint import pprint #to print parameters used by model
# Look at parameters used by our current forest
print('Parameters currently in use:\n')
pprint(rf.get_params())

#select the optimal parameters
from scipy.stats import randint

rs_space={'max_depth':list(np.arange(10, 100, step=10)) + [None],
              'n_estimators':np.arange(10, 1000, step=50),
              'max_features':randint(1,7),
              'criterion':['gini','entropy'],
              'min_samples_leaf':randint(1,4),
              'min_samples_split':np.arange(2, 10, step=2)
         }


from sklearn.model_selection import RandomizedSearchCV

rf = RandomForestClassifier(random_state = 42, class_weight="balanced_subsample")#run model, but we add class weight because there ins an imbalance in values between class of independent variable
rf_random = RandomizedSearchCV(rf, rs_space, n_iter=500, scoring='accuracy', n_jobs=-1, cv=3)
model_random = rf_random.fit(X_train,y_train)

Model Evaluation

#check model performance
print("tuned hpyerparameters :(best parameters) ",model_random.best_params_)
print("accuracy :",model_random.best_score_)

##evaluate model with test data
evaluator(model_random,X_test,y_test)
y_pred = model_random.predict(X_test)
plot_confusion_matrix(y_test, y_pred)

True positives: 21
False positives: 5
True negatives: 3281
False negatives: 345

Accuracy: 90%
Precision: 82%
Recall: 5%

Looking at these results there are a few things to point out:

• Within the test set about 10% of the rows are churners (churn = 1).
• Looking at the true negatives, we have 3281 out of 3286. This means that out of all the negative cases (churn = 0), we predicted 3282 as negative (hence the name True negative). This is great!
• Looking at the false negatives, this is where we have predicted a client to not churn (churn = 0) when in fact they did churn (churn = 1). This number is quite high at 345, we want to get the false negatives to as close to 0 as we can, so this would need to be addressed when improving the model.
• Looking at false positives, this is where we have predicted a client to churn when they actually didnt churn. For this value we can see there are 5 cases, which is great!
• With the true positives, we can see that in total we have 366 clients that churned in the test dataset. However, we are only able to correctly identify 21 of those 366, which is very poor.
• Looking at the accuracy score, this is very misleading! Hence the use of precision and recall is important. The accuracy score is high, but it does not tell us the whole story.
• Looking at the precision score, this shows us a score of 0.82 which is not bad, but could be improved.
• However, the recall shows us that the classifier has a very poor ability to identify positive samples. This would be the main concern for improving this model!

So overall, we’re able to very accurately identify clients that do not churn, but we are not able to predict cases where clients do churn! What we are seeing is that a high % of clients are being identified as not churning when they should be identified as churning. This in turn tells me that the current set of features are not discriminative enough to clearly distinguish between churners and non-churners.

Model understanding

A simple way of understanding the results of a model is to look at feature importances. Feature importances indicate the importance of a feature within the predictive model, there are several ways to calculate feature importance, but with the Random Forest classifier, we’re able to extract feature importances using the built-in method on the trained model. In the Random Forest case, the feature importance represents the number of times each feature is used for splitting across all trees.

feature_importances = pd.DataFrame({
    'features': X_train.columns,
    'importance': model.best_estimator_.feature_importances_
}).sort_values(by='importance', ascending=True).reset_index()

plt.figure(figsize=(15, 25))
plt.title('Feature Importances')
plt.barh(range(len(feature_importances)), feature_importances['importance'], color='b', align='center')
plt.yticks(range(len(feature_importances)), feature_importances['features'])
plt.xlabel('Importance')
plt.show()

Complete modelling and evaluation script -

Insights & Recommendations

Churn is high in SME with 9.7% across 14606 customers for PowerCo The model was able to predict churn but Yearly consumption, forecasted consumption and net margin were important. However the price sensitivity features are scattered around and were not a main driver for churn. More modelling, fine tuning of features and inclusion of more variables is required to potentially improve the model.

A discount strategy of 20% would be effective but only if targeted appropriately. We recommended offering discount to only to high value customers with high churn probability.

Failure is an option here. If things are not failing, you are not innovating enough ~ Elon Musk.

• Project Overview
  • Problem Statement
• Methods
  • Model Assesment
• Results
  • Models Perfomance
• Discussion, Growth & Next Steps

Project Overview

Problem Statement

Going to space has always been a very expensive venture and SpaceX advertises Falcon 9 rocket launches on its website with a cost of 62 million dollars while other providers cost upward of 165 million dollars each, much of the savings is because since 2014 Space X has manage to successfully recover first stage rocket launched vehicles. This has made Space X rockets comparatively cheaper because their rockets are reusable. Although the landing success rate has improved over the years, not all rocket have landed successfully. Space X has been collecting data on its landing activities over the years, therefore if we can determine if the first stage will land, we can determine the cost of a launch. This information can be used if an alternate company wants to bid against SpaceX for a rocket launch.

Methods

We will be predicting the landing success of Falcon 9 rocket launches and Falcon heavy launches from 2010 to 2021. Data was collected by webscraping (Beautifulsoup) tables of SpaceX historical launch records from a Wikipedia page. .

More data on rocket landing outcomes were extracted using SpaceX resting APIs .

Exploratory data analysis and preparatory feature engineering were implemented on the collected data to also visiualise landing outcome of rocket launches using Pandas and Matplotlib.Machine learning models typically cannot process string variables, so when there are text data within a data set, we would encode them (change them to numbers, typically binary) so that the algorithm can assess the relationship between the variable and the output. We used One Hot Encoding (OHE) from the sklearn.preprocessing library. .

The detailed landing outcomes of the rockets were reclassified to 1 (landed) and 0 (did not land) for analysis, missing values and outliers were removed {add data wrangling notebook}.

We then implemented four classification machine learning models, Logistic regression, Support vector machine, Decision tree and K-nearest neighbor. Before modelling all variables were standardised using a StandardScaler so that variables measured at different scales may be reduced to the same scale and contribute equally to model fitting, with landing outcome as the dependant variable.

The hyperparameters for each model were optimised using GridSearchCV, it evaluates the model for each combination of hyperparameters using the cross-validation method (cv =10) and chooses the combination with the best performance for each model before fitting.

The data was split into training and testing data, and a confusion matrix was created to show how the trained models can accurately predict the testing data (show false positve vs false negative). .

Model Assesment

Model perfomance and comparison were determined using accuracy score, precision score, recall score and F1 score.

• Accuracy: This metric meassures how a machine learning model correctly predicts the outcome values in the test data (new data) by dividing the number of correct predictions by the total number of predictions. The accuracy score is measured on a scale of 0 to 1 or as a percentage. The higher the accuracy, the better.

• Precision: This metric measures how a machine learning model correctly predicts the positive outcome values (true postives) in a test data (in this case if the rocket landed successfully). Answers the question: of all observations predicted as positive, what proportion was actually positive?. The accuracy score is measured on a scale of 0 to 1 or as a percentage. The higher the accuracy, the better.

• Recall: This metric measures how often a machine learning model correctly identifies positive instances (true positives) from all the actual positive samples in a test data (in this case if the rocket landed successfully). Answers the question: of all positive observations, how many did we predict as positive? The accuracy score is measured on a scale of 0 to 1 or as a percentage. The higher the accuracy, the better.

• F1 Score: This is the harmonic mean of Precision and Recall. A good F1-score comes when there is a balance between precision and recall.

Results

The goals of the project was:

• Build a model that would accurately predict the landing outcome of rockets (success/fail) . If we can determine if the first stage will land, we can determine the cost of a launch.
• This information can be used if an alternate company wants to bid against SpaceX for a rocket launch.

The landing success of Space X rockets has continued to increase since 2013.

Trend in landing success rate of Space X Falcon 9 rockets since 2010, success rate has had a positive upward trend since 2013.

Models Perfomance

Based upon these,all models can predict the landing success with an accuracy of ~80 across all matrics except for Decision Trees which perfomed lower but not singifcantly so

Model prediction accuracy for all built classification models, in a bar chart.

Confusion Matrix

Discussion, Growth & Next Steps

The way we have coded this up is very much for the “proof of concept”. In practice, we would definitely have to use a large data set and incooperate more variables into the model that could improve the predictions. Payloadmass of the rockets was an influential contributer to hte predictions of this model, (in practice we will have to remove and add some variables to see if htey affect the data). In this case removing payloadmass had a significant effect on the predictions, which hints at other factors being added to hte rocket influencing landing success of rockets. Alose inplace of Decision trees we can incooperate a random forest which offer a more complex visualisation option (nodes of classification). the last section of the code (where we submit a search) isolated, and running from all of the saved objects that we need - we wouldn’t include it in a single script like we have here.

This was tested only in two categories (success or fail), we would want to test on a broader array of categories - most likely having a saved network for more specific conditions that describe and categorise failure and successful landing of rockets in more detail.

• Project Overview and Results
• Data Overview
• Data Import
• Preprocessing
  • Missing Values
  • Outliers
• Split Out Data for Modelling
• Model 1: Logistic Regression
  • Encoding Categorical Variables
  • Feature Selection
  • Model Training
  • Model Assessment
    • Predict On The Test Set
    • Confusion Matrix
    • Classification Performance Metrics
  • Finding The Optimal Classification Threshold
• Model 2: Decision Trees
  • Encode Categorical Variables
  • Model Training
  • Model Assessment
  • Plot Decision Tree
  • Finding the Best Max Depth to Prevent Overfitting
• Model 3: Random Forest (Ensemble)
• Model 4: K Nearest Neighbors (KNN)
• Results
  • Key Drivers of Delivery Club Signups
• Application
• Growth & Next Steps

Project Overview and Results

Problem Statement: Our client, a grocery retailer, sent out mailers in a marketing campaign for their new delivery club. This cost customers $100 per year for membership, and offered free grocery deliveries, rather than the normal cost of $10 per delivery. For this, they sent mailers to their entire customer base (apart from a control group) but this proved expensive. For the next batch of communications, they would like to save costs by only mailing customers that were likely to sign up. Based on the results of the last campaign and the customer data available, we will look to understand the probability of customers signing up for the delivery club. This would allow the client to mail a more targeted selection of customers, lowering costs, and improving ROI.

The goal of this project is to build a model that would accurately predict the customers that would sign up for our client’s delivery club services. This would allow for a much more targeted approach when running the next iteration of the delivery club campaign. A secondary goal is to understand what the drivers for delivery club signups are, so the client can get closer to the customers that need or want this service, and enhance their messaging.

The data was cleaned, preprocessed and split into training and testing sets; four different models were used in the modelling process and based on the goals of the project, the model of choice is the Random Forest model as it was:

• the most consistent performant on the test set across classification accuracy, precision, recall, and F1-score, and
• the feature importance and permutation importance allow the client an understanding of the key drivers behind delivery club signups. See the table below for a summary of all model results.
  
  

Data Overview

We will be predicting the binary signup flag from the campaign_data table in the client database. The key variables to predict this will come from different tables, which we have joined and aggregated to come up with the final dataset. We focused on aggregating data 3 months before the last campaign date.

The final table of data for modelling looks something like this:

Data Import

The first step is to import important packages such as pickle (we saved our final table as a pickle file), pandas, matplotlib etc.

# import required packages
import pandas as pd
import pickle
import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.utils import shuffle
from sklearn.model_selection import train_test_split, cross_val_score, KFold
from sklearn.metrics import confusion_matrix, accuracy_score, precision_score, recall_score, f1_score
from sklearn.preprocessing import OneHotEncoder
from sklearn.feature_selection import RFECV
import plotly.express as px

# import modelling data
data_for_model = pickle.load(open("data/delivery_club_modelling.p", "rb"))

For modelling purposes, the customer_id does not add value to our model, so we remove it from the dataset. It is also necessary to shuffle our data to introduce randomness into the dataset (if it isn’t randomized already).

# drop uneccessary columns
data_for_model.drop("customer_id", axis = 1, inplace = True)

# shuffle data
data_for_model = shuffle(data_for_model, random_state = 42)

Since this is a classification problem, it is a good idea to understand the class balance in the dataset. This would affect what metrics would be most useful in assessing our model.

# assess class balance of dependent variable
data_for_model["signup_flag"].value_counts(normalize = True)

The result of the code above tells us that 69% of customers did not sign up and 31% did. This tells us that while the data isn’t perfectly balanced at 50:50, it isn’t too imbalanced either. Because of this, and as you will see, we make sure to not rely on classification accuracy alone when assessing results - we also analyze Precision, Recall, and F1-Score to give us a complete picture.

Preprocessing

Missing Values

The number of missing values in the data was extremely low (<1%), so instead of applying any imputation (i.e. mean, most common value), we will just remove those rows.

# remove rows where values are missing
data_for_model.isna().sum()
data_for_model.dropna(how = "any", inplace = True)

Outliers

Outliers investigation/removal is an important step in preprocessing data as outliers could hamper the ability of models to generalize. Let’s use the describe() method to investigate the spread of values for our inputs.

Another fun way of visualizing the spread of data is using a box plot to visually check where there are obvious outliers.

data_num = data.select_dtypes('number')
data_num.drop(["signup_flag"], axis = 1, inplace = True)
data_num.plot(kind= 'box', subplots= True, layout = (3, 3), sharex= False, sharey= False, figsize =(16, 10) )
plt.show()

From the boxplots above, outliers seem to exist in all columns except for credit score and product area count. To remove these outliers, we will use the “boxplot approach” where we remove any rows where the values are outside of 2 x interquartile range.

outlier_investigation = data.describe()
outlier_columns = ['distance_from_store', 'total_sales', 'total_items', 'transaction_count','average_basket_value']

for column in outlier_columns:
    lower_quartile = data[column].quantile(0.25)
    upper_quartile = data[column].quantile(0.75)
    iqr = upper_quartile - lower_quartile
    iqr_extended = iqr * 2
    min_border = lower_quartile - iqr_extended
    max_border = upper_quartile + iqr_extended
    
    outliers = data[(data[column] < min_border)|(data[column] > max_border)].index
    print(f"{len(outliers)} outliers detected in column {column}")

    data_for_model = data.drop(outliers)
    

Split Out Data for Modelling

Now that we have cleaned our data, we can split into X (input) and y(output) variables, and then split into train and test sets to ensure that we can validate the accuracy of the predictions on data not used in the training.

X = data_for_model.drop("signup_flag", axis = 1)
y = data_for_model.signup_flag
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 23, stratify = y)

In the code above, we have elected to use 80% of our data for training (and use 20% for assessing the accuracy of the model). The argument stratify=y is to ensure that both our training and test sets have the same proportion of customers who did and did not sign up for delivery club - ensuring a robust dataset both for training and testing the model.

Model 1: Logistic Regression

Encoding Categorical Variables

Machine learning models typically cannot process string variables, so when there are text data within a data set, we would encode them (change them to numbers, typically binary) so that the algorithm can assess the relationship between the variable and the output. We could use One Hot Encoding (OHE) or Label Binarizer from the sklearn.preprocessing library.

An extremely efficient way of encoding categorical variables and scaling the inputs is by using DataFrameMapper from sklearn.pandas library. All the variables are scaled in one step.

# Instantiate the mapper
mapper = DataFrameMapper([
    (['distance_from_store'], StandardScaler()), 
      ('gender', LabelBinarizer()), 
      (['credit_score'], StandardScaler()),
      (['total_sales'],StandardScaler()),
       (['total_items'], StandardScaler()),
      (['transaction_count'],StandardScaler()), 
      (['product_area_count'],StandardScaler()), 
       (['average_basket_value'], StandardScaler())], df_out = True)

# Fit the mapper to training dataset only  
mapper.fit(X_train)

# Transform both train and test sets
Z_train = mapper.transform(X_train)
Z_test = mapper.transform(X_test)

Feature Selection

Feature Selection is the process used to select the input variables that are most important to your Machine Learning task. It can be a very important addition or at least, a consideration, in certain scenarios. The potential benefits of Feature Selection are:

• Improved Model Accuracy - eliminating noise can help true relationships stand out
• Lower Computational Cost - our model becomes faster to train, and faster to make predictions
• Explainability - understanding & explaining outputs for stakeholders & customers becomes much easier

Feature selection can either be done by common sense - common sense feature selection can be as simple as taking out variables we know would absolutely not contribute to the model - like customer_id which we took out in the first few steps. We can also use tools like Correlation Matrix showing variable relationships, or Univariate Testing which helps us understand statistical relationships between variables or even more powerful approaches like Recursive Feature Elimination (RFE) which is an approach that starts with all input variables, and then iteratively removes those with the weakest relationships with the output variable.

For our task, we applied a variation of Recursive Feature Elimination called Recursive Feature Elimination With Cross Validation (RFECV) where we split the data into many “chunks” and iteratively trains & validates models on each “chunk” separately. This means that each time we assess different models with different variables included, or eliminated, the algorithm also knows how accurate each of those models was. From the suite of model scenarios that are created, the algorithm can determine which provided the best accuracy, and thus can infer the best set of input variables to use!

# Instantiate RFECV and the model to be used
clf = LogisticRegression(random_state = 43, max_iter = 1000)
feature_selector = RFECV(clf)

# Fit RFECV to training data
fit = feature_selector.fit(Z_train, y_train)

# Extract and print the optimal no of features
optimal_feature_count = feature_selector.n_features_
print(f"Optimal no of features is {optimal_feature_count}")

#limit train and test data to only include selected variables 
Z_train = Z_train.loc[:, feature_selector.get_support()]
Z_test = Z_test.loc[:, feature_selector.get_support()]

The below code then produces a plot that visualizes the cross-validated classification accuracy with each potential number of features

plt.style.use('seaborn-poster')
plt.plot(range(1, len(fit.cv_results_['mean_test_score']) + 1), fit.cv_results_['mean_test_score'], marker = "o")
plt.ylabel("Classification Accuracy")
plt.xlabel("Number of Features")
plt.title(f"Feature Selection using RFECV \n Optimal number of features is {optimal_feature_count} (at score of {round(max(fit.cv_results_['mean_test_score']),4)})")
plt.tight_layout()
plt.show()

This creates the plot below, which shows us that the highest cross-validated classification accuracy (0.904) is when we include seven of our original input variables. The variable that has been dropped is total_sales but from the chart, we can see that the difference is negligible. However, we will continue with the selected seven!

[]!(/assets/img/blog/enhance-target-accuracy/log-reg-feature-selection-plot.png)

Model Training

Instantiating and training our Logistic Regression model is done using the code below. We use the random_state parameter to ensure reproducible results, meaning any refinements can be compared to past results. We also specify max_iter = 1000 to allow the solver more attempts at finding an optimal regression line, as the default value of 100 was not enough.

# instantiate our model object
clf = LogisticRegression(random_state = 42, max_iter = 1000)

# fit our model using our training & test sets
clf.fit(X_train, y_train)

Model Assessment

Predict On The Test Set

To assess how well our model is predicting on new data - we use the trained model object (here called clf) and ask it to predict the signup_flag variable for the test set.

In the code below we create one object to hold the binary 1/0 predictions, and another to hold the actual prediction probabilities for the positive class.

# predict on the test set
y_pred_class = clf.predict(X_test)
y_pred_prob = clf.predict_proba(X_test)[:,1]

Confusion Matrix

A Confusion Matrix provides us with a visual way to understand how our predictions match up against the actual values for those test set observations.

The below code creates the Confusion Matrix using the confusion_matrix functionality from within scikit-learn and then plots it using matplotlib.

# create the confusion matrix
conf_matrix = confusion_matrix(y_test, y_pred_class)

# plot the confusion matrix
plt.style.use("seaborn-poster")
plt.matshow(conf_matrix, cmap = "coolwarm")
plt.gca().xaxis.tick_bottom()
plt.title("Confusion Matrix")
plt.ylabel("Actual Class")
plt.xlabel("Predicted Class")
for (i, j), corr_value in np.ndenumerate(conf_matrix):
    plt.text(j, i, corr_value, ha = "center", va = "center", fontsize = 20)
plt.show()

The aim is to have a high proportion of observations falling into the top left cell (predicted non-signup and actual non-signup) and the bottom right cell (predicted signup and actual signup).

Since the proportion of signups in our data was around 30:70 we will next analyze not only Classification Accuracy, but also Precision, Recall, and F1-Score which will help us assess how well our model has performed in reality.

Classification Performance Metrics

• Accuracy: A very intuitive metric, the accuracy measures the number of correct classification predictions out of all predictions made. This number could be very misleading when you have very imbalanced data. An example of imbalanced data is a dataset where we have a 98%/2% split across 2 classes (98% of patients without a disease and 2% with a disease). If our model predicts no one has the disease, we get an accuracy of 98% which wouldn’t be a great model in the real world.
• Precision: Answers the question: of all observations predicted as positive, what proportion was actually positive?
• Recall: Answers the question: of all positive observations, how many did we predict as positive?
  Usually, trying to improve Precision has the opposite effect on Recall and vice-versa. So what metric to focus on is purely based on the project goals.
• F1-Score: This is the harmonic mean of Precision and Recall. A good F1-score comes when there is a balance between precision and recall.

In the code below, we utilize in-built functionality from scikit-learn to calculate these four metrics.

# classification accuracy
accuracy_lr = round(accuracy_score(y_test, y_pred_class)*100, 2)
print(f"Accuracy score is {accuracy_lr}, meaning the model coorectly predicts {accuracy_lr}% of the test set observations")

# precision
precision_lr = round(precision_score(y_test, y_pred_class)*100, 2)
print(f"Precision score is {precision_lr}, meaning for our predicted delivery club signups, the model was correct {precision_lr}% of the time")

# recall
recall_lr = round(recall_score(y_test, y_pred_class)*100, 2)
print(f"Recall score is {recall_lr}, meaning of allactual delivery club signups, the model correctly predicts {recall_lr}% of the time")

# f1-score
f1_lr = round(f1_score(y_test, y_pred_class), 2)
print(f"F1 score is {f1_lr}")

Finding The Optimal Classification Threshold

By default, most pre-built classification models & algorithms will just use a 50% probability to discern between a positive class prediction (delivery club signup) and a negative class prediction (delivery club non-signup).

Just because 50% is the default threshold does not mean it is the best one for our task.

Here, we will test many potential classification thresholds, and plot the Precision, Recall & F1-Score, and find an optimal solution!

# set up the list of thresholds to loop through
thresholds = np.arange(0, 1, 0.01)

# create empty lists to append the results to
precision_scores = []
recall_scores = []
f1_scores = []

# loop through each threshold - fit the model - append the results
for threshold in thresholds:
    
    pred_class = (y_pred_prob >= threshold) * 1
    
    precision = precision_score(y_test, pred_class, zero_division = 0)
    precision_scores.append(precision)
    
    recall = recall_score(y_test, pred_class)
    recall_scores.append(recall)
    
    f1 = f1_score(y_test, pred_class)
    f1_scores.append(f1)
    
# extract the optimal f1-score (and it's index)
max_f1 = max(f1_scores)
max_f1_idx = f1_scores.index(max_f1)



# plot the results
plt.style.use("seaborn-poster")
plt.plot(thresholds, precision_scores, label = "Precision", linestyle = "--")
plt.plot(thresholds, recall_scores, label = "Recall", linestyle = "--")
plt.plot(thresholds, f1_scores, label = "F1", linewidth = 5)
plt.title(f"Finding the Optimal Threshold for Classification Model \n Max F1: {round(max_f1,2)} (Threshold = {round(thresholds[max_f1_idx],2)})")
plt.xlabel("Threshold")
plt.ylabel("Assessment Score")
plt.legend(loc = "lower left")
plt.tight_layout()
plt.show()

Along the x-axis of the above plot we have the different classification thresholds that were testing. Along the y-axis we have the performance score for each of our three metrics. As per the legend, we have Precision as a blue dotted line, Recall as an orange dotted line, and F1-Score as a thick green line. You can see the interesting “zero-sum” relationship between Precision & Recall and you can see that the point where Precision & Recall meet is where F1-Score is maximised.

As you can see at the top of the plot, the optimal F1-Score for this model 0.78 and this is obtained at a classification threshold of 0.44. This is higher than the F1-Score of 0.734 that we achieved at the default classification threshold of 0.50!

Model 2: Decision Trees

The same steps in Logistic Regression will be used for the Decision Trees - with some minor differences.

Encode Categorical Variables

Using the same method as in Logistic Regression, we encode the gender variable using DataFrameMapper and LabelBinzrizer. Since this is a decision tree, we do not need to scale the numerical variables, so None is an argument in the mapper, indicating that no transformation is to be done on the numerical variables.

mapper_dt = DataFrameMapper([
    (['distance_from_store'], None), 
      ('gender', LabelBinarizer()), 
      (['credit_score'], None),
      (['total_sales'],None),
       (['total_items'], None),
      (['transaction_count'],None), 
      (['product_area_count'],None), 
       (['average_basket_value'], None)], df_out = True)

mapper_dt.fit(X_train)

Z_train_dt = mapper_dt.transform (X_train)
Z_test_dt = mapper_dt.transform(X_test)

Model Training

The code below instantiates the Decision Tree model with a random state specified for reproducibility and a maximum depth to prevent over-fitting.

# Instantiate our model object
dt = DecisionTreeClassifier(random_state = 42, max_depth = 5)

Model Assessment

As before, we would generate the predicted class on the test data, and calculate the accuracy, precision, recall and F1-score.

# create the confusion matrix
conf_matrix = confusion_matrix(y_test, y_pred_class_dt)

# plot the confusion matrix
plt.style.use("seaborn-poster")
plt.matshow(conf_matrix, cmap = "coolwarm")
plt.gca().xaxis.tick_bottom()
plt.title("Confusion Matrix")
plt.ylabel("Actual Class")
plt.xlabel("Predicted Class")
for (i, j), corr_value in np.ndenumerate(conf_matrix):
    plt.text(j, i, corr_value, ha = "center", va = "center", fontsize = 20)
plt.show()

accuracy_dt = round(accuracy_score(y_test, y_pred_class_dt)*100, 2)
print(f"Accuracy score is {accuracy_dt}, meaning the model correctly predicts {accuracy_dt}% of the test set observations")

# precision
precision_dt =  round(precision_score(y_test, y_pred_class_dt)*100, 2)
print(f"Precision score is {precision_dt}, meaning for our predicted delivery club signups, the model was correct {precision_dt}% of the time")

# recall
recall_dt= round(recall_score(y_test, y_pred_class_dt)*100, 2)
print(f"Recall score is {recall_dt}, meaning of actual delivery club signups, the model correctly predicts {recall_dt}% of the time")

# f1-score
f1_dt = round(f1_score(y_test, y_pred_class_dt), 2)
print(f"F1 score is {f1_dt}")

Plot Decision Tree

For Decision Trees, we can literally plot out the Decision Tree which shows us how the splits were made.

plt.figure(figsize=(25,15))
tree = plot_tree(dt,
                 feature_names = X.columns,
                 filled = True,
                 rounded = True,
                 fontsize = 16)

Finding the Best Max Depth to Prevent Overfitting

max_depth_list = list(range(1,20))
accuracy_scores = []

# loop through each possible depth, train and validate model, append test set f1-score
for depth in max_depth_list:
    
    dt = DecisionTreeClassifier(max_depth = depth, random_state = 42)
    dt.fit(Z_train_dt,y_train)
    y_pred_dt = dt.predict(Z_test_dt)
    accuracy = f1_score(y_test,y_pred_dt)
    accuracy_scores.append(accuracy)
    
# store max accuracy, and optimal depth    
max_accuracy = max(accuracy_scores)
max_accuracy_idx = accuracy_scores.index(max_accuracy)
optimal_depth = max_depth_list[max_accuracy_idx]

df = pd.DataFrame({'max_depth_list': max_depth_list, 'accuracy_scores': accuracy_scores})
                   
# plot accuracy by max depth
px.line(df, x = "max_depth_list", y = "accuracy_scores", 
        title = f"Accuracy (F1 score) by Max Depth \n Optimal Tree Depth: {optimal_depth} (F1 Score: {round(max_accuracy, 4)})")
           

The plot above shows us that the maximum F1-score on the test set is found when applying a maximum depth value of 5.

Model 3: Random Forest (Ensemble)

The same process of instantiating, fitting and assessing the model is followed. For code, go to the Jupyter Notebook

Model 4: K Nearest Neighbors (KNN)

The same process of instantiating, fitting and assessing the model is followed. For code, go to the Jupyter Notebook

Results

The goals of the project were:

• Build a model that would accurately predict the customers that would sign up for the delivery club. This would allow for a much more targeted approach when running the next iteration of the campaign.
• Understand what the drivers for this are, so the client can get closer to the customers that need or want this service, and enhance their messaging.

Based upon these, the chosen model is the Random Forest as it was:

• the most consistently performant on the test set across classification accuracy, precision, recall, and F1-score, and
• the feature importance and permutation importance allow the client an understanding of the key drivers behind delivery club signups. See the table below for a summary of all model results.
  
  
  
  

Key Drivers of Delivery Club Signups

As well, the feature importances and permutation importances of the various input variables are shown below. This shows the key drivers behind the delivery club signups, which in this case are: Distance from Store and Transaction Count.

Application

We now have a model object, and the required pre-processing steps to use this model for the next delivery club campaign. When this is ready to launch we can aggregate the necessary customer information and pass it through, obtaining predicted probabilities for each customer signing up.

Based upon this, we can work with the client to discuss where their budget can stretch to, and contact only the customers with a high propensity to join. This will drastically reduce marketing costs, and result in a much-improved ROI.

Growth & Next Steps

While predictive accuracy was relatively high - other modelling approaches could be tested, especially those somewhat similar to Random Forest, for example, XGBoost and LightGBM to see if even more accuracy could be gained.

We could even look to tune the hyperparameters of the Random Forest, notably regularisation parameters such as tree depth, as well as potentially training on a higher number of Decision Trees in the Random Forest.

From a data point of view, further variables could be collected, and further feature engineering could be undertaken.