Data Science Learning Activity - 2nd Semester

Welcome to the Data Science 2nd Semester Learning Activity repository. This project focuses on hands-on practical learning for data wrangling, missing data imputation, and exploratory data analysis using Python and the pandas library, using real-world-style datasets focused on Nigeria.

---

📂 Project Structure

This repository contains the following datasets and files:

• First Live Session - Introduction to Data Wrangling.ipynb: The primary Jupyter Notebook containing live session exercises, code examples, and step-by-step workflows for data cleaning and wrangling.
• Second Live Session - Data Wrangling & Filtering.ipynb: Jupyter Notebook containing exercises and workflows for data filtering and subsetting.
• nigeria_unemployment_missing_data.csv: A dataset capturing unemployment characteristics in Nigeria (Age, Gender, Region, Education Level, Employment Status, Years of Experience, Monthly Income). This dataset contains intentionally introduced missing values (nulls) to serve as a practical playground for data cleaning.
• nigeria_economic_data.csv: Socio-economic indicators of individuals in Nigeria (including regions, monthly income in NGN, occupation, employment type, and poverty status classification).
• nigeria_nursing_mothers_healthcare.csv: Indicators of maternal and child healthcare services across Nigeria (geopolitical zone, residence type, wealth quintile, skilled birth attendance, facility delivery, exclusive breastfeeding, immunization, and distance to the nearest healthcare facility).

---

🍳 Data Wrangling Workflow & Activities

The data wrangling activities are detailed in the First Live Session - Introduction to Data Wrangling.ipynb notebook and follow a structured methodology:

1. Conceptual Analogy: The Data Kitchen & Chef

The notebook starts with the intuition of a Data Kitchen:

• Ingredients correspond to the columns and features in the dataset.
• Preparation (Peeling, Washing, Slicing) corresponds to the data preprocessing steps needed before modeling or visualization.
• Rotten ingredients represent missing or invalid values that must be handled.

2. Loading the Data

Importing the pandas library and loading the unemployment data for inspection:

import pandas as pd

# Load the dataset
unemployment_data = pd.read_csv('nigeria_unemployment_missing_data.csv')
unemployment_data.head()

3. Data Inspection (Detecting "Rot")

Before applying any corrections, the dataset is audited for shape and missing values:

# Check total records and columns
unemployment_data.shape

# Identify count of missing values per column
unemployment_data.isnull().sum()

4. Handling Missing Values (Wrangling Strategies)

Three main strategies are practiced in the session:

🗑️ Strategy A: Dropping Missing Values (Complete-Case Analysis)

Useful when the missing data is minimal or when we want a baseline dataset with only fully recorded observations:

unemployment_data_1 = unemployment_data.copy()
unemployment_data_1.dropna(inplace=True)

# Compare shapes to see how many rows were dropped
print(unemployment_data.shape)
print(unemployment_data_1.shape)

🔢 Strategy B: Numerical Mean Imputation

For numerical columns with missing values where dropping rows would lose too much data (e.g., Years_Of_Experience), we compute the average and impute:

# Calculate average years of experience
average_experience = unemployment_data['Years_Of_Experience'].mean()

# Fill missing values in a copy of the dataframe
unemployment_data_avg_years = unemployment_data.copy()
unemployment_data_avg_years['Years_Of_Experience'].fillna(average_experience, inplace=True)

🏷️ Strategy C: Categorical Placeholder Imputation

For categorical columns (e.g., Education_Level), we inspect the proportions and impute missing values with a designated placeholder like 'Unknown':

# Check proportional distribution of categories
unemployment_data_avg_years.Education_Level.value_counts(normalize=True)

# Replace float('nan')/nulls with 'Unknown'
unemployment_data_avg_years['Education_Level'] = unemployment_data_avg_years['Education_Level'].fillna('Unknown')

# Re-inspect categories
unemployment_data_avg_years['Education_Level'].value_counts()

---

🔍 Data Filtering & Anomalies Workflow & Activities

The data filtering activities are detailed in the Second Live Session - Data Wrangling & Filtering.ipynb notebook and focus on targeted subsetting, error detection, and cleaning:

1. Conceptual Analogy: The Nightclub Bouncer

Filtering acts as the bouncer in a nightclub for three main reasons:

• Scope & Focus: Keeps the ML models focused on relevant data only.
• Removing Logical Impossibilities: Catches human or machine entry errors by setting logical boundaries.
• Targeted Slicing: Carves out exact slices of interest.

2. Loading the Data

Importing the pandas library and loading the economic dataset:

import pandas as pd

# Load dataset
economic_data = pd.read_csv('nigeria_economic_data.csv')
economic_data.head(2)

3. Filtering Activities

🛑 Activity 1: Filtering Anomalies

Calculating the average income of working-age adults (Age between 18 and 65) to help set a new minimum wage:

# Keep only rows where Age is between 18 and 65 (inclusive)
adult_only_eco_data = economic_data[(economic_data['Age'] >= 18) & (economic_data['Age'] <= 65)]

print(f"Original Dataset Size: {len(economic_data)}")
print(f"New Dataset Size: {len(adult_only_eco_data)}")

🎯 Activity 2: Subsetting for a Specific Goal

Filtering the maternal and child healthcare dataset to focus on the health challenges of the poorest mothers living in rural areas:

# Load healthcare dataset
health_data = pd.read_csv('nigeria_nursing_mothers_healthcare.csv')

# Filter for poorest mothers living in rural areas
targeted_mothers = health_data[(health_data['Residence'] == 'Rural') & (health_data['Wealth_Quintile'] == 'Poorest')]

print(f"Original Dataset Size: {len(health_data)}")
print(f"New Dataset Size: {len(targeted_mothers)}")

🕵️ Activity 3: Investigating Suspicious Data

Detecting logical inconsistencies in the dataset (e.g., individuals under 18 years old claiming to have a Tertiary education level):

# Load unemployment dataset
unemployment_data = pd.read_csv('nigeria_unemployment_missing_data.csv')

# Filter to find anyone under 18 with a Tertiary education level
suspicious_data = unemployment_data[(unemployment_data['Age'] < 18) & (unemployment_data['Education_Level'] == 'Tertiary')]

print(f"Number of Suspicious Records Found: {len(suspicious_data)}")

---

🚀 How to Run the Notebook Locally

1. Clone the Repository:

   git clone https://github.com/iPablo26/data-sceince-instruction.git
   cd data-sceince-instruction

2. Set Up the Environment: Make sure you have python and pandas installed. You can install pandas and Jupyter Lab/Notebook via pip:

   pip install pandas jupyterlab

3. Launch Jupyter Lab:

   jupyter lab

   Open First Live Session - Introduction to Data Wrangling.ipynb and execute the cells sequentially to run the workflows.