Assignment 1
Data Pre-Processing Techniques – Cleaning, Integration, Transformation, Reduction, and Discretization
Lab Assignment
SET A
Q.1. Write a Python program to create an “employee” DataFrame with the following attributes:Employee_ID, Age, Department, Experience, and Performance_Score.Perform data cleaning by handling missing values, removing duplicates, and detecting outliers using the IQR method.
Solution:
#install command #
pip3 install pandas numpy
import pandas as pd
import numpy as np
# -------------------------------
# Create Employee DataFrame
# -------------------------------
employee = {
"Employee_ID": [101, 102, 103, 104, 105, 106, 107, 108, 108, 110],
"Age": [25, 30, np.nan, 45, 28, 60, 29, 27, 27, 120],
"Department": ["HR", "IT", "Finance", "HR", None, "IT", "Sales", "Finance", "Finance",
"IT"],
"Experience": [2, 5, 3, 20, 4, 35, np.nan, 2, 2, 1],
"Performance_Score": [85, 90, 78, np.nan, 88, 95, 82, 87, 87, 99]
}
df = pd.DataFrame(employee)
print("\nOriginal DataFrame")
print(df)
# -------------------------------
# Handle Missing Values
# -------------------------------
# Fill missing Age with mean
df["Age"] = df["Age"].fillna(df["Age"].mean())
# Fill missing Experience with median
df["Experience"] = df["Experience"].fillna(df["Experience"].median())
# Fill missing Performance Score with mean
df["Performance_Score"] =
df["Performance_Score"].fillna(df["Performance_Score"].mean())
# Fill missing Department with mode
df["Department"] = df["Department"].fillna(df["Department"].mode()[0])
print("\nDataFrame After Handling Missing Values")
print(df)
# -------------------------------
# Remove Duplicate Records
# -------------------------------
df = df.drop_duplicates()
print("\nDataFrame After Removing Duplicates")
print(df)
# -------------------------------
# Detect Outliers Using IQR Method
# -------------------------------
numeric_columns = ["Age", "Experience", "Performance_Score"]
for col in numeric_columns:
Q1 = df[col].quantile(0.25)
Q3 = df[col].quantile(0.75)
IQR = Q3 - Q1
lower_limit = Q1 - 1.5 * IQR
upper_limit = Q3 + 1.5 * IQR
outliers = df[(df[col] < lower_limit) | (df[col] > upper_limit)]
print("\n--------------------------------")
print("Column:", col)
print("Lower Limit:", lower_limit)
print("Upper Limit:", upper_limit)
if len(outliers) > 0:
print("Outliers Found:")
print(outliers)
else:
print("No Outliers Found")
Q.2.Write a Python program to create dataframes student_info, student_result and perform data integration on them.
Solution:
#install command :pip3 install pandas #
student_integration.py
import pandas as pd
# ---------------------------------
# Create student_info DataFrame
# ---------------------------------
student_info = pd.DataFrame({
"Student_ID": [101, 102, 103, 104, 105],
"Name": ["Amit", "Priya", "Rahul", "Sneha", "Rohan"],
"Age": [20, 21, 19, 22, 20],
"Department": ["CS", "IT", "CS", "ENTC", "IT"]
})
# ---------------------------------
# Create student_result DataFrame
# ---------------------------------
student_result = pd.DataFrame({
"Student_ID": [101, 102, 103, 104, 105],
"Subject": ["Python", "Python", "Python", "Python", "Python"],
"Marks": [85, 92, 78, 88, 95],
"Grade": ["A", "A+", "B+", "A", "A+"]
})
print("Student Information")
print(student_info)
print("\nStudent Result")
print(student_result)
# ---------------------------------
# Data Integration (Merge)
# ---------------------------------
integrated_data = pd.merge(student_info, student_result, on="Student_ID")
print("\nIntegrated Student Data")
print(integrated_data)
$ python3 student_integration.py
Student Information
Student_ID Name Age Department
0 101 Amit 20 CS
1 102 Priya 21 IT
2 103 Rahul 19 CS
3 104 Sneha 22 ENTC
4 105 Rohan 20 IT
Student Result
Student_ID Subject Marks Grade
0 101 Python 85 A
1 102 Python 92 A+
2 103 Python 78 B+
3 104 Python 88 A
4 105 Python 95 A+
Integrated Student Data
Student_ID Name Age Department Subject Marks Grade
0 101 Amit 20 CS Python 85 A
1 102 Priya 21 IT Python 92 A+
2 103 Rahul 19 CS Python 78 B+
3 104 Sneha 22 ENTC Python 88 A
4 105 Rohan 20 IT Python 95 A+
Q.3. Write a Python program to create a DataFrame using sales_data.csv with the attributes Product_ID, Price, Quantity_Sold, Discount, and Revenue. Apply data transformation techniques on the DataFrame and display the transformed data.
a) Min-Max Scaling
b) Standardization
c) Normalization
Solution:
#install command: pip3 install pandas scikit-learn#
sales_data.csv
Product_ID,Price,Quantity_Sold,Discount,Revenue
101,500,20,10,9000
102,700,15,5,9975
103,300,30,15,7650
104,1000,10,8,9200
105,450,25,12,9900
sales_transformation.py
import pandas as pd
from sklearn.preprocessing import MinMaxScaler, StandardScaler, Normalizer
# ---------------------------------------
# Read CSV File
# ---------------------------------------
df = pd.read_csv("sales_data.csv")
print("Original DataFrame")
print(df)
# Select numerical columns
numeric_columns = ["Price", "Quantity_Sold", "Discount", "Revenue"]
# ---------------------------------------
# a) Min-Max Scaling
# ---------------------------------------
minmax_scaler = MinMaxScaler()
minmax_data = df.copy()
minmax_data[numeric_columns] = minmax_scaler.fit_transform(df[numeric_columns])
print("\nMin-Max Scaled Data")
print(minmax_data)
# ---------------------------------------
# b) Standardization (Z-score)
# ---------------------------------------
standard_scaler = StandardScaler()
standard_data = df.copy()
standard_data[numeric_columns] = standard_scaler.fit_transform(df[numeric_columns])
print("\nStandardized Data")
print(standard_data)
# ---------------------------------------
# c) Normalization
# ---------------------------------------
normalizer = Normalizer()
normalized_data = df.copy()
normalized_data[numeric_columns] = normalizer.fit_transform(df[numeric_columns])
print("\nNormalized Data")
print(normalized_data)
Q.4Write a Python program to create a dataframe using customer_data.csv and perform encoding techniques:
[Consider appropriate attributes for following operations]
a) Label Encoding
b) One-Hot Encoding
Solution:
#install command : pip3 install pandas scikit-learn#
customer_data.csv
Customer_ID,Gender,City,Membership,Purchase_Amount
101,Male,Pune,Gold,2500
102,Female,Mumbai,Silver,1800
103,Male,Nashik,Gold,3200
104,Female,Pune,Platinum,4500
105,Male,Mumbai,Silver,2100
customer_encoding.py
import pandas as pd
from sklearn.preprocessing import LabelEncoder
# ---------------------------------
# Read CSV File
# ---------------------------------
df = pd.read_csv("customer_data.csv")
print("Original DataFrame")
print(df)
# ---------------------------------
# a) Label Encoding
# ---------------------------------
label_df = df.copy()
le = LabelEncoder()
# Label Encode Gender
label_df["Gender"] = le.fit_transform(label_df["Gender"])
# Label Encode Membership
label_df["Membership"] = le.fit_transform(label_df["Membership"])
print("\nDataFrame after Label Encoding")
print(label_df)
# ---------------------------------
# b) One-Hot Encoding
# ---------------------------------
onehot_df = pd.get_dummies(
df,
columns=["City", "Membership"],
dtype=int
)
print("\nDataFrame after One-Hot Encoding")
print(onehot_df)
#OUTPUT#
python3 customer_encoding.py
Original DataFrame
Customer_ID Gender City Membership Purchase_Amount
0 101 Male Pune Gold 2500
1 102 Female Mumbai Silver 1800
2 103 Male Nashik Gold 3200
3 104 Female Pune Platinum 4500
4 105 Male Mumbai Silver 2100
DataFrame after Label Encoding
Customer_ID Gender City Membership Purchase_Amount
0 101 1 Pune 0 2500
1 102 0 Mumbai 2 1800
2 103 1 Nashik 0 3200
3 104 0 Pune 1 4500
4 105 1 Mumbai 2 2100
DataFrame after One-Hot Encoding
Customer_ID Gender Purchase_Amount City_Mumbai City_Nashik City_Pune
Membership_Gold Membership_Platinum Membership_Silver
0 101 Male 2500 0 0 1 1 0
0
1 102 Female 1800 1 0 0 0 0
1
2 103 Male 3200 0 1 0 1 0
0
3 104 Female 4500 0 0 1 0 1
0
4 105 Male 2100 1 0 0 0 0
1
Q.5.Write a Python program to create a dataframe using age_data.csv and perform Data Discretization
using:
a) Equal Width Binning
b) Equal Frequency Binning
#install command: pip3 install pandas #
age_data.csv
Person_ID,Name,Age
1,Amit,18
2,Priya,22
3,Rahul,25
4,Sneha,28
5,Rohan,31
6,Neha,35
7,Ajay,40
8,Pooja,45
9,Karan,50
10,Anita,55
age_discretization.py
import pandas as pd
# ---------------------------------
# Read CSV File
# ---------------------------------
df = pd.read_csv("age_data.csv")
print("Original DataFrame")
print(df)
# ---------------------------------
# a) Equal Width Binning
# ---------------------------------
df["Equal_Width_Bin"] = pd.cut(df["Age"], bins=3)
print("\nDataFrame after Equal Width Binning")
print(df)
# ---------------------------------
# b) Equal Frequency Binning
# ---------------------------------
df["Equal_Frequency_Bin"] = pd.qcut(df["Age"], q=3)
print("\nDataFrame after Equal Frequency Binning")
print(df)
#output#
python3 age_discretization.py
Original DataFrame
Person_ID Name Age
0 1 Amit 18
1 2 Priya 22
2 3 Rahul 25
3 4 Sneha 28
4 5 Rohan 31
5 6 Neha 35
6 7 Ajay 40
7 8 Pooja 45
8 9 Karan 50
9 10 Anita 55
DataFrame after Equal Width Binning
Person_ID Name Age Equal_Width_Bin
0 1 Amit 18 (17.963, 30.333]
1 2 Priya 22 (17.963, 30.333]
2 3 Rahul 25 (17.963, 30.333]
3 4 Sneha 28 (17.963, 30.333]
4 5 Rohan 31 (30.333, 42.667]
5 6 Neha 35 (30.333, 42.667]
6 7 Ajay 40 (30.333, 42.667]
7 8 Pooja 45 (42.667, 55.0]
8 9 Karan 50 (42.667, 55.0]
9 10 Anita 55 (42.667, 55.0]
DataFrame after Equal Frequency Binning
Person_ID Name Age Equal_Width_Bin Equal_Frequency_Bin
0 1 Amit 18 (17.963, 30.333] (17.999, 28.0]
1 2 Priya 22 (17.963, 30.333] (17.999, 28.0]
2 3 Rahul 25 (17.963, 30.333] (17.999, 28.0]
3 4 Sneha 28 (17.963, 30.333] (17.999, 28.0]
4 5 Rohan 31 (30.333, 42.667] (28.0, 40.0]
5 6 Neha 35 (30.333, 42.667] (28.0, 40.0]
6 7 Ajay 40 (30.333, 42.667] (28.0, 40.0]
7 8 Pooja 45 (42.667, 55.0] (40.0, 55.0]
8 9 Karan 50 (42.667, 55.0] (40.0, 55.0]
9 10 Anita 55 (42.667, 55.0] (40.0, 55.0]
SET B
1. Write a Python program to create a dataframe using iris flower dataset and reduce 4D data to 2D data using PCA.
Solution:
#Install Command : pip3 install pandas scikit-learn matplotlib #
iris_pca.py
# Import required libraries
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
# -------------------------------
# Load Iris Dataset
# -------------------------------
iris = load_iris()
# Create DataFrame
df = pd.DataFrame(iris.data, columns=iris.feature_names)
df["Species"] = iris.target
print("Original Iris Data (First 5 Rows)")
print(df.head())
# -------------------------------
# Apply PCA (4D to 2D)
# -------------------------------
X = iris.data
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)
# Create DataFrame for PCA result
pca_df = pd.DataFrame(X_pca, columns=["Principal_Component_1",
"Principal_Component_2"])
pca_df["Species"] = iris.target
print("\n2D Data After PCA")
print(pca_df.head())
# -------------------------------
# Plot PCA Result
# -------------------------------
plt.figure(figsize=(8,6))
for i, species in enumerate(iris.target_names):
plt.scatter(
X_pca[iris.target == i, 0],
X_pca[iris.target == i, 1],
label=species
)
plt.title("PCA of Iris Dataset (4D to 2D)")
plt.xlabel("Principal Component 1")
plt.ylabel("Principal Component 2")
plt.legend()
plt.grid(True)
plt.show()
#OUTPUT#
kgdm@kgdm-OptiPlex-3000:~$ python3 iris_pca.py
Original Iris Data (First 5 Rows)
sepal length (cm) sepal width (cm) petal length (cm) petal width (cm) Species
0 5.1 3.5 1.4 0.2 0
1 4.9 3.0 1.4 0.2 0
2 4.7 3.2 1.3 0.2 0
3 4.6 3.1 1.5 0.2 0
4 5.0 3.6 1.4 0.2 0
2D Data After PCA
Principal_Component_1 Principal_Component_2 Species
0 -2.684126 0.319397 0
1 -2.714142 -0.177001 0
2 -2.888991 -0.144949 0
3 -2.745343 -0.318299 0
4 -2.728717 0.326755 0
Q.2.Write a Python program to perform data reduction techniques as
a) Feature Selection to identify important features for disease prediction.
b) Feature extraction (Use Patient_health.csv – Pno, Age, BP, Chol, Sugar(0 = No, 1 = Yes), BMI, Disease)
#install command : pip install pandas scikit-learn#
feature_reduction.py
# Import libraries
import pandas as pd
from sklearn.feature_selection import SelectKBest, chi2
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
# Load dataset
df = pd.read_csv("Patient_health.csv")
print("Original Dataset:")
print(df)
# Independent variables
X = df[['Age', 'BP', 'Chol', 'Sugar', 'BMI']]
# Target variable
y = df['Disease']
# -----------------------------
# (a) Feature Selection
# -----------------------------
selector = SelectKBest(score_func=chi2, k=3)
X_new = selector.fit_transform(X, y)
selected_features = X.columns[selector.get_support()]
print("\nSelected Important Features:")
print(selected_features)
print("\nFeature Scores:")
for feature, score in zip(X.columns, selector.scores_):
print(feature, ":", round(score, 2))
# -----------------------------
# (b) Feature Extraction (PCA)
# -----------------------------
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
print("\nData after PCA:")
pca_df = pd.DataFrame(X_pca, columns=['PC1', 'PC2'])
print(pca_df)
print("\nExplained Variance Ratio:")
print(pca.explained_variance_ratio_)
Patient_health.csv
Pno,Age,BP,Chol,Sugar,BMI,Disease
1,45,130,220,1,28.5,1
2,30,120,180,0,22.1,0
3,55,145,250,1,31.2,1
4,40,135,210,0,26.4,0
5,60,150,260,1,33.5,1
6,35,125,190,0,24.0,0
7,50,140,230,1,29.8,1
8,28,118,170,0,21.5,0
9,65,155,270,1,34.2,1
10,38,128,200,0,25.3,0
OUTPUT
kgdm@kgdm-OptiPlex-3000:~$ python3 feature_reduction.py
Original Dataset
Pno Age BP Chol Sugar BMI Disease
0 1 45 130 220 1 28.5 1
1 2 30 120 180 0 22.1 0
2 3 55 145 250 1 31.2 1
3 4 40 135 210 0 26.4 0
4 5 60 150 260 1 33.5 1
Selected Important Features:
Index(['Age', 'BP', 'Chol'], dtype='object')
Feature Scores
Age : 24.25
BP : 6.56
Chol : 35.96
Sugar : 5.0
BMI : 5.19
Reduced Data using PCA
Principal Component 1 Principal Component 2
0 0.376461 1.012546
1 -2.643857 0.097995
2 2.028674 0.162954
3 -0.820753 -0.776063
4 2.792598 -0.160510
5 -1.922656 -0.219351
6 1.220146 0.521821
7 -2.999687 0.251987
8 3.385629 -0.459506
9 -1.416554 -0.431872
Explained Variance Ratio
[0.94744106 0.04942025]
Q.3.Load Superstore dataset(Order_ID, Region, Category, Sales,Profit, Discount) and perform Data Cube Aggregation to analyze region-wise and category-wise sales performance.
#Install command: pip install pandas#
Superstore.csv
Order_ID,Region,Category,Sales,Profit,Discount
1001,East,Furniture,500,50,0.10
1002,West,Technology,800,120,0.05
1003,South,Office Supplies,300,40,0.00
1004,North,Furniture,700,80,0.15
1005,East,Technology,900,150,0.10
1006,West,Office Supplies,400,60,0.05
1007,South,Furniture,650,70,0.20
1008,North,Technology,1000,180,0.05
1009,East,Office Supplies,350,45,0.00
1010,West,Furniture,550,65,0.10
data_cube.py
import pandas as pd
# Load dataset
df = pd.read_csv("Superstore.csv")
print("Original Dataset")
print(df)
# -------------------------------
# 1. Region-wise Sales
# -------------------------------
print("\nRegion-wise Sales")
region_sales = df.groupby("Region")["Sales"].sum()
print(region_sales)
# -------------------------------
# 2. Category-wise Sales
# -------------------------------
print("\nCategory-wise Sales")
category_sales = df.groupby("Category")["Sales"].sum()
print(category_sales)
# -------------------------------
# 3. Region and Category-wise Sales & Profit
# -------------------------------
print("\nData Cube Aggregation (Region x Category)")
cube = df.pivot_table(
index="Region",
columns="Category",
values=["Sales", "Profit"],
aggfunc="sum",
fill_value=0,
margins=True
)
print(cube)
OUTPUT
$ python3 data_cube.py
Original Dataset
Order_ID Region Category Sales Profit Discount
0 1001 East Furniture 500 50 0.10
1 1002 West Technology 800 120 0.05
2 1003 South Office Supplies 300 40 0.00
3 1004 North Furniture 700 80 0.15
4 1005 East Technology 900 150 0.10
5 1006 West Office Supplies 400 60 0.05
6 1007 South Furniture 650 70 0.20
7 1008 North Technology 1000 180 0.05
8 1009 East Office Supplies 350 45 0.00
9 1010 West Furniture 550 65 0.10
Region-wise Sales
Region
East 1750
North 1700
South 950
West 1750
Name: Sales, dtype: int64
Category-wise Sales
Category
Furniture 2400
Office Supplies 1050
Technology 2700
Name: Sales, dtype: int64
Data Cube Aggregation (Region x Category)
Profit Sales
Category Furniture Office Supplies Technology All Furniture Office Supplies Technology All
Region
East 50 45 150 245 500 350 900 1750
North 80 0 180 260 700 0 1000 1700
South 70 40 0 110 650 300 0 950
West 65 60 120 245 550 400 800 1750
All 265 145 450 860 2400 1050 2700 6150
Q.4.Load Employee(eid, age, workclass, education, hours-per-week, income) dataset and perform Numerosity Reduction.
Solution:
#Install Command:pip install pandas scikit-learn#
Employee.csv
eid,age,workclass,education,hours-per-week,income
1,25,Private,Bachelors,40,<=50K
2,45,Government,Masters,50,>50K
3,35,Private,Bachelors,45,<=50K
4,50,Self-emp,PhD,60,>50K
5,28,Private,Graduate,38,<=50K
6,40,Government,Masters,48,>50K
7,32,Self-emp,Bachelors,42,<=50K
8,55,Private,PhD,55,>50K
9,29,Government,Graduate,40,<=50K
10,47,Private,Masters,52,>50K
numerosity_reduction.py
# Import libraries
import pandas as pd
from sklearn.preprocessing import LabelEncoder
from sklearn.cluster import KMeans
# Load dataset
df = pd.read_csv("Employee.csv")
print("Original Dataset:")
print(df)
# Encode categorical columns
le = LabelEncoder()
df['workclass'] = le.fit_transform(df['workclass'])
df['education'] = le.fit_transform(df['education'])
df['income'] = le.fit_transform(df['income'])
# Select features for clustering
X = df[['age', 'workclass', 'education', 'hours-per-week']]
# Apply K-Means (3 clusters)
kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
df['Cluster'] = kmeans.fit_predict(X)
print("\nDataset after Numerosity Reduction (Cluster Assignment):")
print(df)
print("\nCluster Centers:")
print(pd.DataFrame(kmeans.cluster_centers_,
columns=['age', 'workclass', 'education', 'hours-per-week']))
//OUTPUT//
$ python3 numerosity_reduction.py
Original Dataset:
eid age workclass education hours-per-week income
0 1 25 Private Bachelors 40 <=50K
1 2 45 Government Masters 50 >50K
2 3 35 Private Bachelors 45 <=50K
3 4 50 Self-emp PhD 60 >50K
4 5 28 Private Graduate 38 <=50K
5 6 40 Government Masters 48 >50K
6 7 32 Self-emp Bachelors 42 <=50K
7 8 55 Private PhD 55 >50K
8 9 29 Government Graduate 40 <=50K
9 10 47 Private Masters 52 >50K
Dataset after Numerosity Reduction (Cluster Assignment):
eid age workclass education hours-per-week income Cluster
0 1 25 1 0 40 0 1
1 2 45 0 2 50 1 2
2 3 35 1 0 45 0 1
3 4 50 2 3 60 1 0
4 5 28 1 1 38 0 1
5 6 40 0 2 48 1 2
6 7 32 2 0 42 0 1
7 8 55 1 3 55 1 0
8 9 29 0 1 40 0 1
9 10 47 1 2 52 1 2
Cluster Centers:
age workclass education hours-per-week
0 52.5 1.500000 3.0 57.5
1 29.8 1.000000 0.4 41.0
2 44.0 0.333333 2.0 50.0
No comments:
Post a Comment