Assignment 1

1. Data Science Project Life Cycle

A data science project moves through a repeatable set of stages, each feeding the next.

graph LR
A[Business Understanding] --> B[Data Acquisition]
B --> C[Data Preparation]
C --> D[Exploratory Data Analysis]
D --> E[Modeling]
E --> F[Evaluation]
F --> G[Deployment]
G --> H[Monitoring & Maintenance]

Business Understanding: Define the problem, objectives, and success metrics with stakeholders.
Data Acquisition: Collect data from databases, APIs, sensors, logs, or third party sources.
Data Preparation: Clean, transform, and structure raw data into usable form.
Exploratory Data Analysis (EDA): Summarize and visualize data to spot patterns, outliers, and relationships.
Modeling: Apply statistical or machine learning techniques to build predictive or descriptive models.
Evaluation: Test model performance against defined metrics such as accuracy, precision, or RMSE.
Deployment: Integrate the model into production systems for real world use.
Monitoring & Maintenance: Track performance over time and retrain as data drifts.

The cycle is iterative. Poor results at evaluation often send the project back to data preparation or even business understanding.

2. OSEMN Framework

OSEMN stands for Obtain, Scrub, Explore, Model, Interpret. It is a practical five step framework describing what a data scientist actually does day to day.

Obtain: Gather data from files, databases, web scraping, or APIs.
Scrub: Clean the data by handling missing values, duplicates, and inconsistent formats.
Explore: Use statistics and visualization to understand distributions and relationships.
Model: Build machine learning or statistical models to answer the business question.
Interpret: Translate model output into insights and actionable recommendations for stakeholders.

Stage	Focus	Typical Tools
Obtain	Data collection	SQL, APIs, web scraping
Scrub	Data cleaning	Pandas, R, OpenRefine
Explore	Pattern discovery	Matplotlib, Seaborn, Tableau
Model	Prediction/inference	Scikit-learn, R, TensorFlow
Interpret	Communication	Reports, dashboards, storytelling

[!NOTE]
OSEMN overlaps heavily with the general data science life cycle, but it is framed as a workflow for individual practitioners rather than a project management process.

3. What is Data Science?

Data Science is an interdisciplinary field that combines statistics, computer science, and domain knowledge to extract meaningful insights and knowledge from structured and unstructured data.

Importance of Data Science:

Enables data driven decision making instead of relying on intuition alone
Uncovers hidden patterns and trends invisible to manual analysis
Powers automation through predictive models and machine learning
Helps organizations forecast demand, detect fraud, and personalize experiences

Applications of Data Science:

Healthcare: Disease prediction, medical image analysis, drug discovery
Finance: Credit scoring, fraud detection, algorithmic trading
E-commerce: Recommendation systems, dynamic pricing, customer segmentation
Transportation: Route optimization, self-driving vehicle perception
Social Media: Sentiment analysis, content recommendation, ad targeting

4. Sources of Data in Data Science

Data used in data science projects comes from several distinct categories.

Structured Data Sources: Relational databases, spreadsheets, and enterprise systems like ERP or CRM software.
Unstructured Data Sources: Text documents, images, audio, video, and social media posts.
Semi-structured Data Sources: JSON, XML, and log files that have some organizational structure without a strict schema.
Web Sources: Web scraping, public APIs, and open government data portals.
Sensor and IoT Sources: Data streamed continuously from devices such as wearables, industrial sensors, and smart appliances.
Transactional Sources: Point-of-sale systems, online transactions, and banking records.

[!TIP]
Real projects usually blend multiple source types. A retail analytics project might combine transactional data, web clickstream logs, and social media sentiment in the same pipeline.

5. Digital Universe and Information Commons

The Digital Universe refers to the total volume of digital data created, replicated, and consumed globally at any given time, a figure that has grown exponentially with smartphones, IoT, and social media.

Key characteristics of the digital universe:

Growing at an exponential rate year over year
Dominated by unstructured data such as images, video, and text
A large share is generated by consumers rather than enterprises, though enterprises are responsible for storing and managing most of it

The Information Commons is the idea that certain data and information resources should be treated as shared public goods, freely accessible for collective use rather than locked behind private ownership. It draws a parallel to the traditional "commons" concept in economics, applied to knowledge and data.

Digital Universe vs Information Commons

Aspect	Digital Universe	Information Commons
What it is	Total volume of digital data	Shared, open pool of information
Ownership	Mixed private and public	Intended to be collectively accessible
Example	All data generated by smartphones globally	Wikipedia, open government datasets

Assignment 2

1. Need and Importance of Data Preprocessing

Raw data collected from real world sources is rarely clean. It contains missing values, duplicates, inconsistent formats, and noise that can distort analysis if used directly.

Why preprocessing matters:

Improves data quality, which directly improves model accuracy
Removes noise and inconsistencies that could mislead algorithms
Converts data into a format algorithms can actually consume, since most models require numeric, well-structured input
Reduces computational cost by eliminating redundant or irrelevant features
Prevents biased or misleading conclusions caused by dirty data

[!WARNING]
Skipping preprocessing rarely fails loudly. Models still train and produce numbers, just wrong ones, which makes the mistake far more dangerous than an outright crash.

2. Handling Missing Values

Missing values appear when data is not recorded, lost during collection, or intentionally left blank. Several methods exist to handle them depending on the situation.

Deletion Methods:

Listwise deletion: Remove entire rows containing any missing value
Pairwise deletion: Use available data for each specific calculation, ignoring missing entries only where relevant

Imputation Methods:

Mean/Median/Mode imputation: Replace missing numeric values with the column mean or median, and categorical values with the mode
Forward/Backward fill: Carry the previous or next valid value forward, common in time series data
Regression imputation: Predict the missing value using a regression model built on other features
K-Nearest Neighbors (KNN) imputation: Estimate the missing value based on the values of the most similar records

Method	Best For	Drawback
Listwise deletion	Small amount of missing data	Loses information, shrinks dataset
Mean/Median imputation	Numeric columns, quick fixes	Reduces variance, ignores relationships
KNN imputation	Datasets with correlated features	Computationally expensive on large data

3. Data Manipulation: Sorting, Grouping, and Ranking

Sorting arranges records based on one or more column values, either ascending or descending, making trends and extremes easier to spot.

Grouping splits data into subsets based on a categorical variable, then applies an aggregate function like sum, mean, or count to each subset. This is often called the "split-apply-combine" pattern.

Ranking assigns a position number to each record based on its value relative to others, useful for leaderboards, percentile analysis, or identifying top performers.

The examples below use a sales data frame in R to demonstrate all three operations separately.

Sorting

Use order() to sort rows by a column. Negative sign flips to descending order.

sorted_data <- sales_df[order(-sales_df[["revenue"]]), ]

Rows are now arranged from highest to lowest revenue.

Grouping

Use group_by() with summarise() from dplyr to aggregate within each category.

library(dplyr)
grouped_data <- sales_df %>%
  group_by(region) %>%
  summarise(avg_revenue = mean(revenue))

Each region now has a single row showing its average revenue.

Ranking

Use rank() to assign a position to each row based on a column value.

sales_df[["rank"]] <- rank(-sales_df[["revenue"]])

The salesperson with the highest revenue receives rank 1.

4. Reading, Selecting, and Filtering Data in R

Steps are sequential: read the data into memory first, then select the columns you need, then filter down to the rows that matter.

Step 1: Read the Data

data <- read.csv("sales.csv")

This loads the file into a data frame, the primary tabular structure in R. Use head(data) immediately after to verify the structure loaded correctly.

Step 2: Select Specific Columns

selected_data <- data[, c("product", "revenue")]

Only the product and revenue columns are kept. Everything else is dropped from the working copy.

Step 3: Filter Rows by Condition

filtered_data <- data[data[["revenue"]] > 10000, ]

Only rows where revenue exceeds 10000 survive. The trailing comma inside the brackets is required — it tells R you are indexing rows, not columns.

Step 4: Combine All Three with dplyr

library(dplyr)
result <- data %>%
  filter(revenue > 10000) %>%
  select(product, revenue)

The dplyr pipeline reads left to right: filter first to shrink the row count, then select to narrow the columns, producing a clean subset in one chain.

5. Filtering, Rearranging, and Grouping Compared

These three operations solve different problems, though they are often chained together in the same pipeline.

Filtering reduces the number of rows by keeping only those matching a condition, without touching column structure.

filter(data, revenue > 5000)

Rearranging (Sorting) changes the row order without removing or adding any data.

arrange(data, desc(revenue))

Grouping changes how subsequent operations are applied, splitting data by category before aggregation, rather than changing the visible row count on its own.

data %>% group_by(region) %>% summarise(total = sum(revenue))

Operation	Changes Row Count	Changes Row Order	Requires Aggregation
Filtering	Yes	No	No
Rearranging	No	Yes	No
Grouping	Only after aggregation	No inherently	Usually yes

[!TIP]
A common pipeline order is filter first to shrink the dataset, group next to organize by category, then summarise to aggregate, keeping each stage's intent clear and the code readable.

Assignment 3

1. Formulation of Hypothesis

A hypothesis is a candidate function mapping inputs to outputs, proposed as an approximation of the true target concept.

Steps: define the input space (X) and output space (Y), pick a hypothesis representation, then use training examples to narrow the hypothesis space (H) down to the best fit.

Example: For "days suitable for playing tennis" with attributes Outlook, Temperature, Humidity, Wind:

h = <Sunny, ?, Normal, ?>

This says the day is suitable when Outlook is Sunny and Humidity is Normal; ? means any value is fine for that attribute.

2. PAC Learning

PAC (Probably Approximately Correct) learning says a concept is learnable if an algorithm can, from a reasonable number of examples, output a hypothesis that is approximately correct (error ≤ ε) with high probability (≥ 1 − δ).

Probably: succeeds with probability at least 1 − δ, not certainty
Approximately Correct: error stays under ε, not zero

Required sample size grows with log(|H|) and shrinks as ε and δ are relaxed.

[!TIP]
PAC learning bounds the chance of getting a good model, not the model itself.

3. VC Dimension

The VC (Vapnik-Chervonenkis) Dimension is the size of the largest point set a hypothesis space H can shatter (correctly classify under every possible labeling).

Example: A line in 2D can shatter any 3 points but not 4, so linear classifiers in 2D have VC dimension 3.

Significance:

Higher VC dimension → richer, more flexible hypothesis space
Too high relative to training size → overfitting
Replaces log(|H|) in PAC bounds when H is infinite

4. Candidate Elimination Algorithm

Maintains two boundary sets across the version space of hypotheses consistent with the data:

S (specific boundary): generalized just enough by each positive example
G (general boundary): specialized just enough by each negative example

Example (tennis dataset, attributes Sky, AirTemp, Humidity, Wind, Water, Forecast):

Ex	Sky	AirTemp	Humidity	Wind	Water	Forecast	EnjoySport
1	Sunny	Warm	Normal	Strong	Warm	Same	Yes
2	Sunny	Warm	High	Strong	Warm	Same	Yes
3	Rainy	Cold	High	Strong	Warm	Change	No
4	Sunny	Warm	High	Strong	Cool	Change	Yes

S starts at <Sunny,Warm,Normal,Strong,Warm,Same> after example 1, generalizes to <Sunny,Warm,?,Strong,Warm,Same> after example 2. Example 3 (negative) specializes G to hypotheses like <Sunny,?,?,?,?,?> and <?,Warm,?,?,?,?>. Example 4 generalizes S further, converging toward <Sunny,Warm,?,Strong,?,?>.

[!NOTE]
Converges to the correct hypothesis only if it's actually representable in H and the data is noise-free.

5. Hypothesis Elimination vs Candidate Elimination

Aspect	Hypothesis Elimination (Find-S / List-Then-Eliminate)	Candidate Elimination
Tracks	One hypothesis, or a full explicit list	Compact S and G boundaries
Negative examples	Find-S ignores them	Used to specialize G
Efficiency	List-Then-Eliminate is exponential in \|H\|	Polynomial, independent of \|H\|
Noise sensitivity	One bad example misleads Find-S	Can collapse the whole version space