Introduction to Data Science

A statistician working with 200 survey rows and an analyst working with 2 billion clickstream events are technically doing the same job. In practice, almost nothing about their day-to-day work overlaps.

Data Science is the discipline of extracting knowledge and actionable insight from structured and unstructured data using statistics, computer science, and domain expertise.

The gap between "we have data" and "we understand something" is not solved by writing more queries. It is a full pipeline problem, one that touches storage, cleaning, modeling, and communication in that order.

Why It Emerged as a Separate Field

Statistics existed for centuries before anyone needed the word "data scientist." What changed is scale. A statistician working with 200 survey rows and an analyst working with 2 billion clickstream events are not doing the same job, even if the underlying math shares a name.

Statistics: Small, curated datasets, hypothesis-driven, often collected specifically for the study
Data Science: Large, messy, often repurposed datasets, exploratory as often as hypothesis-driven
Software Engineering overlap: Data doesn't arrive clean, it arrives from APIs, logs, and scrapers that need real code to tame

[!NOTE]
A textbook will tell you Data Science sits at the intersection of statistics, domain knowledge, and computer science. True, but the boring version of that sentence is: half the job is data cleaning, and no course teaches that half well.

What a Data Scientist Actually Does Day to Day

The Digital Universe

Most of the data generated today was never typed by a human. A sensor, a camera, or a background process logged it without anyone asking it to.

The Digital Universe refers to the total volume of digital data created, replicated, and consumed globally in a given year, spanning structured records, unstructured media, and machine-generated logs.

How Fast It's Growing

The growth isn't linear. Three forces compound on each other: more people online, more devices per person, and each device generating richer data (video instead of text, sensor streams instead of single readings).

More users: Smartphone penetration crossing into rural and developing markets
More devices: One person now carries a phone, wears a watch, and drives a car that all log data independently
Richer formats: A single video call generates orders of magnitude more data than the text message it replaced

[!TIP]
When a syllabus says "digital universe is doubling every two years," the number itself matters less than the reason: it isn't just more people typing, it's each action generating heavier data than the same action did five years ago.

Structured vs Unstructured Share

Most of the digital universe is not sitting in neat rows and columns.

This 80/20 split is precisely why traditional relational databases, built for structured data, stopped being sufficient on their own, and why tools built for unstructured and semi-structured data became necessary.

Sources of Data

A retail company's "customer data" is actually five disconnected systems that have never talked to each other: point-of-sale logs, a loyalty app, warehouse sensors, social media mentions, and call center transcripts.

Sources of Data are the origin points from which raw data is captured before any cleaning, structuring, or analysis takes place.

Internal vs External Sources

Internal: Data generated by an organization's own operations, transaction logs, HR records, internal sensors
External: Data acquired from outside the organization, government census data, purchased market research, public APIs

The distinction matters for reliability. Internal data was collected with a known process, so its quirks are at least documented somewhere. External data arrives with none of that context, and assuming it was collected the same way your own systems collect data is a common, expensive mistake.

Primary vs Secondary Sources

• Primary data: Collected first-hand for a specific purpose, a survey you designed, an experiment you ran
• Secondary data: Collected by someone else for a different purpose, then reused, a government dataset, a research paper's published results

[!WARNING]
Secondary data is convenient because it's already collected, but the sampling method, time period, and definitions used by whoever collected it may not match your assumptions. Always check the methodology section before trusting the numbers.

Common Real-World Sources

Sensors and IoT: Temperature, GPS, motion, generating continuous streams rather than discrete records
Social media: Text, images, engagement metrics, high in volume but noisy and unstructured
Transactional systems: POS, banking, e-commerce, structured and reliable but often siloed across departments
Web scraping and APIs: Publicly available data pulled programmatically, subject to rate limits and terms of service

Sources determine everything downstream. A model trained on biased or narrow sources inherits that bias regardless of how sophisticated the algorithm is.

Information Commons

A library used to mean a room full of books, nothing more. Now it means a room full of workstations, group study pods, and a librarian who helps you access a licensed dataset rather than just find a physical book.

The Information Commons is a shared physical or digital space, often within libraries or universities, designed to give communities collaborative access to information resources, tools, and technical support.

What Makes It a "Commons"

The term borrows from the economic idea of a commons, a resource shared by a community rather than owned exclusively by one party. Applied to information, this means shared access to computing resources, databases, and expertise that no single individual would maintain on their own.

Shared resources: Computing labs, software licenses, database subscriptions
Shared expertise: Data librarians, research support staff
Shared space: Physical or virtual areas designed for collaborative work

Why It Matters for Data Science

Not every organization or student has access to enterprise-grade tools, licensed datasets, or high-performance computing. Information Commons initiatives close part of that gap by pooling resources at an institutional level.

[!NOTE]
A university's Information Commons is often the first place students touch tools like SPSS, SAS, or a proper GPU cluster, resources that would otherwise sit behind a paywall or a hardware budget most individuals don't have.

Data Science Project Life Cycle: The OSEMN Framework

The model failed in production. Not because the algorithm was wrong, the data feeding it was.

OSEMN (pronounced "awesome") stands for Obtain, Scrub, Explore, Model, iNterpret, a five-stage framework describing the practical workflow of a data science project from raw data to communicated result.

The Five Stages

Here's the flow end to end, one stage feeding directly into the next.

flowchart LR
    A[Obtain] --> B[Scrub]
    B --> C[Explore]
    C --> D[Model]
    D --> E[iNterpret]

Step 1: Obtain

Data is gathered from whatever sources apply, databases, APIs, flat files, scraped pages. The goal at this stage is access, not quality.

What happens: Querying databases, calling APIs, downloading files
Common tools: SQL, HTTP clients, web scraping libraries

Step 2: Scrub

Raw data is almost never usable as-is. Missing values, duplicate rows, inconsistent formatting, and outright errors get fixed here.

Consider a customer table with mixed date formats before cleaning.

String rawDate = "12/01/2026";
LocalDate parsed = LocalDate.parse(rawDate,
    DateTimeFormatter.ofPattern("dd/MM/yyyy"));

Standardizing the date format before any analysis prevents a silent bug where 12/01 gets read as December 1st in one row and January 12th in another.

[!CAUTION]
Scrubbing typically consumes more project time than every other stage combined. Budgeting a day for cleaning when it needs a week is the single most common estimation failure in data science projects.

Step 3: Explore

Exploratory Data Analysis (EDA) uncovers patterns, distributions, and relationships in the cleaned data before any formal modeling begins.

Univariate analysis: Examining one variable at a time, distributions, means, outliers
Bivariate analysis: Examining relationships between two variables, correlation, scatter plots
Visualization: Histograms, box plots, and heatmaps to surface what summary statistics hide

Step 4: Model

Statistical or machine learning models are applied to the explored data to predict outcomes or classify patterns.

// Simplified linear regression coefficient calculation
double slope = covariance(x, y) / variance(x);
double intercept = meanY - slope * meanX;

The model itself is rarely the hard part. Choosing the right model for the question, and validating it wasn't just memorizing the training data, is.

Step 5: iNterpret

Results are translated into a decision or recommendation that someone outside the data team can act on. A model with 94% accuracy means nothing to a stakeholder unless it's tied to what changes because of it.

Technical result: "The model achieved 0.89 AUC"
Interpreted result: "This model catches 89% of fraudulent transactions before they clear, saving an estimated 2 crore annually"

Why the Order Matters

Skipping straight from Obtain to Model, bypassing Scrub and Explore, is the fastest way to build a model on garbage data and not notice until it fails in production.

OSEMN isn't strictly linear in practice, most real projects loop back to Scrub after Explore reveals a problem missed the first time. The framework describes the stages that must happen, not a rule that they happen exactly once each.