Introduction to Data Science

Data Science is the domain of study that deals with vast volumes of data using modern tools and techniques to find unseen patterns, derive meaningful information, and support business decisions. It draws on statistics, computer science, and domain expertise, and uses machine learning algorithms to build predictive models from data collected across many different sources and formats.

Why It Emerged as a Separate Field

Statistics existed long before the term "data scientist." What changed is scale, the volume, velocity, and variety of data available for analysis.

Statistics: Small, curated datasets, hypothesis-driven, often collected specifically for the study
Data Science: Large, messy, often repurposed datasets, exploratory as often as hypothesis-driven
Software Engineering overlap: Data rarely arrives clean, it arrives from APIs, logs, and scrapers that require real code to structure

[!NOTE]
Data Science is typically described as sitting at the intersection of statistics, domain knowledge, and computer science. In practice, a substantial share of the work is data cleaning and preparation rather than modeling.

Skills of a Data Scientist

A data scientist's core skill set spans several areas rather than a single specialization.

Probability and Statistics: Foundation for hypothesis testing, distributions, and inference
Machine Learning and Deep Learning: Building predictive and classification models
Data Wrangling: Cleaning, transforming, and structuring raw data for analysis
Programming: Typically Python, R, or SQL for data manipulation and modeling
Domain Knowledge: Understanding the business or scientific context the data comes from

Applications of Data Science

Data science techniques apply across a wide range of industries, most commonly in the following areas.

What a Data Scientist Actually Does Day to Day

The Digital Universe

The Digital Universe refers to the entirety of digital data generated, stored, processed, and transmitted globally, encompassing all data created by individuals, organizations, and devices. It is significant for understanding the scale and impact of digital information on society, economy, and technology.

The 5 V's of the Digital Universe

The digital universe is commonly characterized using five defining properties.

Volume: The vast and exponentially growing amount of data, from social media, financial transactions, IoT devices, and multimedia content
Variety: The multiple formats data takes, structured (databases, spreadsheets), unstructured (text, images, video), and semi-structured (JSON, XML)
Velocity: The speed at which data is generated and processed, exemplified by real-time sensor feeds, social media streams, and financial markets
Veracity: The accuracy and reliability of data, given that the digital universe contains both high-quality signal and significant noise
Value: The actionable insight that can be extracted from data to inform decisions and create advantage

[!TIP]
The commonly cited claim that the digital universe doubles every two years reflects not just more users, but each action generating heavier data than the same action did previously — a video call versus the text message it replaced, for instance.

Uses of the Digital Universe

• Insight generation: Uncovering patterns and correlations across large datasets
• Predictive analytics: Forecasting inventory needs, patient outcomes, or market trends
• Personalization: Tailoring recommendations on streaming and e-commerce platforms
• Operational efficiency: Optimizing manufacturing and logistics using IoT and real-time data
• Fraud detection and security: Monitoring transaction and traffic patterns for anomalies

Structured, Unstructured, and Semi-Structured Data

Most of the digital universe is not stored in structured, tabular form.

A fourth category, metadata, refers to data about data, such as file size, creation date, or author, and is used to describe and organize the other three types. This 80/20 structured-unstructured split is why traditional relational databases became insufficient on their own, and why tools for unstructured and semi-structured data became necessary.

Challenges of the Digital Universe

Working with data at this scale introduces problems that don't exist with small, curated datasets.

Noise and veracity: A large share of collected data is unreliable or irrelevant, separating signal from noise takes real effort
Storage and processing: Traditional relational systems and single-machine tools cannot scale to the required volume, requiring distributed frameworks like Hadoop and Spark
Privacy and security: Managing personal and sensitive data at scale requires compliance with regulations such as GDPR and CCPA
Governance: Establishing consistent policies and standards for how data is collected, stored, and used across an organization

Sources of Data

Sources of Data are the origin points from which raw data is captured before any cleaning, structuring, or analysis takes place. A data source may be the initial location where data is born, or any refined dataset that another process subsequently accesses and reuses.

Internal vs External Sources

Internal: Data generated by an organization's own operations, transaction logs, HR records, internal sensors
External: Data acquired from outside the organization, government census data, purchased market research, public APIs, satellite data

Internal data was collected under a known process, so its quirks are typically documented. External data arrives without that context, and assuming it was collected the same way an organization's own systems collect data is a common source of error.

Primary vs Secondary Sources

• Primary data: Collected first-hand for a specific purpose, a survey designed for the study, an experiment run by the researcher, interview data
• Secondary data: Collected by someone else for a different purpose, then reused, a government dataset, a research paper's published results, internet browsing data

[!WARNING]
Secondary data is convenient because it already exists, but the sampling method, time period, and definitions used by the original collector may not match the assumptions of the new analysis. The methodology section should be checked before the numbers are trusted.

Common Real-World Sources by Data Type

Structured sources: Relational databases (MySQL, PostgreSQL), NoSQL databases (MongoDB), spreadsheets
Unstructured sources: Text files, multimedia (images, audio, video), documents
Semi-structured sources: Server and application logs, web scraping outputs, HTML pages
APIs and streaming sources: RESTful and SOAP APIs, real-time streams from Kafka or MQTT
Sensors and IoT: Temperature, GPS, and motion data, generating continuous streams rather than discrete records

Sources determine everything downstream. A model trained on biased or narrow sources inherits that bias regardless of the algorithm's sophistication.

Information Commons

The Information Commons is a shared environment, physical or digital, often within libraries or universities, that facilitates access to data, tools, and computing resources for research, learning, and innovation. It supports open science and collaborative teamwork by pooling infrastructure that no single individual would maintain independently.

Components of Information Commons

Data repositories: Centralized, cataloged datasets, such as census data or the Human Genome Project
Computing resources: Cloud platforms (AWS, GCP, Azure) and institutional HPC clusters
Collaborative workspaces: Physical or virtual spaces such as data science labs or platforms like Jupyter Notebooks and GitHub
Software and visualization tools: Statistical software (R, SAS), ML frameworks, and visualization tools (Tableau, D3.js)
Knowledge repositories and training programs: Digital libraries (IEEE Xplore, arXiv), workshops, and online courses

Why It Matters for Data Science

Not every organization or student has access to enterprise-grade tools, licensed datasets, or high-performance computing. Information Commons initiatives close part of that gap by pooling resources at an institutional level, which also supports data governance and ethics frameworks around privacy and responsible use.

[!NOTE]
A university's Information Commons is often the first point of access to tools like SPSS, SAS, or a GPU cluster, resources that otherwise sit behind a paywall or a hardware budget most individuals do not have.

Data Science Project Life Cycle

The Data Science Project Life Cycle describes the structured sequence of stages a project follows, from understanding a business objective to deploying a working model. Skipping or rushing any stage propagates errors into every stage after it.

The Stages

Step 1: Business Understanding

The entire cycle revolves around a concrete business objective. Without a specific problem to solve, there is no way to judge whether an analysis succeeded.

Step 2: Data Understanding

All reachable data relevant to the problem is gathered and described, its structure, relevance, and data types, typically in close coordination with the business team that knows what data exists.

Step 3: Preparation of Data

Relevant data is selected, merged across sources, cleaned, and missing or inaccurate values are handled through removal or imputation. Outliers are checked using tools like box plots. This stage is usually the most time-consuming in the entire life cycle.

Step 4: Exploratory Data Analysis

Distributions and relationships between variables are examined graphically, using bar charts, scatter plots, and heat maps, before any model is built.

Step 5: Data Modeling

The organized data is used to select and tune an appropriate model, classification, regression, or clustering, balancing performance against generalizability so the model does not simply memorize the training data.

Step 6: Model Evaluation

The model is tested on unseen data against a defined set of evaluation metrics. If results are unsatisfactory, the modeling process is repeated until an acceptable standard is reached.

Step 7: Model Deployment

The evaluated model is deployed in its intended format and channel, the final stage of the life cycle.

[!CAUTION]
If any stage is performed improperly, it affects every subsequent stage and the effort invested can be wasted, poor data collection means an unreliable model regardless of how well later stages are executed.

The OSEMN Framework

OSEMN (pronounced "awesome") is a five-stage acronym, Obtain, Scrub, Explore, Model, iNterpret, that maps directly onto the broader project life cycle above and is the specific framework named in this syllabus.

flowchart LR
    A[Obtain] --> B[Scrub]
    B --> C[Explore]
    C --> D[Model]
    D --> E[iNterpret]

Obtain

Data is collected from credible sources, internal channels or external ones, depending on the business or marketing question being answered.

What happens: Querying databases, calling APIs, downloading files
Common tools: SQL, HTTP clients, web scraping libraries

Scrub

Data seldom arrives in a form ready for analysis. It typically contains conflicting, erroneous, or superfluous values, effectively noise, that must be moved from a dirty state to a clean, usable one.

Consider a customer table with mixed date formats before cleaning.

String rawDate = "12/01/2026";
LocalDate parsed = LocalDate.parse(rawDate,
    DateTimeFormatter.ofPattern("dd/MM/yyyy"));

Standardizing the date format before analysis prevents a silent bug where 12/01 is read as December 1st in one row and January 12th in another.

[!CAUTION]
Scrubbing typically consumes more project time than every other stage combined. Underestimating cleaning time is one of the most common estimation failures in data science projects.

Explore

The cleaned data is investigated using multiple methodologies to surface noteworthy statistical patterns and anomalies beneath the surface.

Univariate analysis: Examining one variable at a time, distributions, means, outliers
Bivariate analysis: Examining relationships between two variables, correlation, scatter plots
Visualization: Histograms, box plots, and heatmaps to surface what summary statistics hide

Model

A statistical technique is used to better understand which method best predicts unknown output values from known input data.

// Simplified linear regression coefficient calculation
double slope = covariance(x, y) / variance(x);
double intercept = meanY - slope * meanX;

Choosing the appropriate model for the question, and validating that it generalizes rather than memorizes the training data, is typically the harder part of this stage.

iNterpret

The questions that originally prompted the analysis are revisited, and the actionable value of the investigation is communicated through data visualizations, along with an honest assessment of confidence in the results.

Technical result: "The model achieved 0.89 AUC"
Interpreted result: "This model catches 89% of fraudulent transactions before they clear, saving an estimated 2 crore annually"

Why the Order Matters

OSEMN is not strictly linear in practice. Most real projects loop back to Scrub after Explore reveals a problem missed initially. The framework describes the stages that must occur, not a guarantee that each occurs exactly once.