Skip to main content

10.2 - Pandas for Data Analysis

Theory 35 min Advanced

What is Pandas?​

Pandas provides two main data structures:

  • Series: 1D labeled array (like a column)
  • DataFrame: 2D labeled table (like a spreadsheet)
import pandas as pd

# Series
s = pd.Series([1, 2, 3, 4], index=["a", "b", "c", "d"])
s["b"]   # 2

# DataFrame
df = pd.DataFrame({
    "name": ["Alice", "Bob", "Charlie"],
    "age": [30, 25, 35],
    "salary": [95000, 72000, 88000]
})

Loading Data​

# CSV
df = pd.read_csv("employees.csv")
df = pd.read_csv("data.csv", sep=";", encoding="utf-8",
                  parse_dates=["date_column"])

# JSON
df = pd.read_json("data.json")

# Excel
df = pd.read_excel("report.xlsx", sheet_name="Sheet1")

# From dict/list
df = pd.DataFrame(records)   # list of dicts
df = pd.DataFrame.from_dict(data)

Exploring Data​

df.head()          # first 5 rows
df.tail(10)        # last 10 rows
df.shape           # (rows, cols)
df.dtypes          # data types per column
df.info()          # summary: types, nulls, memory
df.describe()      # statistics: count, mean, std, min, max

# Check missing values
df.isnull().sum()  # null count per column
df.isnull().any()  # True if any nulls

Selecting Data​

# Column selection
df["name"]                     # Series
df[["name", "salary"]]         # DataFrame (double brackets)

# Row selection by position
df.iloc[0]                     # first row
df.iloc[0:3]                   # rows 0-2
df.iloc[0:3, 1:3]              # rows 0-2, cols 1-2

# Row selection by label
df.loc[df["age"] > 30]         # filter by condition
df.loc[0:2, ["name", "age"]]   # rows 0-2, specific cols

Filtering​

# Single condition
senior = df[df["age"] > 30]

# Multiple conditions (use & | ~, NOT and/or)
filtered = df[(df["age"] > 25) & (df["salary"] > 80000)]
filtered = df[(df["dept"] == "Eng") | (df["salary"] > 90000)]
excluded = df[~df["name"].str.contains("Bob")]

# isin
df[df["dept"].isin(["Engineering", "Marketing"])]

# between
df[df["salary"].between(70000, 100000)]

# Query string
df.query("age > 30 and salary > 80000")

Data Cleaning​

# Handle missing values
df.dropna()                    # drop rows with any NaN
df.dropna(subset=["salary"])   # drop only if salary is NaN
df.fillna(0)                   # replace NaN with 0
df["salary"].fillna(df["salary"].mean(), inplace=True)

# Remove duplicates
df.drop_duplicates()
df.drop_duplicates(subset=["email"])

# Rename columns
df.rename(columns={"name": "full_name", "dept": "department"})

# Change dtypes
df["salary"] = df["salary"].astype(float)
df["date"] = pd.to_datetime(df["date"])

# String operations
df["name"].str.lower()
df["name"].str.strip()
df["email"].str.contains("@gmail")

Grouping and Aggregation​

# groupby — like SQL GROUP BY
dept_stats = df.groupby("department")["salary"].mean()

# Multiple aggregations
df.groupby("department").agg({
    "salary": ["mean", "min", "max", "count"],
    "age": "mean"
})

# Named aggregation
df.groupby("department").agg(
    avg_salary=("salary", "mean"),
    headcount=("name", "count"),
    max_salary=("salary", "max")
)

# Value counts
df["department"].value_counts()

Sorting and Ranking​

df.sort_values("salary", ascending=False)
df.sort_values(["department", "salary"], ascending=[True, False])

df["rank"] = df["salary"].rank(ascending=False)

Apply and Transform​

# Apply a function to each row or column
df["tax"] = df["salary"].apply(lambda x: x * 0.3 if x > 80000 else x * 0.2)

def categorize_salary(salary):
    if salary >= 90000: return "High"
    if salary >= 70000: return "Medium"
    return "Low"

df["level"] = df["salary"].apply(categorize_salary)

# Apply to entire DataFrame
df.apply(pd.to_numeric, errors="coerce")

Key Vocabulary​

TermDefinition
DataFrame2D labeled data table — rows and columns
Series1D labeled array — one column
ilocInteger-location based indexing (position)
locLabel-based indexing (row/column names)
groupby()Group rows by a column for aggregation
agg()Apply multiple aggregation functions
apply()Apply a function element-wise or row/column-wise
NaNNot a Number — pandas' representation of missing values

Summary​

  • pd.read_csv() and pd.DataFrame() are the main entry points
  • df.head(), df.info(), df.describe() give instant data overview
  • Filter with boolean conditions using &, |, ~ (not and/or)
  • groupby().agg() is the most powerful analysis pattern
  • Clean data with dropna(), fillna(), drop_duplicates()
  • apply() applies any function to column/row values