Python has become a default choice for data analysis largely because of pandas, a library that makes tabular data straightforward to load, clean and summarise. This post covers the operations that make up most day-to-day analysis work.

Setting up

The core libraries for analysis are pandas for data handling, NumPy for numerical operations, and Matplotlib or Seaborn for plotting:

pip install pandas numpy matplotlib seaborn

Working in a notebook environment such as Jupyter or Google Colab is useful while exploring, because you can run a step, look at the output, and adjust before moving on.

Loading data

pandas reads most common formats directly into a DataFrame — a two-dimensional table with labelled rows and columns:

import pandas as pd

df = pd.read_csv("data.csv")
df = pd.read_excel("data.xlsx")
df = pd.read_json("data.json")

First look at the data

Before any analysis, understand the shape and quality of what you have loaded:

df.head()          # first five rows
df.shape           # (rows, columns)
df.info()          # column types and non-null counts
df.describe()      # summary statistics for numeric columns
df.isnull().sum()  # missing values per column

df.info() is the most informative of these. It shows whether a column pandas has typed as object should actually be numeric or a date — a common cause of confusing results later.

Selecting and filtering

# a single column
df["city"]

# several columns
df[["city", "price"]]

# rows matching a condition
df[df["price"] > 100000]

# multiple conditions - note the parentheses
df[(df["price"] > 100000) & (df["city"] == "Kathmandu")]

The parentheses around each condition are required. Without them Python's operator precedence produces an error that is not obvious from the message.

Cleaning

Real data is rarely tidy. The usual tasks are handling missing values, fixing types and removing duplicates:

# drop rows with any missing value
df = df.dropna()

# or fill with a sensible default
df["price"] = df["price"].fillna(df["price"].median())

# convert a column to a proper type
df["date"] = pd.to_datetime(df["date"])
df["price"] = pd.to_numeric(df["price"], errors="coerce")

# remove duplicate rows
df = df.drop_duplicates()

Choosing between dropping and filling is a judgement call. Dropping rows loses data; filling introduces values that were never measured. Whichever you choose, record the decision — it affects how the results should be read.

Grouping and aggregating

Most analytical questions are some form of "what is the average X per Y":

# mean price by city
df.groupby("city")["price"].mean()

# several statistics at once
df.groupby("city")["price"].agg(["count", "mean", "median", "max"])

# group by more than one column
df.groupby(["city", "property_type"])["price"].mean()

Sorting and saving

df.sort_values("price", ascending=False).head(10)

df.to_csv("cleaned.csv", index=False)

Passing index=False prevents pandas writing its row index as an extra unnamed column — a small detail that causes confusion when the file is opened elsewhere.

A sensible workflow

  1. Load the data and inspect its shape and types
  2. Check for missing values and duplicates
  3. Clean and convert types, recording the decisions made
  4. Explore with grouping and summary statistics
  5. Visualise the patterns worth communicating
  6. Save the cleaned dataset separately from the raw file

Always keep the raw file unchanged. Being able to re-run the cleaning from the original is what makes an analysis reproducible.