Decision Tree

What is a decision tree?

A decision tree classifies things by asking a sequence of yes/no questions — exactly like a flowchart or the game “20 questions”:

Is sweetness < 4.2?
├─ yes → Apple
└─ no  → Is size < 5.1?
         ├─ yes → Banana
         └─ no  → Cherry

The machine-learning part is that the tree learns the questions by itself from labelled examples.

In the 3D model, each dot on the floor is a fruit with two features (sweetness, size) and its true type (colour). Every question becomes a wall that cuts the floor, and the tree diagram grows above.

How a split is chosen — Gini impurity

For a group of examples, the Gini impurity is

Gini = 1 − Σ (pₖ)²

where pₖ is the fraction of class k in the group. A pure group (all one class) has Gini 0; a 50/50 mix of two classes has 0.5.

To grow a node, the algorithm tries every feature and every threshold between neighbouring values, splits the group in two, and computes the weighted Gini of the children. It picks the split with the lowest weighted impurity — the biggest information gain. Then it repeats on each side (recursion).

When to stop

A node becomes a leaf (predicting its majority class) when:

• it is pure, or
• the maximum depth is reached, or
• it has too few examples, or
• no split improves impurity enough.

Without such limits the tree keeps splitting until every leaf has one point — it memorises the training data, including its noise. That’s overfitting.

Code

from sklearn.tree import DecisionTreeClassifier, export_text

# [sweetness, size] → fruit
X = [[2, 3], [3, 7], [1, 5], [6, 2], [7, 3], [8, 4], [6, 8], [7, 9], [9, 7]]
y = ["apple", "apple", "apple", "banana", "banana", "banana", "cherry", "cherry", "cherry"]

tree = DecisionTreeClassifier(criterion="gini", max_depth=3).fit(X, y)
print(export_text(tree, feature_names=["sweetness", "size"]))
print(tree.predict([[7, 2]]))   # ['banana']

A tiny Gini calculator to check the numbers yourself:

from collections import Counter

def gini(labels):
    n = len(labels)
    return 1 - sum((c / n) ** 2 for c in Counter(labels).values())

print(gini(["apple"] * 5))                    # 0.0  (pure)
print(gini(["apple", "banana"] * 3))          # 0.5  (50/50 mix)

Strengths and weaknesses

Random forests and boosting

Combining many trees fixes most weaknesses:

• Random forest: train hundreds of trees on random samples of the data and features; let them vote.
• Gradient boosting (XGBoost, LightGBM): build trees one after another, each fixing the previous ones’ mistakes. These win many tabular-data competitions.

Where are decision trees used?

Loan approval, medical triage, customer churn prediction, fraud detection — anywhere decisions must be explainable.

Common mistakes

• Letting the tree grow without limits (overfitting).
• Trusting a single tree’s feature importances too much.
• Forgetting that entropy / information gain is an alternative to Gini (results are usually similar).