Item Co-Occurrence Analysis (Market Basket Exploration)¶
Objective¶
The goal of this analysis is to identify meaningful relationships between products based on transaction-level purchase data. Using synthetic retail data (to ensure confidentiality), this notebook explores how frequently items are purchased together and quantifies the strength of their associations.
Approach¶
The analysis follows a structured workflow:
- Data preparation and transaction grouping
- Construction of item–item co-occurrence matrices
- Calculation of key association metrics:
- Support
- Confidence
- Lift
- Jaccard similarity
- Visualization of relationships through:
- Heatmaps
- Network graphs
- Item-level co-occurrence breakdowns
The Purpose¶
Co-occurrence analysis has direct applications in:
- Product recommendation systems
- Cross-selling strategies
- Store layout optimization
- Bundling and promotional campaigns
- Demand forecasting and inventory planning
By quantifying item relationships, businesses can move beyond intuition and design data-driven strategies that improve customer experience and increase revenue.
Key Takeaway¶
This notebook demonstrates how relatively simple statistical methods, when structured correctly and paired with strong visualization, can generate actionable insights about customer purchasing behavior. While synthetic data is used here for safety, the methodology mirrors what can be applied directly to real retail or e-commerce datasets.
All data used in this analysis is synthetic and generated for demonstration purposes.
In [1]:
import pandas as pd
import numpy as np
from itertools import combinations
from collections import Counter
In [12]:
# --- Dataset contains 5000 rows of synthetic sales data ---
data = pd.read_csv("synthetic_sales_data__co-occurence.csv")
In [13]:
data.head()
Out[13]:
| Date | Document No | Item No | Sales Amt | Sales Qty | |
|---|---|---|---|---|---|
| 0 | 3/27/2025 0:00 | PS-INV12000 | 010-55602 | $420.00 | 2.0 |
| 1 | 3/27/2025 0:00 | PS-INV12000 | 009-44513 | $1,875.00 | 1.0 |
| 2 | 3/27/2025 0:00 | PS-INV12000 | 011-66701 | $6,900.00 | 2.0 |
| 3 | 2/27/2025 0:00 | PS-INV12004 | 007-22343 | $4,350.00 | 3.0 |
| 4 | 2/27/2025 0:00 | PS-INV12004 | 012-77802 | $1,350.00 | 2.0 |
In [14]:
# --- 0) Keep only what we need; normalize types ---
df = data.rename(columns=lambda c: c.strip()) # trim any stray spaces in headers
df = df[['Document No', 'Item No', 'Date']].copy()
df['Item No'] = df['Item No'].astype(str).str.strip()
df['Document No'] = df['Document No'].astype(str).str.strip()
# (optional) if Date is a string:
df['Date'] = pd.to_datetime(df['Date'], errors='coerce')
In [15]:
# --- 1) One row per (document, item) so duplicates on the same ticket don't inflate pairs ---
ticket_items = df.dropna(subset=['Document No', 'Item No']).drop_duplicates(['Document No', 'Item No'])
In [16]:
# Useful stats
total_docs = ticket_items['Document No'].nunique()
items_per_doc = ticket_items.groupby('Document No')['Item No'].nunique()
print(f"Total tickets: {total_docs} | avg items/ticket: {items_per_doc.mean():.2f}")
Total tickets: 1882 | avg items/ticket: 2.66
In [17]:
# --- 2) Build item pairs per document and count across documents ---
pair_counter = Counter()
for items in ticket_items.groupby('Document No')['Item No'].apply(list):
uniq = sorted(set(items))
if len(uniq) >= 2:
pair_counter.update(combinations(uniq, 2)) # unordered pairs A<B
pairs_df = (
pd.DataFrame([(a,b,cnt) for (a,b), cnt in pair_counter.items()],
columns=['ItemA','ItemB','CoDocCount'])
.sort_values('CoDocCount', ascending=False)
.reset_index(drop=True)
)
In [18]:
# --- 3) Per-item document frequency (for confidence/lift) ---
item_doc_counts = (
ticket_items.groupby('Item No')['Document No'].nunique()
.rename('DocsWithItem')
)
pairs_df = (
pairs_df
.merge(item_doc_counts.rename('DocsA'), left_on='ItemA', right_index=True)
.merge(item_doc_counts.rename('DocsB'), left_on='ItemB', right_index=True)
)
In [19]:
# Support, Confidence, Lift
pairs_df['Support'] = pairs_df['CoDocCount'] / total_docs
pairs_df['Conf(A→B)'] = pairs_df['CoDocCount'] / pairs_df['DocsA']
pairs_df['Conf(B→A)'] = pairs_df['CoDocCount'] / pairs_df['DocsB']
pairs_df['Lift'] = (
pairs_df['CoDocCount'] / total_docs
) / (
(pairs_df['DocsA']/total_docs) * (pairs_df['DocsB']/total_docs)
)
In [20]:
# --- 4) Top N "frequently bought together" pairs (overall) ---
top_pairs = pairs_df.sort_values(['CoDocCount','Lift'], ascending=False).head(100)
top_pairs
Out[20]:
| ItemA | ItemB | CoDocCount | DocsA | DocsB | Support | Conf(A→B) | Conf(B→A) | Lift | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 012-77801 | 012-77802 | 120 | 263 | 255 | 0.063762 | 0.456274 | 0.470588 | 3.367479 |
| 1 | 007-22343 | 012-77802 | 114 | 246 | 255 | 0.060574 | 0.463415 | 0.447059 | 3.420182 |
| 2 | 007-22341 | 010-55601 | 111 | 254 | 340 | 0.058980 | 0.437008 | 0.326471 | 2.418967 |
| 3 | 009-44512 | 010-55603 | 111 | 230 | 415 | 0.058980 | 0.482609 | 0.267470 | 2.188601 |
| 4 | 010-55602 | 010-55603 | 110 | 335 | 415 | 0.058448 | 0.328358 | 0.265060 | 1.489085 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 92 | 008-33422 | 010-55602 | 16 | 349 | 335 | 0.008502 | 0.045845 | 0.047761 | 0.257555 |
| 83 | 008-33422 | 010-55601 | 16 | 349 | 340 | 0.008502 | 0.045845 | 0.047059 | 0.253767 |
| 108 | 006-01512 | 008-33424 | 15 | 195 | 115 | 0.007970 | 0.076923 | 0.130435 | 1.258863 |
| 99 | 008-33424 | 009-44512 | 15 | 115 | 230 | 0.007970 | 0.130435 | 0.065217 | 1.067297 |
| 106 | 007-22341 | 011-66702 | 15 | 254 | 148 | 0.007970 | 0.059055 | 0.101351 | 0.750958 |
100 rows × 9 columns
In [21]:
def co_items(target_item, k=20, sort_by='CoDocCount'):
sub = pairs_df[(pairs_df['ItemA']==target_item) | (pairs_df['ItemB']==target_item)].copy()
if sub.empty:
return sub
# Put the "other" item in one column
sub['Other'] = np.where(sub['ItemA']==target_item, sub['ItemB'], sub['ItemA'])
# Pick the directional confidence that matches the target
sub['Confidence'] = np.where(
sub['ItemA']==target_item, sub['Conf(A→B)'], sub['Conf(B→A)']
)
cols = ['Other','CoDocCount','Confidence','Lift','DocsA','DocsB','ItemA','ItemB']
return sub.sort_values([sort_by,'Lift'], ascending=False)[cols].head(k)
In [22]:
co_items('012-77801', k=20)
Out[22]:
| Other | CoDocCount | Confidence | Lift | DocsA | DocsB | ItemA | ItemB | |
|---|---|---|---|---|---|---|---|---|
| 0 | 012-77802 | 120 | 0.456274 | 3.367479 | 263 | 255 | 012-77801 | 012-77802 |
| 6 | 007-22343 | 108 | 0.410646 | 3.141612 | 246 | 263 | 007-22343 | 012-77801 |
| 28 | 006-01511 | 25 | 0.095057 | 0.553862 | 323 | 263 | 006-01511 | 012-77801 |
| 26 | 010-55603 | 25 | 0.095057 | 0.431078 | 415 | 263 | 010-55603 | 012-77801 |
| 41 | 009-44513 | 21 | 0.079848 | 0.661999 | 227 | 263 | 009-44513 | 012-77801 |
| 47 | 010-55602 | 20 | 0.076046 | 0.427218 | 335 | 263 | 010-55602 | 012-77801 |
| 49 | 010-55601 | 19 | 0.072243 | 0.399888 | 340 | 263 | 010-55601 | 012-77801 |
| 78 | 007-22342 | 17 | 0.064639 | 0.538275 | 226 | 263 | 007-22342 | 012-77801 |
| 110 | 011-66702 | 15 | 0.057034 | 0.725259 | 148 | 263 | 011-66702 | 012-77801 |
| 102 | 007-22341 | 15 | 0.057034 | 0.422592 | 254 | 263 | 007-22341 | 012-77801 |
| 112 | 008-33422 | 15 | 0.057034 | 0.307560 | 349 | 263 | 008-33422 | 012-77801 |
| 130 | 008-33424 | 14 | 0.053232 | 0.871152 | 115 | 263 | 008-33424 | 012-77801 |
| 133 | 008-33421 | 13 | 0.049430 | 0.288903 | 322 | 263 | 008-33421 | 012-77801 |
| 159 | 006-01513 | 12 | 0.045627 | 0.698136 | 123 | 263 | 006-01513 | 012-77801 |
| 152 | 006-01512 | 12 | 0.045627 | 0.440363 | 195 | 263 | 006-01512 | 012-77801 |
| 156 | 003-16146 | 12 | 0.045627 | 0.440363 | 195 | 263 | 003-16146 | 012-77801 |
| 169 | 009-44512 | 11 | 0.041825 | 0.342238 | 230 | 263 | 009-44512 | 012-77801 |
| 177 | 011-66701 | 10 | 0.038023 | 0.347373 | 206 | 263 | 011-66701 | 012-77801 |
| 187 | 008-33423 | 8 | 0.030418 | 0.245696 | 233 | 263 | 008-33423 | 012-77801 |
In [23]:
min_docs = 5 # tweakable
popular = item_doc_counts[item_doc_counts >= min_docs].index
ticket_items_pop = ticket_items[ticket_items['Item No'].isin(popular)]
# then rebuild the pair_counter using ticket_items_pop instead of ticket_items
In [24]:
top_pairs.to_csv("top_item_pairs_q1.csv", index=False)
In [25]:
co_items('012-77801').to_csv('co_items_012-77801_q1.csv', index=False)
In [26]:
# ─────────────────────────────────────────────
# VIZ 1 — Co-occurrence Heatmap (20×20)
# ─────────────────────────────────────────────
import matplotlib.pyplot as plt
import seaborn as sns
# Build a symmetric item×item matrix filled with Lift values
all_items = sorted(item_doc_counts.index.tolist())
heat_df = pd.DataFrame(0.0, index=all_items, columns=all_items)
for _, row in pairs_df.iterrows():
a, b, lift = row['ItemA'], row['ItemB'], row['Lift']
if a in heat_df.index and b in heat_df.columns:
heat_df.loc[a, b] = lift
heat_df.loc[b, a] = lift # mirror
# Diagonal = 1 (an item always co-occurs with itself)
for item in all_items:
heat_df.loc[item, item] = 1.0
fig, ax = plt.subplots(figsize=(13, 11))
sns.heatmap(
heat_df,
annot=True, fmt='.2f',
cmap='YlOrRd',
linewidths=0.5,
linecolor='#eeeeee',
cbar_kws={'label': 'Lift'},
ax=ax
)
ax.set_title('Item Co-occurrence Heatmap — Lift', fontsize=15, pad=14)
ax.set_xlabel('Item No', fontsize=11)
ax.set_ylabel('Item No', fontsize=11)
ax.tick_params(axis='x', rotation=45, labelsize=9)
ax.tick_params(axis='y', rotation=0, labelsize=9)
plt.tight_layout()
plt.savefig('heatmap_lift.png', dpi=150, bbox_inches='tight')
plt.show()
print('Saved → heatmap_lift.png')
Saved → heatmap_lift.png
In [27]:
# ─────────────────────────────────────────────
# VIZ 2 — Network Graph (filtered by Lift)
# ─────────────────────────────────────────────
import networkx as nx
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import matplotlib.colors as mcolors
# ── Tunable filters ──────────────────────────
MIN_LIFT = 1.5 # only draw edges where lift > this
MIN_CODOCOUNT = 20 # and co-doc count > this (removes rare pairs)
# ─────────────────────────────────────────────
filtered = pairs_df[
(pairs_df['Lift'] >= MIN_LIFT) &
(pairs_df['CoDocCount'] >= MIN_CODOCOUNT)
].copy()
G = nx.Graph()
for _, row in filtered.iterrows():
G.add_edge(
row['ItemA'], row['ItemB'],
weight=row['CoDocCount'],
lift=row['Lift']
)
# Node size = how many documents contain that item
node_sizes = [
item_doc_counts.get(n, 1) * 3
for n in G.nodes()
]
# Edge width = CoDocCount (normalised)
max_w = max(d['weight'] for _, _, d in G.edges(data=True))
edge_widths = [d['weight'] / max_w * 8 for _, _, d in G.edges(data=True)]
# Edge color = Lift
lifts = [d['lift'] for _, _, d in G.edges(data=True)]
norm = mcolors.Normalize(vmin=min(lifts), vmax=max(lifts))
cmap = cm.get_cmap('RdYlGn')
edge_colors = [cmap(norm(l)) for l in lifts]
pos = nx.spring_layout(G, seed=42, k=2.5)
fig, ax = plt.subplots(figsize=(14, 10))
nx.draw_networkx_nodes(G, pos, node_size=node_sizes,
node_color='steelblue', alpha=0.85, ax=ax)
nx.draw_networkx_labels(G, pos, font_size=8, font_color='white',
font_weight='bold', ax=ax)
nx.draw_networkx_edges(G, pos, width=edge_widths,
edge_color=edge_colors, alpha=0.8, ax=ax)
# Colorbar for lift
sm = plt.cm.ScalarMappable(cmap=cmap, norm=norm)
sm.set_array([])
plt.colorbar(sm, ax=ax, label='Lift', shrink=0.7)
ax.set_title(
f'Item Co-occurrence Network (Lift ≥ {MIN_LIFT}, CoDocCount ≥ {MIN_CODOCOUNT})\n'
f'Node size = item frequency | Edge width = co-doc count | Edge color = lift',
fontsize=12, pad=12
)
ax.axis('off')
plt.tight_layout()
plt.savefig('network_graph.png', dpi=150, bbox_inches='tight')
plt.show()
print(f'Nodes: {G.number_of_nodes()} | Edges: {G.number_of_edges()}')
print('Saved → network_graph.png')
/var/folders/wf/rz7j7ttx3g3408q18k2p97y40000gn/T/ipykernel_1474/909010283.py:40: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed two minor releases later. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap(obj)`` instead.
cmap = cm.get_cmap('RdYlGn')
/var/folders/wf/rz7j7ttx3g3408q18k2p97y40000gn/T/ipykernel_1474/909010283.py:50: DeprecationWarning: `alltrue` is deprecated as of NumPy 1.25.0, and will be removed in NumPy 2.0. Please use `all` instead.
nx.draw_networkx_edges(G, pos, width=edge_widths,
Nodes: 17 | Edges: 20 Saved → network_graph.png
In [28]:
# ─────────────────────────────────────────────
# VIZ 3 — Per-item co-occurrence bar chart
# Drop-in replacement / companion to co_items()
# ─────────────────────────────────────────────
import matplotlib.pyplot as plt
def plot_co_items(target_item, k=15, sort_by='Lift', min_codocount=5):
"""
Horizontal bar chart of the top-k items that co-occur with target_item.
Parameters
----------
target_item : str — the item you want to investigate
k : int — how many co-items to show
sort_by : str — 'Lift', 'CoDocCount', or 'Confidence'
min_codocount : int — hide pairs with fewer co-documents than this
"""
sub = co_items(target_item, k=50, sort_by=sort_by) # fetch wide, trim below
if sub.empty:
print(f'No co-occurrence data found for {target_item}')
return
sub = sub[sub['CoDocCount'] >= min_codocount].head(k)
if sub.empty:
print(f'No pairs survive the min_codocount={min_codocount} filter.')
return
sub = sub.sort_values(sort_by, ascending=True) # ascending so top bar is highest
colors = ['#d73027' if l >= 2 else '#fc8d59' if l >= 1 else '#91bfdb'
for l in sub['Lift']]
fig, ax = plt.subplots(figsize=(9, max(4, len(sub) * 0.45)))
bars = ax.barh(sub['Other'], sub[sort_by], color=colors, edgecolor='white', height=0.65)
# Annotate with CoDocCount and Lift
for bar, (_, row) in zip(bars, sub.iterrows()):
ax.text(
bar.get_width() + bar.get_width() * 0.01,
bar.get_y() + bar.get_height() / 2,
f" n={int(row['CoDocCount'])} lift={row['Lift']:.2f}",
va='center', fontsize=8, color='#333333'
)
ax.set_xlabel(sort_by, fontsize=11)
ax.set_ylabel('Co-occurring Item', fontsize=11)
ax.set_title(f'Top co-occurring items with {target_item} (sorted by {sort_by})',
fontsize=12, pad=10)
# Legend
from matplotlib.patches import Patch
legend_elements = [
Patch(facecolor='#d73027', label='Lift >= 2 (strong)'),
Patch(facecolor='#fc8d59', label='1 <= Lift < 2 (moderate)'),
Patch(facecolor='#91bfdb', label='Lift < 1 (negative assoc.)'),
]
ax.legend(handles=legend_elements, loc='lower right', fontsize=8)
ax.spines[['top', 'right']].set_visible(False)
plt.tight_layout()
fname = f'co_items_{target_item.replace("-","_")}.png'
plt.savefig(fname, dpi=150, bbox_inches='tight')
plt.show()
print(f'Saved → {fname}')
# ── Example calls ─────────────────────────────
plot_co_items('012-77801', sort_by='Lift')
# plot_co_items('007-22341', sort_by='CoDocCount')
# plot_co_items('008-33421', sort_by='Confidence')
Saved → co_items_012_77801.png
In [29]:
# ─────────────────────────────────────────────────────────────
# ANALYTICS 5 — Jaccard Similarity
# More balanced than Lift when item frequencies are unequal.
# Formula: CoDocCount / (DocsA + DocsB - CoDocCount)
# Range: 0 (never together) → 1 (always together)
# ─────────────────────────────────────────────────────────────
pairs_df['Jaccard'] = (
pairs_df['CoDocCount'] /
(pairs_df['DocsA'] + pairs_df['DocsB'] - pairs_df['CoDocCount'])
)
# Show top pairs ranked by Jaccard — compare with Lift ranking
print('=== Top 20 pairs by Jaccard ===')
display(
pairs_df[['ItemA','ItemB','CoDocCount','DocsA','DocsB','Jaccard','Lift']]
.sort_values('Jaccard', ascending=False)
.head(20)
.reset_index(drop=True)
)
# Pairs where Lift and Jaccard rankings disagree most
# (high lift but low Jaccard = one item is very rare, inflating lift)
pairs_df['LiftRank'] = pairs_df['Lift'].rank(ascending=False)
pairs_df['JaccardRank'] = pairs_df['Jaccard'].rank(ascending=False)
pairs_df['RankDiff'] = (pairs_df['LiftRank'] - pairs_df['JaccardRank']).abs()
print('\n=== Pairs where Lift and Jaccard rankings disagree most ===')
display(
pairs_df[['ItemA','ItemB','CoDocCount','DocsA','DocsB','Lift','Jaccard','RankDiff']]
.sort_values('RankDiff', ascending=False)
.head(10)
.reset_index(drop=True)
)
=== Top 20 pairs by Jaccard ===
| ItemA | ItemB | CoDocCount | DocsA | DocsB | Jaccard | Lift | |
|---|---|---|---|---|---|---|---|
| 0 | 012-77801 | 012-77802 | 120 | 263 | 255 | 0.301508 | 3.367479 |
| 1 | 007-22343 | 012-77802 | 114 | 246 | 255 | 0.294574 | 3.420182 |
| 2 | 009-44513 | 011-66701 | 96 | 227 | 206 | 0.284866 | 3.863650 |
| 3 | 007-22343 | 012-77801 | 108 | 246 | 263 | 0.269327 | 3.141612 |
| 4 | 007-22341 | 007-22342 | 101 | 254 | 226 | 0.266491 | 3.311302 |
| 5 | 008-33422 | 008-33423 | 109 | 349 | 233 | 0.230444 | 2.522695 |
| 6 | 007-22341 | 010-55601 | 111 | 254 | 340 | 0.229814 | 2.418967 |
| 7 | 009-44513 | 010-55602 | 101 | 227 | 335 | 0.219089 | 2.499599 |
| 8 | 003-16146 | 010-55601 | 96 | 195 | 340 | 0.218679 | 2.725068 |
| 9 | 008-33421 | 008-33423 | 97 | 322 | 233 | 0.211790 | 2.433210 |
| 10 | 007-22342 | 010-55601 | 98 | 226 | 340 | 0.209402 | 2.400260 |
| 11 | 009-44512 | 010-55603 | 111 | 230 | 415 | 0.207865 | 2.188601 |
| 12 | 009-44512 | 010-55602 | 97 | 230 | 335 | 0.207265 | 2.369293 |
| 13 | 010-55602 | 011-66701 | 92 | 335 | 206 | 0.204900 | 2.508970 |
| 14 | 006-01512 | 008-33421 | 82 | 195 | 322 | 0.188506 | 2.457780 |
| 15 | 008-33421 | 008-33422 | 104 | 322 | 349 | 0.183422 | 1.741693 |
| 16 | 006-01511 | 008-33421 | 99 | 323 | 322 | 0.181319 | 1.791416 |
| 17 | 010-55602 | 010-55603 | 110 | 335 | 415 | 0.171875 | 1.489085 |
| 18 | 006-01511 | 006-01512 | 74 | 323 | 195 | 0.166667 | 2.211130 |
| 19 | 006-01511 | 008-33422 | 95 | 323 | 349 | 0.164645 | 1.586044 |
=== Pairs where Lift and Jaccard rankings disagree most ===
| ItemA | ItemB | CoDocCount | DocsA | DocsB | Lift | Jaccard | RankDiff | |
|---|---|---|---|---|---|---|---|---|
| 0 | 003-16146 | 008-33424 | 7 | 195 | 115 | 0.587469 | 0.023102 | 84.5 |
| 1 | 008-33424 | 011-66701 | 9 | 115 | 206 | 0.714985 | 0.028846 | 72.0 |
| 2 | 008-33421 | 010-55603 | 23 | 322 | 415 | 0.323924 | 0.032213 | 64.0 |
| 3 | 006-01511 | 010-55601 | 24 | 323 | 340 | 0.411291 | 0.037559 | 63.0 |
| 4 | 010-55603 | 012-77801 | 25 | 415 | 263 | 0.431078 | 0.038285 | 62.0 |
| 5 | 008-33422 | 008-33424 | 12 | 349 | 115 | 0.562701 | 0.026549 | 62.0 |
| 6 | 010-55601 | 010-55602 | 22 | 340 | 335 | 0.363512 | 0.033691 | 60.0 |
| 7 | 006-01511 | 006-01513 | 10 | 323 | 123 | 0.473709 | 0.022936 | 54.0 |
| 8 | 006-01513 | 010-55601 | 11 | 123 | 340 | 0.495026 | 0.024336 | 52.0 |
| 9 | 008-33421 | 010-55602 | 21 | 322 | 335 | 0.366385 | 0.033019 | 51.5 |
In [30]:
# ─────────────────────────────────────────────────────────────
# ANALYTICS 6 — Monthly Trend of Top Pairs
# Are strongest bundles stable over time, or seasonal?
# ─────────────────────────────────────────────────────────────
import matplotlib.pyplot as plt
import matplotlib.ticker as mticker
# ── Tunable ───────────────────────────────────
TOP_N_PAIRS = 6 # how many pairs to track
# ──────────────────────────────────────────────
# Pick the top N pairs by overall CoDocCount
top_pair_labels = (
pairs_df.sort_values('CoDocCount', ascending=False)
.head(TOP_N_PAIRS)
.apply(lambda r: f"{r['ItemA']} + {r['ItemB']}", axis=1)
.tolist()
)
top_pair_tuples = [
(label.split(' + ')[0], label.split(' + ')[1])
for label in top_pair_labels
]
# Add YearMonth to the base ticket_items dataframe
ti = ticket_items.copy()
ti['YearMonth'] = ti['Date'].dt.to_period('M')
# For each month, count how many documents contain each pair
monthly_records = []
for ym, grp in ti.groupby('YearMonth'):
doc_items_month = grp.groupby('Document No')['Item No'].apply(set)
for (a, b), label in zip(top_pair_tuples, top_pair_labels):
count = sum(1 for items in doc_items_month if a in items and b in items)
monthly_records.append({'YearMonth': ym, 'Pair': label, 'CoDocCount': count})
monthly_df = pd.DataFrame(monthly_records)
monthly_pivot = monthly_df.pivot(index='YearMonth', columns='Pair', values='CoDocCount').fillna(0)
monthly_pivot.index = monthly_pivot.index.astype(str)
fig, ax = plt.subplots(figsize=(13, 5))
for col in monthly_pivot.columns:
ax.plot(monthly_pivot.index, monthly_pivot[col], marker='o', linewidth=2, label=col)
ax.set_title(f'Monthly Co-occurrence Count — Top {TOP_N_PAIRS} Pairs', fontsize=13, pad=12)
ax.set_xlabel('Month', fontsize=11)
ax.set_ylabel('# Documents containing pair', fontsize=11)
ax.yaxis.set_major_locator(mticker.MaxNLocator(integer=True))
ax.tick_params(axis='x', rotation=45)
ax.legend(fontsize=8, bbox_to_anchor=(1.01, 1), loc='upper left')
ax.spines[['top','right']].set_visible(False)
ax.grid(axis='y', linestyle='--', alpha=0.4)
plt.tight_layout()
plt.savefig('monthly_pair_trend.png', dpi=150, bbox_inches='tight')
plt.show()
print('Saved → monthly_pair_trend.png')
Saved → monthly_pair_trend.png
In [31]:
# ─────────────────────────────────────────────────────────────
# ANALYTICS 7 — 'Frequently Bought Alone' Flag
# Items with low average pair-confidence are solo purchases.
# These are the best candidates for upsell / bundle promotion.
# ─────────────────────────────────────────────────────────────
import matplotlib.pyplot as plt
# ── Tunable ───────────────────────────────────
SOLO_THRESHOLD = 0.15 # avg confidence below this → flagged as solo
# ──────────────────────────────────────────────
# For each item, collect all confidence values where it appears (A→B or B→A)
conf_records = []
for _, row in pairs_df.iterrows():
conf_records.append({'Item No': row['ItemA'], 'Confidence': row['Conf(A→B)']})
conf_records.append({'Item No': row['ItemB'], 'Confidence': row['Conf(B→A)']})
conf_df = pd.DataFrame(conf_records)
avg_conf = (
conf_df.groupby('Item No')['Confidence']
.mean()
.rename('AvgConfidence')
.reset_index()
.merge(item_doc_counts.reset_index().rename(columns={'Item No':'Item No','DocsWithItem':'DocsWithItem'}), on='Item No')
.sort_values('AvgConfidence')
.reset_index(drop=True)
)
avg_conf['SoloBuyer'] = avg_conf['AvgConfidence'] < SOLO_THRESHOLD
print(f'=== Items flagged as frequently bought alone (AvgConfidence < {SOLO_THRESHOLD}) ===')
display(avg_conf[avg_conf['SoloBuyer']][['Item No','DocsWithItem','AvgConfidence']])
# Visualise
colors = ['#e74c3c' if s else '#3498db' for s in avg_conf['SoloBuyer']]
fig, ax = plt.subplots(figsize=(10, 5))
ax.bar(avg_conf['Item No'], avg_conf['AvgConfidence'], color=colors, edgecolor='white')
ax.axhline(SOLO_THRESHOLD, color='black', linestyle='--', linewidth=1.2,
label=f'Solo threshold ({SOLO_THRESHOLD})')
ax.set_xlabel('Item No', fontsize=11)
ax.set_ylabel('Avg Pair Confidence', fontsize=11)
ax.set_title('Average Pair Confidence per Item\n'
'Red = frequently bought alone (upsell candidates)', fontsize=12, pad=10)
ax.tick_params(axis='x', rotation=45, labelsize=8)
ax.spines[['top','right']].set_visible(False)
ax.legend(fontsize=9)
plt.tight_layout()
plt.savefig('solo_buyer_flag.png', dpi=150, bbox_inches='tight')
plt.show()
print('Saved → solo_buyer_flag.png')
=== Items flagged as frequently bought alone (AvgConfidence < 0.15) ===
| Item No | DocsWithItem | AvgConfidence | |
|---|---|---|---|
| 0 | 012-77801 | 263 | 0.098459 |
| 1 | 010-55601 | 340 | 0.098607 |
| 2 | 008-33422 | 349 | 0.100739 |
| 3 | 010-55603 | 415 | 0.100951 |
| 4 | 008-33421 | 322 | 0.101177 |
| 5 | 003-16146 | 195 | 0.102294 |
| 6 | 006-01511 | 323 | 0.102330 |
| 7 | 010-55602 | 335 | 0.102907 |
| 8 | 007-22341 | 254 | 0.103191 |
| 9 | 007-22343 | 246 | 0.104407 |
| 10 | 011-66701 | 206 | 0.104497 |
| 11 | 008-33423 | 233 | 0.104811 |
| 12 | 012-77802 | 255 | 0.105263 |
| 13 | 009-44512 | 230 | 0.105950 |
| 14 | 006-01512 | 195 | 0.106073 |
| 15 | 007-22342 | 226 | 0.107126 |
| 16 | 009-44513 | 227 | 0.107350 |
| 17 | 006-01513 | 123 | 0.108686 |
| 18 | 011-66702 | 148 | 0.109531 |
| 19 | 008-33424 | 115 | 0.114416 |
Saved → solo_buyer_flag.png
In [32]:
# ─────────────────────────────────────────────────────────────
# ANALYTICS 8 — Item Frequency Bar Chart
# Baseline context: how often each item appears across documents
# ─────────────────────────────────────────────────────────────
import matplotlib.pyplot as plt
freq = (
item_doc_counts
.sort_values(ascending=False)
.reset_index()
)
freq.columns = ['Item No', 'DocsWithItem']
freq['PctDocs'] = freq['DocsWithItem'] / total_docs * 100
fig, ax = plt.subplots(figsize=(12, 5))
bars = ax.bar(
freq['Item No'], freq['DocsWithItem'],
color='steelblue', edgecolor='white'
)
# Annotate each bar with % of total documents
for bar, pct in zip(bars, freq['PctDocs']):
ax.text(
bar.get_x() + bar.get_width() / 2,
bar.get_height() + 2,
f'{pct:.1f}%',
ha='center', va='bottom', fontsize=7.5, color='#333'
)
ax.set_xlabel('Item No', fontsize=11)
ax.set_ylabel('# Documents (invoices)', fontsize=11)
ax.set_title('Item Frequency — Documents Containing Each Item\n'
'(% = share of all invoices)', fontsize=12, pad=10)
ax.tick_params(axis='x', rotation=45, labelsize=8)
ax.spines[['top','right']].set_visible(False)
plt.tight_layout()
plt.savefig('item_frequency.png', dpi=150, bbox_inches='tight')
plt.show()
print('Saved → item_frequency.png')
print()
print(freq.to_string(index=False))
Saved → item_frequency.png Item No DocsWithItem PctDocs 010-55603 415 22.051010 008-33422 349 18.544102 010-55601 340 18.065887 010-55602 335 17.800213 006-01511 323 17.162593 008-33421 322 17.109458 012-77801 263 13.974495 012-77802 255 13.549416 007-22341 254 13.496281 007-22343 246 13.071201 008-33423 233 12.380446 009-44512 230 12.221041 009-44513 227 12.061637 007-22342 226 12.008502 011-66701 206 10.945802 003-16146 195 10.361318 006-01512 195 10.361318 011-66702 148 7.863974 006-01513 123 6.535600 008-33424 115 6.110521
In [ ]: