Caizhan Xu

Xuye101.github.io

Q2 Survival Analysis Report: Telco Customer Churn

1. Introduction

This report documents the complete survival analysis process applied to the IBM Telco Customer Churn dataset. The analysis aims to understand customer retention dynamics — specifically, how long customers stay with the telecom provider and which factors influence their likelihood of churning. Survival analysis is the appropriate methodology because it handles time-to-event data and naturally accommodates censored observations (customers who have not yet churned by the end of the observation period).

2. Data Acquisition & Preprocessing

2.1 Dataset Source

The dataset was downloaded from IBM’s public GitHub repository:

URL: https://raw.githubusercontent.com/IBM/telco-customer-churn-on-icp4d/master/data/Telco-Customer-Churn.csv
Original size: 7,043 customers × 21 features

2.2 Data Schema

A predefined PySpark schema was used to enforce correct data types:

Field	Type	Role
`customerID`	String	Identifier
`gender`	String	Demographic
`seniorCitizen`	Double	Demographic
`partner`	String	Demographic
`dependents`	String	Demographic
`tenure`	Double	Time variable (T) — months as customer
`phoneService`	String	Service attribute
`multipleLines`	String	Service attribute
`internetService`	String	Service attribute
`onlineSecurity`	String	Service attribute
`onlineBackup`	String	Service attribute
`deviceProtection`	String	Service attribute
`techSupport`	String	Service attribute
`streamingTV`	String	Service attribute
`streamingMovies`	String	Service attribute
`contract`	String	Contract type
`paperlessBilling`	String	Billing preference
`paymentMethod`	String	Payment preference
`monthlyCharges`	Double	Financial
`totalCharges`	Double	Financial
`churnString`	String	Event indicator — “Yes” or “No”

2.3 Data Curation (Bronze → Silver Layer)

The data was processed through a PySpark pipeline with three key transformations:

Target Encoding: Converted churnString (“Yes”/”No”) into a numeric churn variable (1 = churned/event occurred, 0 = retained/censored).
Contract Filter: Retained only Month-to-month contract customers. This focuses the analysis on the most volatile customer segment where churn risk is highest.
Internet Service Filter: Excluded customers with no internet service, as they represent a fundamentally different product segment.

2.4 Resulting Dataset

Metric	Value
Filtered rows	3,351
Time variable (T)	`tenure` (months)
Event indicator (C)	`churn` (1 = churned, 0 = censored)
Analysis scope	Month-to-month contracts with internet service only

The Spark DataFrame was then converted to Pandas for use with the lifelines survival analysis library.

3. Kaplan-Meier Analysis — Population Level

3.1 Methodology

The Kaplan-Meier (KM) estimator is a non-parametric method that estimates the survival function S(t) — the probability that a customer survives (remains with the company) beyond time t.

T (Duration): tenure — number of months the customer has been subscribed
C (Event/Observation): churn — whether the customer has churned (1) or is censored (0)

3.2 Results

Median Survival Time: 34.0 months

This means that 50% of month-to-month customers with internet service are expected to churn within 34 months of joining. This is a critical benchmark for the business — it represents the “half-life” of their customer base.

3.3 Interpretation

The KM survival curve starts at 1.0 (100% retention at month 0) and declines over time. The curve shows the classic pattern of telecom churn:

Sharp initial decline in the first few months — many customers churn early, suggesting onboarding or first-impression issues
Gradual flattening over time — customers who stay longer become increasingly loyal

4. Survival Analysis by Covariates & Log-Rank Tests

4.1 Methodology

For each categorical feature, we:

Plotted stratified Kaplan-Meier curves (one curve per category)
Performed a pairwise Log-Rank test to assess whether the survival curves differ significantly

The Log-Rank test null hypothesis is: the survival curves of all groups are identical. A p-value < 0.05 indicates statistically significant differences between groups.

4.2 Covariate 1: Gender

KM Curve Observation: The survival curves for Male and Female customers are nearly identical — the two lines overlap almost perfectly throughout the entire observation period.

Log-Rank Test Results:

Statistic	Value
Test Statistic	2.039
p-value	0.153
-log₂(p)	2.705

Conclusion: The p-value (0.153) is greater than 0.05, so we fail to reject the null hypothesis. There is no statistically significant difference in churn behavior between male and female customers. Gender is not a meaningful predictor of churn in this dataset.

4.3 Covariate 2: Online Security

KM Curve Observation: The survival curves for customers with and without Online Security show a clear and substantial separation. Customers who have Online Security service maintain a much higher survival probability over time compared to those who do not.

Log-Rank Test Results:

Statistic	Value
Test Statistic	141.60
p-value	1.19 × 10⁻³²
-log₂(p)	106.05

Conclusion: The p-value is extremely small (1.19 × 10⁻³²), providing overwhelming evidence to reject the null hypothesis. Online Security has a highly significant effect on customer retention. Customers who subscribe to Online Security are substantially less likely to churn.

Business Implication: Promoting and incentivizing Online Security adoption should be a high-priority retention strategy.

5. Cox Proportional Hazards (CPH) — Data Preparation

5.1 Methodology

The Cox Proportional Hazards model is a semi-parametric regression approach that estimates the effect of multiple covariates simultaneously on the hazard rate (instantaneous churn risk). Unlike the KM estimator which examines one variable at a time, CPH allows us to:

Control for confounding variables
Quantify the magnitude of each factor’s effect (via hazard ratios)
Build a predictive model for individual customer churn risk

5.2 Feature Engineering

Selected covariates for the CPH model:

Original Feature	Encoded Variable(s)	Retained for Model
`dependents`	`dependents_Yes`, `dependents_No`	`dependents_Yes` (baseline: No)
`internetService`	`internetService_DSL`, `internetService_Fiber optic`	`internetService_DSL` (baseline: Fiber optic)
`onlineBackup`	`onlineBackup_Yes`, `onlineBackup_No`	`onlineBackup_Yes` (baseline: No)
`techSupport`	`techSupport_Yes`, `techSupport_No`	`techSupport_Yes` (baseline: No)

One-hot encoding was applied to convert categorical variables into binary (0/1) indicators. To avoid multicollinearity (the dummy variable trap), one reference category was dropped from each original feature, serving as the implicit baseline for comparison.

5.3 Final Model Input

The resulting dataset for CPH fitting contains 6 columns:

Column	Type	Description
`churn`	Float (0/1)	Event indicator
`tenure`	Float	Duration in months
`dependents_Yes`	Boolean	Has dependents
`internetService_DSL`	Boolean	Uses DSL internet
`onlineBackup_Yes`	Boolean	Has online backup
`techSupport_Yes`	Boolean	Has tech support

5.4 Status

The notebook’s Block 6 completes the data preparation stage for the CPH model. The actual model fitting (CoxPHFitter().fit()) and coefficient interpretation would follow in a subsequent Block 7 (not present in the current notebook).

6. Summary of Key Findings

Analysis	Key Result	Statistical Significance
Population KM	Median survival = 34.0 months	N/A (descriptive)
Gender (Log-Rank)	No difference in churn behavior	p = 0.153 → Not significant
Online Security (Log-Rank)	Strong retention effect	p = 1.19 × 10⁻³² → Highly significant
CPH Model	Data prepared; model fitting pending	—

Key Insights

Customer half-life is 34 months — half of all month-to-month internet customers churn within this timeframe.
Gender does not matter for churn prediction — marketing and retention strategies need not be gender-differentiated.
Online Security is a powerful retention driver — customers with this service are significantly more likely to stay. This should be a focal point for retention programs.
Early churn risk — the KM curve’s steep initial decline suggests that the first few months are critical for customer retention efforts.

7. Tools & Technologies Used

Component	Technology
Data Processing	PySpark (SparkSession, DataFrame transformations)
Statistical Analysis	lifelines (KaplanMeierFitter, pairwise_logrank_test, CoxPHFitter)
Visualization	matplotlib, seaborn
Data Manipulation	pandas, numpy
Language	Python 3.12