Overfitting and Regularization: Finding the Balance

How to detect overfitting in your models and techniques to prevent it using regularization.

D
Dery Febriantara Developer
Overfitting and Regularization: Finding the Balance

Overfitting is one of the most common and challenging problems in machine learning. Understanding why it happens and how to prevent it is fundamental to building models that actually work in the real world. In this comprehensive guide, we’ll explore the theory behind overfitting, the bias-variance tradeoff, and a complete arsenal of regularization techniques to combat it.

Table of Contents

  1. Understanding Overfitting
  2. The Bias-Variance Tradeoff
  3. Detecting Overfitting
  4. L1 Regularization (Lasso)
  5. L2 Regularization (Ridge)
  6. Elastic Net
  7. Regularization in Neural Networks
  8. Data-Based Regularization
  9. Model Architecture Regularization
  10. Advanced Techniques

Understanding Overfitting

What is Overfitting?

Overfitting occurs when your model learns the training data too well, including its noise and random fluctuations, rather than the underlying patterns. The model essentially memorizes the training examples instead of learning generalizable rules.

Think of it like a student who memorizes all the answers to practice tests but doesn’t understand the underlying concepts. They’ll ace the practice tests but fail when given new questions.

The fundamental problem:

  • Your model performs exceptionally well on training data
  • But performs poorly on new, unseen data
  • The model has “overfit” to the specific training examples

Visual Intuition

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

# Generate noisy data
np.random.seed(42)
X = np.linspace(0, 1, 20).reshape(-1, 1)
y = np.sin(2 * np.pi * X).ravel() + np.random.normal(0, 0.3, 20)

# Create models of different complexity
fig, axes = plt.subplots(1, 3, figsize=(15, 4))
degrees = [1, 4, 15]
titles = ['Underfitting (Degree 1)', 'Good Fit (Degree 4)', 'Overfitting (Degree 15)']

X_plot = np.linspace(0, 1, 100).reshape(-1, 1)

for ax, degree, title in zip(axes, degrees, titles):
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    model.fit(X, y)
    y_plot = model.predict(X_plot)

    ax.scatter(X, y, color='blue', label='Data')
    ax.plot(X_plot, y_plot, color='red', label='Model')
    ax.plot(X_plot, np.sin(2 * np.pi * X_plot),
            color='green', linestyle='--', label='True function')
    ax.set_title(title)
    ax.legend()
    ax.set_ylim(-2, 2)

plt.tight_layout()
plt.show()

Signs of Overfitting

1. Large Gap Between Training and Test Performance

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification

# Generate synthetic data
X, y = make_classification(n_samples=1000, n_features=20,
                           n_informative=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Overfitted model - no constraints
overfit_model = DecisionTreeClassifier(random_state=42)
overfit_model.fit(X_train, y_train)

print("Overfitted Decision Tree:")
print(f"  Training accuracy: {overfit_model.score(X_train, y_train):.4f}")  # ~1.00
print(f"  Test accuracy: {overfit_model.score(X_test, y_test):.4f}")        # ~0.82
print(f"  Gap: {overfit_model.score(X_train, y_train) - overfit_model.score(X_test, y_test):.4f}")

# Well-regularized model
good_model = DecisionTreeClassifier(max_depth=5, min_samples_leaf=10, random_state=42)
good_model.fit(X_train, y_train)

print("\nRegularized Decision Tree:")
print(f"  Training accuracy: {good_model.score(X_train, y_train):.4f}")    # ~0.89
print(f"  Test accuracy: {good_model.score(X_test, y_test):.4f}")          # ~0.87
print(f"  Gap: {good_model.score(X_train, y_train) - good_model.score(X_test, y_test):.4f}")

2. Model Complexity Indicators

# Check model complexity
print(f"Overfitted tree depth: {overfit_model.get_depth()}")        # Often 20+
print(f"Overfitted tree leaves: {overfit_model.get_n_leaves()}")    # Often 100+

print(f"Good tree depth: {good_model.get_depth()}")                  # 5
print(f"Good tree leaves: {good_model.get_n_leaves()}")              # Much fewer

3. High Variance in Cross-Validation

from sklearn.model_selection import cross_val_score

# Overfitted model shows high variance
overfit_scores = cross_val_score(DecisionTreeClassifier(), X, y, cv=10)
print(f"Overfitted CV scores: {overfit_scores}")
print(f"Mean: {overfit_scores.mean():.4f}, Std: {overfit_scores.std():.4f}")

# Good model shows lower variance
good_scores = cross_val_score(DecisionTreeClassifier(max_depth=5, min_samples_leaf=10),
                               X, y, cv=10)
print(f"Good model CV scores: {good_scores}")
print(f"Mean: {good_scores.mean():.4f}, Std: {good_scores.std():.4f}")

Why Does Overfitting Happen?

1. Model Too Complex

  • Too many parameters relative to training data
  • Model has enough capacity to memorize training examples

2. Not Enough Training Data

  • Small datasets make it easy to memorize
  • Insufficient examples to learn generalizable patterns

3. Training Too Long

  • Neural networks can memorize if trained excessively
  • Validation loss starts increasing while training loss decreases

4. Noisy Data

  • Model learns the noise as if it were signal
  • Outliers can dramatically influence the model

5. Feature Engineering Issues

  • Too many features relative to samples
  • Irrelevant features that correlate with training labels by chance

The Bias-Variance Tradeoff

The bias-variance tradeoff is one of the most fundamental concepts in machine learning. Understanding it is crucial for building models that generalize well.

Decomposing Prediction Error

For any prediction, the expected error can be decomposed as:

$$\text{Expected Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Error}$$

Bias: Error from wrong assumptions in the learning algorithm

  • High bias = model is too simple
  • Model can’t capture the underlying patterns
  • Results in underfitting

Variance: Error from sensitivity to small fluctuations in training data

  • High variance = model is too complex
  • Model captures noise as if it were signal
  • Results in overfitting

Irreducible Error: Noise inherent in the data that no model can capture

Visual Representation

Low VarianceHigh Variance
Low BiasIdeal ModelOverfitting
High BiasUnderfittingWorst Case

Mathematical Formulation

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

def compute_bias_variance(X, y, model_class, degree, n_bootstrap=100):
    """Estimate bias and variance through bootstrap sampling."""
    n_samples = len(X)
    X_test = np.linspace(X.min(), X.max(), 100).reshape(-1, 1)

    predictions = np.zeros((n_bootstrap, len(X_test)))

    for i in range(n_bootstrap):
        # Bootstrap sample
        indices = np.random.choice(n_samples, n_samples, replace=True)
        X_boot, y_boot = X[indices], y[indices]

        # Fit model
        poly = PolynomialFeatures(degree)
        X_poly = poly.fit_transform(X_boot)
        model = LinearRegression()
        model.fit(X_poly, y_boot)

        # Predict
        predictions[i] = model.predict(poly.transform(X_test))

    # True function (for synthetic data)
    y_true = np.sin(2 * np.pi * X_test).ravel()

    # Bias: difference between average prediction and true value
    mean_prediction = predictions.mean(axis=0)
    bias_squared = ((mean_prediction - y_true) ** 2).mean()

    # Variance: spread of predictions
    variance = predictions.var(axis=0).mean()

    return bias_squared, variance

# Generate data
np.random.seed(42)
X = np.linspace(0, 1, 50).reshape(-1, 1)
y = np.sin(2 * np.pi * X).ravel() + np.random.normal(0, 0.3, 50)

# Compute for different polynomial degrees
degrees = [1, 3, 5, 10, 15]
print("Degree | Bias² | Variance | Total")
print("-" * 40)
for degree in degrees:
    bias_sq, var = compute_bias_variance(X, y, LinearRegression, degree)
    print(f"  {degree:2d}   | {bias_sq:.4f} |  {var:.4f}  | {bias_sq + var:.4f}")

Practical Implications

High Bias (Underfitting):

  • Training and test errors are both high
  • Model is too simple to capture patterns
  • Solutions: More features, more complex model, less regularization

High Variance (Overfitting):

  • Low training error, high test error
  • Model is too sensitive to training data
  • Solutions: More data, simpler model, more regularization, feature selection
# Demonstration with different model complexities
from sklearn.linear_model import Ridge
import matplotlib.pyplot as plt

# Generate data
np.random.seed(42)
n_samples = 30
X = np.sort(np.random.uniform(0, 1, n_samples)).reshape(-1, 1)
y = np.sin(4 * X).ravel() + np.random.normal(0, 0.3, n_samples)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Compare models of increasing complexity
degrees = range(1, 16)
train_errors = []
test_errors = []

for degree in degrees:
    poly = PolynomialFeatures(degree)
    X_train_poly = poly.fit_transform(X_train)
    X_test_poly = poly.transform(X_test)

    model = Ridge(alpha=0.001)  # Small regularization
    model.fit(X_train_poly, y_train)

    train_errors.append(mean_squared_error(y_train, model.predict(X_train_poly)))
    test_errors.append(mean_squared_error(y_test, model.predict(X_test_poly)))

plt.figure(figsize=(10, 6))
plt.plot(degrees, train_errors, 'b-o', label='Training Error')
plt.plot(degrees, test_errors, 'r-o', label='Test Error')
plt.axvline(x=degrees[np.argmin(test_errors)], color='g', linestyle='--',
            label=f'Optimal Degree: {degrees[np.argmin(test_errors)]}')
plt.xlabel('Polynomial Degree (Model Complexity)')
plt.ylabel('Mean Squared Error')
plt.title('Bias-Variance Tradeoff: Training vs Test Error')
plt.legend()
plt.yscale('log')
plt.show()

Detecting Overfitting

Before applying regularization, you need to know if your model is overfitting. Here are comprehensive techniques for detection.

Learning Curves

Learning curves show how model performance changes with training set size.

from sklearn.model_selection import learning_curve
import matplotlib.pyplot as plt
import numpy as np

def plot_learning_curve(estimator, X, y, title, cv=5):
    """Plot learning curve to diagnose bias/variance."""
    train_sizes, train_scores, test_scores = learning_curve(
        estimator, X, y, cv=cv, n_jobs=-1,
        train_sizes=np.linspace(0.1, 1.0, 10),
        scoring='neg_mean_squared_error'
    )

    train_scores_mean = -train_scores.mean(axis=1)
    train_scores_std = train_scores.std(axis=1)
    test_scores_mean = -test_scores.mean(axis=1)
    test_scores_std = test_scores.std(axis=1)

    plt.figure(figsize=(10, 6))
    plt.fill_between(train_sizes,
                     train_scores_mean - train_scores_std,
                     train_scores_mean + train_scores_std, alpha=0.1, color="blue")
    plt.fill_between(train_sizes,
                     test_scores_mean - test_scores_std,
                     test_scores_mean + test_scores_std, alpha=0.1, color="orange")
    plt.plot(train_sizes, train_scores_mean, 'o-', color="blue", label="Training score")
    plt.plot(train_sizes, test_scores_mean, 'o-', color="orange", label="Cross-validation score")

    plt.xlabel("Training examples")
    plt.ylabel("Mean Squared Error")
    plt.title(title)
    plt.legend(loc="best")
    plt.grid(True)
    plt.show()

# Example with overfitting model
from sklearn.tree import DecisionTreeRegressor

plot_learning_curve(
    DecisionTreeRegressor(max_depth=None),  # Overfitting
    X, y, "Learning Curve - Overfitting Model"
)

plot_learning_curve(
    DecisionTreeRegressor(max_depth=3),     # Well-regularized
    X, y, "Learning Curve - Regularized Model"
)

Interpreting Learning Curves:

PatternDiagnosisSolution
Large gap, both improveOverfittingMore data, regularization
Small gap, both plateau highUnderfittingMore features, complex model
Small gap, both plateau lowGood fitModel is appropriate
Training perfect, test poorSevere overfittingStrong regularization needed

Validation Curves

Validation curves show how performance changes with a hyperparameter.

from sklearn.model_selection import validation_curve

def plot_validation_curve(estimator, X, y, param_name, param_range, title):
    """Plot validation curve to find optimal hyperparameter."""
    train_scores, test_scores = validation_curve(
        estimator, X, y,
        param_name=param_name,
        param_range=param_range,
        cv=5, scoring='neg_mean_squared_error', n_jobs=-1
    )

    train_mean = -train_scores.mean(axis=1)
    train_std = train_scores.std(axis=1)
    test_mean = -test_scores.mean(axis=1)
    test_std = test_scores.std(axis=1)

    plt.figure(figsize=(10, 6))
    plt.semilogx(param_range, train_mean, 'b-o', label='Training score')
    plt.semilogx(param_range, test_mean, 'r-o', label='Cross-validation score')
    plt.fill_between(param_range, train_mean - train_std, train_mean + train_std,
                     alpha=0.1, color='blue')
    plt.fill_between(param_range, test_mean - test_std, test_mean + test_std,
                     alpha=0.1, color='red')

    plt.xlabel(param_name)
    plt.ylabel('Mean Squared Error')
    plt.title(title)
    plt.legend()
    plt.grid(True)
    plt.show()

# Example: Ridge regression alpha
from sklearn.linear_model import Ridge

plot_validation_curve(
    Ridge(), X, y,
    param_name='alpha',
    param_range=np.logspace(-4, 4, 20),
    title='Validation Curve - Ridge Alpha'
)

Cross-Validation Analysis

from sklearn.model_selection import cross_val_score, KFold
import pandas as pd

def analyze_cv_results(model, X, y, cv=10):
    """Detailed cross-validation analysis."""
    kfold = KFold(n_splits=cv, shuffle=True, random_state=42)

    train_scores = []
    test_scores = []

    for train_idx, test_idx in kfold.split(X):
        X_train, X_test = X[train_idx], X[test_idx]
        y_train, y_test = y[train_idx], y[test_idx]

        model.fit(X_train, y_train)
        train_scores.append(model.score(X_train, y_train))
        test_scores.append(model.score(X_test, y_test))

    results = pd.DataFrame({
        'Fold': range(1, cv + 1),
        'Train Score': train_scores,
        'Test Score': test_scores,
        'Gap': [t - v for t, v in zip(train_scores, test_scores)]
    })

    print(results.to_string(index=False))
    print("\nSummary:")
    print(f"  Train Mean: {np.mean(train_scores):.4f} (±{np.std(train_scores):.4f})")
    print(f"  Test Mean:  {np.mean(test_scores):.4f} (±{np.std(test_scores):.4f})")
    print(f"  Avg Gap:    {np.mean([t - v for t, v in zip(train_scores, test_scores)]):.4f}")

    # Diagnosis
    avg_gap = np.mean([t - v for t, v in zip(train_scores, test_scores)])
    test_std = np.std(test_scores)

    if avg_gap > 0.1:
        print("\n⚠️  Warning: Large train-test gap suggests OVERFITTING")
    if test_std > 0.1:
        print("⚠️  Warning: High variance in test scores suggests OVERFITTING")
    if np.mean(test_scores) < 0.7:
        print("⚠️  Warning: Low test scores might indicate UNDERFITTING")

# Example usage
from sklearn.datasets import make_regression
X, y = make_regression(n_samples=200, n_features=20, noise=10, random_state=42)

print("Overfitted Model Analysis:")
analyze_cv_results(DecisionTreeRegressor(max_depth=None), X, y)

print("\n" + "="*60 + "\n")

print("Regularized Model Analysis:")
analyze_cv_results(DecisionTreeRegressor(max_depth=5), X, y)

L1 Regularization (Lasso)

L1 regularization, also known as Lasso (Least Absolute Shrinkage and Selection Operator), adds the sum of absolute values of coefficients to the loss function.

Mathematical Formulation

The Lasso objective function is:

$$\text{Loss} = \sum_{i=1}^{n}(y_i - \hat{y}i)^2 + \alpha \sum{j=1}^{p}|\beta_j|$$

Where:

  • First term is the ordinary least squares loss
  • Second term is the L1 penalty
  • α (alpha) controls the regularization strength
  • β_j are the model coefficients

Key Properties

1. Feature Selection (Sparsity)

  • L1 tends to produce sparse solutions
  • Some coefficients become exactly zero
  • Acts as automatic feature selection

2. Geometric Interpretation

  • L1 constraint region is a diamond (in 2D)
  • Optimal solution often hits corners
  • Corners correspond to zero coefficients

Implementation

from sklearn.linear_model import Lasso, LassoCV
from sklearn.datasets import make_regression
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
import numpy as np
import matplotlib.pyplot as plt

# Generate data with many features, only some are relevant
np.random.seed(42)
n_samples, n_features = 500, 100
n_informative = 10

X, y, true_coef = make_regression(
    n_samples=n_samples,
    n_features=n_features,
    n_informative=n_informative,
    noise=10,
    coef=True,
    random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Scale features (important for regularization)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Train Lasso with different alpha values
alphas = np.logspace(-4, 2, 50)
coef_history = []
train_scores = []
test_scores = []

for alpha in alphas:
    lasso = Lasso(alpha=alpha, max_iter=10000)
    lasso.fit(X_train_scaled, y_train)
    coef_history.append(lasso.coef_)
    train_scores.append(lasso.score(X_train_scaled, y_train))
    test_scores.append(lasso.score(X_test_scaled, y_test))

coef_history = np.array(coef_history)

# Plot coefficient paths
plt.figure(figsize=(12, 5))

plt.subplot(1, 2, 1)
for i in range(n_features):
    plt.semilogx(alphas, coef_history[:, i], alpha=0.7)
plt.xlabel('Alpha (Regularization Strength)')
plt.ylabel('Coefficient Value')
plt.title('Lasso Coefficient Paths')
plt.axhline(y=0, color='k', linestyle='--', alpha=0.3)

plt.subplot(1, 2, 2)
plt.semilogx(alphas, train_scores, 'b-', label='Train R²')
plt.semilogx(alphas, test_scores, 'r-', label='Test R²')
plt.xlabel('Alpha')
plt.ylabel('R² Score')
plt.title('Lasso Performance vs Alpha')
plt.legend()

plt.tight_layout()
plt.show()

Cross-Validated Lasso

# Use cross-validation to find optimal alpha
lasso_cv = LassoCV(alphas=np.logspace(-4, 2, 100), cv=5, random_state=42)
lasso_cv.fit(X_train_scaled, y_train)

print(f"Best alpha: {lasso_cv.alpha_:.6f}")
print(f"Train R²: {lasso_cv.score(X_train_scaled, y_train):.4f}")
print(f"Test R²: {lasso_cv.score(X_test_scaled, y_test):.4f}")
print(f"Non-zero coefficients: {np.sum(lasso_cv.coef_ != 0)} / {n_features}")

# Compare with true informative features
true_informative = np.where(true_coef != 0)[0]
selected_features = np.where(lasso_cv.coef_ != 0)[0]

print(f"\nTrue informative features: {len(true_informative)}")
print(f"Selected by Lasso: {len(selected_features)}")
print(f"Correctly selected: {len(set(true_informative) & set(selected_features))}")

When to Use L1 Regularization

Best suited for:

  • High-dimensional data (many features)
  • When you suspect only a few features are relevant
  • When you need interpretable, sparse models
  • Feature selection during modeling

Limitations:

  • If features are correlated, Lasso tends to pick one arbitrarily
  • May be unstable with multicollinear features
  • Solution path isn’t smooth (coefficient jumps)

L2 Regularization (Ridge)

L2 regularization, also known as Ridge regression or Tikhonov regularization, adds the sum of squared coefficients to the loss function.

Mathematical Formulation

The Ridge objective function is:

$$\text{Loss} = \sum_{i=1}^{n}(y_i - \hat{y}i)^2 + \alpha \sum{j=1}^{p}\beta_j^2$$

Key Properties

1. Shrinkage Without Elimination

  • All coefficients are shrunk toward zero
  • But never become exactly zero
  • All features are retained in the model

2. Geometric Interpretation

  • L2 constraint region is a circle (in 2D)
  • Optimal solution rarely hits exact axes
  • Produces smooth coefficient paths

3. Closed-Form Solution

$$\hat{\beta}_{ridge} = (X^TX + \alpha I)^{-1}X^Ty$$

Implementation

from sklearn.linear_model import Ridge, RidgeCV

# Train Ridge with different alpha values
alphas = np.logspace(-4, 4, 50)
coef_history = []
train_scores = []
test_scores = []

for alpha in alphas:
    ridge = Ridge(alpha=alpha)
    ridge.fit(X_train_scaled, y_train)
    coef_history.append(ridge.coef_)
    train_scores.append(ridge.score(X_train_scaled, y_train))
    test_scores.append(ridge.score(X_test_scaled, y_test))

coef_history = np.array(coef_history)

# Plot coefficient paths
plt.figure(figsize=(12, 5))

plt.subplot(1, 2, 1)
for i in range(n_features):
    plt.semilogx(alphas, coef_history[:, i], alpha=0.7)
plt.xlabel('Alpha (Regularization Strength)')
plt.ylabel('Coefficient Value')
plt.title('Ridge Coefficient Paths')
plt.axhline(y=0, color='k', linestyle='--', alpha=0.3)

plt.subplot(1, 2, 2)
plt.semilogx(alphas, train_scores, 'b-', label='Train R²')
plt.semilogx(alphas, test_scores, 'r-', label='Test R²')
plt.xlabel('Alpha')
plt.ylabel('R² Score')
plt.title('Ridge Performance vs Alpha')
plt.legend()

plt.tight_layout()
plt.show()

Cross-Validated Ridge

# Use cross-validation to find optimal alpha
ridge_cv = RidgeCV(alphas=np.logspace(-4, 4, 100), cv=5)
ridge_cv.fit(X_train_scaled, y_train)

print(f"Best alpha: {ridge_cv.alpha_:.6f}")
print(f"Train R²: {ridge_cv.score(X_train_scaled, y_train):.4f}")
print(f"Test R²: {ridge_cv.score(X_test_scaled, y_test):.4f}")
print(f"Non-zero coefficients: {np.sum(ridge_cv.coef_ != 0)} / {n_features}")  # All are non-zero

L1 vs L2 Comparison

# Side-by-side comparison
fig, axes = plt.subplots(1, 3, figsize=(15, 4))

# Coefficient comparison
lasso = Lasso(alpha=0.1).fit(X_train_scaled, y_train)
ridge = Ridge(alpha=1.0).fit(X_train_scaled, y_train)

axes[0].bar(range(n_features), lasso.coef_, alpha=0.7, label='Lasso')
axes[0].bar(range(n_features), ridge.coef_, alpha=0.5, label='Ridge')
axes[0].set_xlabel('Feature Index')
axes[0].set_ylabel('Coefficient Value')
axes[0].set_title('Coefficient Comparison')
axes[0].legend()

# Coefficient magnitude distribution
axes[1].hist(np.abs(lasso.coef_), bins=30, alpha=0.7, label='Lasso', density=True)
axes[1].hist(np.abs(ridge.coef_), bins=30, alpha=0.5, label='Ridge', density=True)
axes[1].set_xlabel('|Coefficient|')
axes[1].set_ylabel('Density')
axes[1].set_title('Coefficient Distribution')
axes[1].legend()

# Sparsity comparison at different alphas
alphas = np.logspace(-4, 2, 30)
lasso_sparsity = []
ridge_sparsity = []

for alpha in alphas:
    lasso = Lasso(alpha=alpha, max_iter=10000).fit(X_train_scaled, y_train)
    ridge = Ridge(alpha=alpha).fit(X_train_scaled, y_train)
    lasso_sparsity.append(np.sum(lasso.coef_ == 0))
    ridge_sparsity.append(np.sum(np.abs(ridge.coef_) < 1e-6))

axes[2].semilogx(alphas, lasso_sparsity, 'b-o', label='Lasso')
axes[2].semilogx(alphas, ridge_sparsity, 'r-o', label='Ridge')
axes[2].set_xlabel('Alpha')
axes[2].set_ylabel('Number of Zero Coefficients')
axes[2].set_title('Sparsity vs Alpha')
axes[2].legend()

plt.tight_layout()
plt.show()

When to Use L2 Regularization

Best suited for:

  • When all features might be relevant
  • Multicollinear data (correlated features)
  • When you want stable coefficient estimates
  • As a default regularization choice

Limitations:

  • Doesn’t perform feature selection
  • All features retained (interpretability)
  • May need feature selection as a separate step

Elastic Net

Elastic Net combines L1 and L2 regularization, offering the best of both worlds.

Mathematical Formulation

$$\text{Loss} = \sum_{i=1}^{n}(y_i - \hat{y}i)^2 + \alpha \cdot \rho \sum{j=1}^{p}|\beta_j| + \alpha \cdot \frac{(1-\rho)}{2} \sum_{j=1}^{p}\beta_j^2$$

Where:

  • α (alpha) controls overall regularization strength
  • ρ (l1_ratio) controls the mix between L1 and L2
  • ρ = 1: Pure Lasso
  • ρ = 0: Pure Ridge

Implementation

from sklearn.linear_model import ElasticNet, ElasticNetCV

# Train Elastic Net with different l1_ratios
l1_ratios = [0.1, 0.3, 0.5, 0.7, 0.9]
results = []

for l1_ratio in l1_ratios:
    elastic = ElasticNetCV(l1_ratio=l1_ratio, cv=5, random_state=42)
    elastic.fit(X_train_scaled, y_train)

    results.append({
        'l1_ratio': l1_ratio,
        'best_alpha': elastic.alpha_,
        'train_r2': elastic.score(X_train_scaled, y_train),
        'test_r2': elastic.score(X_test_scaled, y_test),
        'non_zero': np.sum(elastic.coef_ != 0)
    })

results_df = pd.DataFrame(results)
print(results_df.to_string(index=False))

Multi-Task Elastic Net

For problems with multiple related targets:

from sklearn.linear_model import MultiTaskElasticNet

# Create multi-output target
y_multi = np.column_stack([y, y + np.random.normal(0, 10, len(y))])

multi_elastic = MultiTaskElasticNet(alpha=0.1, l1_ratio=0.5)
multi_elastic.fit(X_train_scaled, y_multi[train_idx])

# All outputs share the same feature sparsity pattern
print(f"Coefficient matrix shape: {multi_elastic.coef_.shape}")

When to Use Elastic Net

Best suited for:

  • When you want feature selection AND stability
  • Groups of correlated features (selects groups together)
  • When Lasso is too sparse or Ridge keeps too many features
  • As a robust default choice
# Find optimal l1_ratio automatically
elastic_cv = ElasticNetCV(
    l1_ratio=[0.1, 0.5, 0.7, 0.9, 0.95, 0.99, 1],
    alphas=np.logspace(-4, 2, 50),
    cv=5,
    random_state=42
)
elastic_cv.fit(X_train_scaled, y_train)

print(f"Best l1_ratio: {elastic_cv.l1_ratio_}")
print(f"Best alpha: {elastic_cv.alpha_:.6f}")
print(f"Test R²: {elastic_cv.score(X_test_scaled, y_test):.4f}")

Regularization in Neural Networks

Neural networks have many parameters and are prone to overfitting. Several techniques help prevent this.

Dropout

Dropout randomly sets a fraction of neurons to zero during training, preventing co-adaptation of neurons.

import torch
import torch.nn as nn
import torch.optim as optim

class DropoutNetwork(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, dropout_rate=0.5):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(input_size, hidden_size),
            nn.ReLU(),
            nn.Dropout(dropout_rate),
            nn.Linear(hidden_size, hidden_size),
            nn.ReLU(),
            nn.Dropout(dropout_rate),
            nn.Linear(hidden_size, output_size)
        )

    def forward(self, x):
        return self.network(x)

# Create model
model = DropoutNetwork(input_size=100, hidden_size=256, output_size=10, dropout_rate=0.5)

# Dropout is only active during training
model.train()   # Dropout active
model.eval()    # Dropout disabled

Key Points About Dropout:

  • During training: randomly zero out neurons with probability p
  • During inference: all neurons active, weights scaled by (1-p)
  • Acts as ensemble of many subnetworks
  • Typical rates: 0.2-0.5 (input layers often use smaller rates)

Weight Decay (L2 Regularization in Optimizers)

# Add weight decay (L2 regularization) to optimizer
optimizer = optim.Adam(model.parameters(), lr=0.001, weight_decay=0.01)

# Or use SGD with weight decay
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9, weight_decay=0.0001)

# AdamW - decoupled weight decay (recommended)
optimizer = optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.01)

Why AdamW over Adam with weight_decay?

  • Adam’s weight_decay is coupled with gradient-based updates
  • AdamW decouples weight decay from gradient updates
  • Results in better generalization

Early Stopping

Stop training when validation loss stops improving.

# PyTorch early stopping implementation
class EarlyStopping:
    def __init__(self, patience=7, min_delta=0, restore_best_weights=True):
        self.patience = patience
        self.min_delta = min_delta
        self.restore_best_weights = restore_best_weights
        self.best_loss = None
        self.counter = 0
        self.best_weights = None

    def __call__(self, val_loss, model):
        if self.best_loss is None:
            self.best_loss = val_loss
            self.best_weights = model.state_dict().copy()
        elif val_loss > self.best_loss - self.min_delta:
            self.counter += 1
            if self.counter >= self.patience:
                if self.restore_best_weights:
                    model.load_state_dict(self.best_weights)
                return True
        else:
            self.best_loss = val_loss
            self.best_weights = model.state_dict().copy()
            self.counter = 0
        return False

# Usage in training loop
early_stopping = EarlyStopping(patience=5)

for epoch in range(1000):
    train_loss = train_epoch(model, train_loader)
    val_loss = validate(model, val_loader)

    if early_stopping(val_loss, model):
        print(f"Early stopping at epoch {epoch}")
        break

Keras/TensorFlow Implementation

from tensorflow import keras
from tensorflow.keras import layers, regularizers
from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau

# Model with L2 regularization
model = keras.Sequential([
    layers.Dense(256, activation='relu',
                 kernel_regularizer=regularizers.l2(0.01)),
    layers.Dropout(0.5),
    layers.Dense(128, activation='relu',
                 kernel_regularizer=regularizers.l2(0.01)),
    layers.Dropout(0.3),
    layers.Dense(10, activation='softmax')
])

model.compile(
    optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['accuracy']
)

# Callbacks for regularization
callbacks = [
    EarlyStopping(
        monitor='val_loss',
        patience=10,
        restore_best_weights=True
    ),
    ReduceLROnPlateau(
        monitor='val_loss',
        factor=0.5,
        patience=5,
        min_lr=1e-7
    )
]

history = model.fit(
    X_train, y_train,
    validation_split=0.2,
    epochs=100,
    callbacks=callbacks,
    batch_size=32
)

Batch Normalization

While primarily for faster training, BatchNorm also provides regularization.

class BatchNormNetwork(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(input_size, hidden_size),
            nn.BatchNorm1d(hidden_size),
            nn.ReLU(),
            nn.Linear(hidden_size, hidden_size),
            nn.BatchNorm1d(hidden_size),
            nn.ReLU(),
            nn.Linear(hidden_size, output_size)
        )

    def forward(self, x):
        return self.network(x)

Layer Normalization

Alternative to BatchNorm, especially for sequences and transformers:

class LayerNormNetwork(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.ln1 = nn.LayerNorm(hidden_size)
        self.fc2 = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        x = self.fc1(x)
        x = self.ln1(x)
        x = torch.relu(x)
        x = self.fc2(x)
        return x

Label Smoothing

Prevents overconfident predictions by softening target labels.

# Instead of hard labels [0, 1, 0, 0]
# Use soft labels [0.025, 0.925, 0.025, 0.025]

class LabelSmoothingLoss(nn.Module):
    def __init__(self, num_classes, smoothing=0.1):
        super().__init__()
        self.smoothing = smoothing
        self.num_classes = num_classes

    def forward(self, pred, target):
        confidence = 1.0 - self.smoothing
        smooth_label = self.smoothing / (self.num_classes - 1)

        true_dist = torch.full_like(pred, smooth_label)
        true_dist.scatter_(1, target.unsqueeze(1), confidence)

        return torch.mean(torch.sum(-true_dist * torch.log_softmax(pred, dim=1), dim=1))

# Or use PyTorch's built-in
criterion = nn.CrossEntropyLoss(label_smoothing=0.1)

Data-Based Regularization

Sometimes the best regularization comes from the data itself.

Getting More Data

More training data is often the most effective regularization technique.

# Learning curve showing effect of data size
from sklearn.model_selection import learning_curve

train_sizes, train_scores, test_scores = learning_curve(
    model, X, y,
    train_sizes=np.linspace(0.1, 1.0, 10),
    cv=5
)

# If test score is still improving at 100% data, you'd benefit from more data

Data Augmentation

Create synthetic training examples to expand your dataset.

Image Augmentation:

from torchvision import transforms
import albumentations as A
from albumentations.pytorch import ToTensorV2

# PyTorch transforms
train_transform = transforms.Compose([
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.RandomRotation(15),
    transforms.RandomResizedCrop(224, scale=(0.8, 1.0)),
    transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.1),
    transforms.RandomGrayscale(p=0.1),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    transforms.RandomErasing(p=0.2)
])

# Albumentations (faster, more options)
train_transform = A.Compose([
    A.RandomRotate90(),
    A.Flip(),
    A.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.1, rotate_limit=45, p=0.5),
    A.OneOf([
        A.GaussNoise(var_limit=(10, 50)),
        A.GaussianBlur(blur_limit=7),
        A.MotionBlur(blur_limit=7),
    ], p=0.3),
    A.OneOf([
        A.OpticalDistortion(),
        A.GridDistortion(),
        A.ElasticTransform(),
    ], p=0.3),
    A.CoarseDropout(max_holes=8, max_height=16, max_width=16, p=0.3),
    A.Normalize(),
    ToTensorV2()
])

Text Augmentation:

import random

def augment_text(text, aug_prob=0.1):
    """Simple text augmentation techniques."""
    words = text.split()
    augmented = []

    for word in words:
        r = random.random()

        if r < aug_prob:
            # Random deletion
            continue
        elif r < 2 * aug_prob:
            # Random swap with next word
            augmented.append(word)
            if augmented:
                i = len(augmented) - 1
                if i > 0:
                    augmented[i], augmented[i-1] = augmented[i-1], augmented[i]
        else:
            augmented.append(word)

    return ' '.join(augmented)

# Using nlpaug library for advanced augmentation
# pip install nlpaug
import nlpaug.augmenter.word as naw

# Synonym replacement
aug = naw.SynonymAug(aug_src='wordnet')
augmented_text = aug.augment(text)

# Back-translation
aug = naw.BackTranslationAug(
    from_model_name='facebook/wmt19-en-de',
    to_model_name='facebook/wmt19-de-en'
)

Tabular Data Augmentation:

from sklearn.neighbors import NearestNeighbors
from imblearn.over_sampling import SMOTE

# SMOTE for class imbalance
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_train, y_train)

# Noise injection
def add_noise(X, noise_factor=0.1):
    noise = np.random.normal(0, noise_factor, X.shape)
    return X + noise

X_augmented = add_noise(X_train, noise_factor=0.1)

# Mixup augmentation
def mixup(X, y, alpha=0.2):
    """Mixup: creates virtual training examples."""
    batch_size = len(X)
    indices = np.random.permutation(batch_size)

    lambda_ = np.random.beta(alpha, alpha, batch_size)
    lambda_ = np.maximum(lambda_, 1 - lambda_)

    X_mixed = lambda_.reshape(-1, 1) * X + (1 - lambda_).reshape(-1, 1) * X[indices]
    y_mixed = lambda_ * y + (1 - lambda_) * y[indices]

    return X_mixed, y_mixed

Cross-Validation

Using all your data efficiently.

from sklearn.model_selection import (
    cross_val_score,
    StratifiedKFold,
    RepeatedStratifiedKFold,
    LeaveOneOut
)

# Standard K-Fold
scores = cross_val_score(model, X, y, cv=5)

# Stratified K-Fold (maintains class proportions)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=cv)

# Repeated cross-validation (more stable estimates)
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=42)
scores = cross_val_score(model, X, y, cv=cv)

# Leave-One-Out (for small datasets)
cv = LeaveOneOut()
scores = cross_val_score(model, X, y, cv=cv)

Model Architecture Regularization

Constraining model complexity through architecture choices.

Limiting Tree Depth

from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier

# Decision Tree constraints
tree = DecisionTreeClassifier(
    max_depth=5,              # Limit depth
    min_samples_split=10,     # Minimum samples to split
    min_samples_leaf=5,       # Minimum samples in leaf
    max_features='sqrt',      # Consider subset of features
    ccp_alpha=0.01            # Cost complexity pruning
)

# Random Forest constraints
rf = RandomForestClassifier(
    n_estimators=100,
    max_depth=10,
    min_samples_split=5,
    min_samples_leaf=2,
    max_features='sqrt',
    max_samples=0.8,          # Bootstrap sample size
    n_jobs=-1
)

# Gradient Boosting constraints
gb = GradientBoostingClassifier(
    n_estimators=100,
    max_depth=3,              # Typically small for boosting
    learning_rate=0.1,        # Smaller = more regularization
    subsample=0.8,            # Row subsampling
    max_features='sqrt',
    min_samples_leaf=10
)

XGBoost Regularization

import xgboost as xgb

model = xgb.XGBClassifier(
    n_estimators=100,
    max_depth=6,
    learning_rate=0.1,

    # L1 regularization (like Lasso)
    reg_alpha=0.1,

    # L2 regularization (like Ridge)
    reg_lambda=1.0,

    # Minimum loss reduction for split
    gamma=0.1,

    # Subsampling
    subsample=0.8,
    colsample_bytree=0.8,
    colsample_bylevel=0.8,

    # Minimum child weight
    min_child_weight=5,

    # Early stopping
    early_stopping_rounds=10
)

# Train with evaluation set
model.fit(
    X_train, y_train,
    eval_set=[(X_test, y_test)],
    verbose=False
)

Neural Network Architecture

class RegularizedNetwork(nn.Module):
    """Network with multiple regularization techniques."""

    def __init__(self, input_size, hidden_sizes, output_size, dropout_rate=0.5):
        super().__init__()

        layers = []
        in_size = input_size

        for hidden_size in hidden_sizes:
            layers.extend([
                nn.Linear(in_size, hidden_size),
                nn.BatchNorm1d(hidden_size),
                nn.ReLU(),
                nn.Dropout(dropout_rate)
            ])
            in_size = hidden_size

        layers.append(nn.Linear(in_size, output_size))

        self.network = nn.Sequential(*layers)

        # Weight initialization for regularization
        self._initialize_weights()

    def _initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Linear):
                # Xavier/Glorot initialization
                nn.init.xavier_normal_(m.weight)
                if m.bias is not None:
                    nn.init.zeros_(m.bias)

    def forward(self, x):
        return self.network(x)

# Create a smaller, regularized network instead of a huge one
model = RegularizedNetwork(
    input_size=100,
    hidden_sizes=[64, 32],  # Smaller hidden layers
    output_size=10,
    dropout_rate=0.3
)

Advanced Techniques

Bayesian Regularization

Use priors on weights for automatic regularization.

# Using sklearn's BayesianRidge
from sklearn.linear_model import BayesianRidge

model = BayesianRidge(
    alpha_1=1e-6,   # Shape parameter for Gamma prior over alpha
    alpha_2=1e-6,   # Inverse scale parameter for Gamma prior over alpha
    lambda_1=1e-6,  # Shape parameter for Gamma prior over lambda
    lambda_2=1e-6   # Inverse scale parameter for Gamma prior over lambda
)
model.fit(X_train, y_train)

# Get uncertainty estimates
y_pred, y_std = model.predict(X_test, return_std=True)
print(f"Prediction uncertainty: {y_std.mean():.4f}")

Spectral Normalization

Constrains the spectral norm of weight matrices.

from torch.nn.utils import spectral_norm

class SpectralNormNetwork(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.fc1 = spectral_norm(nn.Linear(input_size, hidden_size))
        self.fc2 = spectral_norm(nn.Linear(hidden_size, hidden_size))
        self.fc3 = spectral_norm(nn.Linear(hidden_size, output_size))

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        return self.fc3(x)

Gradient Clipping

Prevents exploding gradients during training.

# PyTorch gradient clipping
optimizer.zero_grad()
loss.backward()

# Clip by value
torch.nn.utils.clip_grad_value_(model.parameters(), clip_value=1.0)

# Or clip by norm (more common)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

optimizer.step()

Virtual Adversarial Training

Regularizes model smoothness with adversarial examples.

def virtual_adversarial_loss(model, x, epsilon=1.0, xi=1e-6, num_iters=1):
    """Compute virtual adversarial perturbation."""
    with torch.no_grad():
        pred = torch.softmax(model(x), dim=1)

    d = torch.randn_like(x)
    d = d / torch.norm(d, dim=1, keepdim=True)

    for _ in range(num_iters):
        d.requires_grad = True
        pred_hat = torch.softmax(model(x + xi * d), dim=1)
        kl_div = torch.sum(pred * (torch.log(pred + 1e-10) - torch.log(pred_hat + 1e-10)))
        kl_div.backward()
        d = d.grad.detach()
        d = d / torch.norm(d, dim=1, keepdim=True)

    r_adv = epsilon * d
    pred_adv = torch.softmax(model(x + r_adv), dim=1)

    return torch.sum(pred * (torch.log(pred + 1e-10) - torch.log(pred_adv + 1e-10)))

Stochastic Weight Averaging (SWA)

Averages weights along training trajectory for better generalization.

from torch.optim.swa_utils import AveragedModel, SWALR

# Create SWA model
swa_model = AveragedModel(model)
swa_scheduler = SWALR(optimizer, swa_lr=0.05)

# Training loop
for epoch in range(100):
    train_epoch(model, train_loader, optimizer)

    if epoch > 75:  # Start SWA after 75% of training
        swa_model.update_parameters(model)
        swa_scheduler.step()
    else:
        scheduler.step()

# Update batch normalization statistics for SWA model
torch.optim.swa_utils.update_bn(train_loader, swa_model)

Practical Guidelines

Choosing Regularization Strategy

Start Here
    │
    ├── Linear Model?
    │   ├── Many irrelevant features? → Lasso (L1)
    │   ├── Correlated features? → Ridge (L2) or Elastic Net
    │   └── Uncertain? → Elastic Net
    │
    ├── Tree-Based Model?
    │   ├── Limit max_depth
    │   ├── Increase min_samples_leaf
    │   └── Use ensemble methods
    │
    └── Neural Network?
        ├── Always use weight decay
        ├── Add Dropout (0.2-0.5)
        ├── Use BatchNorm
        └── Implement Early Stopping
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from scipy.stats import uniform, loguniform

# Grid search for regularization parameters
param_grid = {
    'alpha': [0.001, 0.01, 0.1, 1, 10, 100],
    'l1_ratio': [0.1, 0.3, 0.5, 0.7, 0.9]
}

grid_search = GridSearchCV(
    ElasticNet(max_iter=10000),
    param_grid,
    cv=5,
    scoring='neg_mean_squared_error',
    n_jobs=-1
)
grid_search.fit(X_train_scaled, y_train)
print(f"Best params: {grid_search.best_params_}")

# Random search (more efficient for large search spaces)
param_distributions = {
    'alpha': loguniform(1e-4, 100),
    'l1_ratio': uniform(0, 1)
}

random_search = RandomizedSearchCV(
    ElasticNet(max_iter=10000),
    param_distributions,
    n_iter=100,
    cv=5,
    scoring='neg_mean_squared_error',
    n_jobs=-1,
    random_state=42
)
random_search.fit(X_train_scaled, y_train)
print(f"Best params: {random_search.best_params_}")

Complete Example: Building a Regularized Pipeline

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
import numpy as np

# Create a comprehensive regularized pipeline
def create_regularized_model(model_type='elastic_net'):
    """Create a regularized model pipeline."""

    if model_type == 'elastic_net':
        model = ElasticNetCV(l1_ratio=[0.1, 0.5, 0.7, 0.9, 0.95, 0.99],
                             cv=5, random_state=42)
    elif model_type == 'ridge':
        model = RidgeCV(cv=5)
    elif model_type == 'lasso':
        model = LassoCV(cv=5, random_state=42)
    elif model_type == 'tree':
        model = DecisionTreeRegressor(
            max_depth=5,
            min_samples_split=10,
            min_samples_leaf=5
        )
    elif model_type == 'rf':
        model = RandomForestRegressor(
            n_estimators=100,
            max_depth=10,
            min_samples_split=5,
            n_jobs=-1,
            random_state=42
        )

    pipeline = Pipeline([
        ('scaler', StandardScaler()),
        ('model', model)
    ])

    return pipeline

# Evaluate multiple regularized models
models = ['elastic_net', 'ridge', 'lasso', 'tree', 'rf']
results = []

for model_type in models:
    pipeline = create_regularized_model(model_type)
    scores = cross_val_score(pipeline, X, y, cv=5, scoring='r2')
    results.append({
        'model': model_type,
        'mean_r2': scores.mean(),
        'std_r2': scores.std()
    })

results_df = pd.DataFrame(results).sort_values('mean_r2', ascending=False)
print(results_df.to_string(index=False))

Summary

Key Takeaways

  1. Overfitting is about generalization - Your model needs to work on new data, not memorize training data.

  2. Bias-variance tradeoff is fundamental - Find the sweet spot between underfitting and overfitting.

  3. Multiple detection methods - Use learning curves, validation curves, and cross-validation to diagnose issues.

  4. Choose regularization wisely:

    • L1 (Lasso): Feature selection, sparse models
    • L2 (Ridge): Stable estimates, correlated features
    • Elastic Net: Best of both worlds
    • Dropout/Early Stopping: Neural networks

  5. Data augmentation is powerful - Sometimes the best regularization is more (varied) data.

  6. Start simple, add complexity - Begin with simpler models and add regularization gradually.

Quick Reference

TechniqueBest ForKey Parameter
L1 (Lasso)Feature selectionalpha
L2 (Ridge)Correlated featuresalpha
Elastic NetGeneral usealpha, l1_ratio
DropoutNeural networksdropout_rate
Early StoppingAny iterative modelpatience
Weight DecayNeural networksweight_decay
Max DepthTreesmax_depth
Data AugmentationLimited dataaug_probability

Further Reading

  • “The Elements of Statistical Learning” - Hastie, Tibshirani, Friedman
  • “Deep Learning” - Goodfellow, Bengio, Courville (Chapter 7: Regularization)
  • “Pattern Recognition and Machine Learning” - Bishop
  • Scikit-learn Regularization Guide - Official documentation
  • “A Disciplined Approach to Neural Network Hyper-Parameters” - Leslie Smith