Skip to main content
xplainabledocs
Version: v1.4.1

XEvolutionaryNetwork

Weight Optimization for XRegressor Models

XEvolutionaryNetwork is a layer-based optimization framework specifically designed for optimizing the leaf weights of XRegressor models. It chains together optimization layers (Tighten and Evolve) to iteratively improve model predictions.

Overview​

XEvolutionaryNetwork takes a pre-trained XRegressor model and optimizes its leaf node weights through a series of configurable layers. It is inspired by deep learning frameworks but is applied over additive models for weight optimization.

The network can be stopped mid-training and resumed at any time. You can track remaining and completed layers using the future_layers and completed_layers attributes.

Key Benefits​

  • Layer-based architecture: Chain multiple optimization strategies together
  • Flexible depth: No limit on the number of layers
  • Pausable training: Stop and resume at any point
  • Feature-level optimization: Optimize weights for a subset of features using the subset parameter

Architecture Overview​

XEvolutionaryNetwork uses two types of optimization layers:

  1. Tighten: A leaf boosting algorithm that iteratively adjusts individual leaf weights to minimize error
  2. Evolve: A genetic algorithm that mutates and evolves leaf weight chromosomes across generations

These tools are separate from XEvolutionaryNetwork and serve different purposes:

  • XParamOptimiser: Bayesian optimization for hyperparameter tuning of XClassifier models (not a layer in XEvolutionaryNetwork)
  • Target: Finds model leaf nodes that achieve a target prediction value
  • NLPOptimiser: Bayesian optimization for NLP preprocessing parameters

XEvolutionaryNetwork API​

1from xplainable.core.optimisation.genetic import XEvolutionaryNetwork
2from xplainable.core.optimisation.layers import Tighten, Evolve
3from xplainable.core.models import XRegressor
4

Constructor​

1XEvolutionaryNetwork(model: XRegressor, apply_range: bool = False)
2

Parameters:

  • model (XRegressor): The pre-trained XRegressor model to optimize.
  • apply_range (bool): Whether to clip predictions to the model's prediction_range during optimization. Defaults to False.

Methods​

add_layer(layer, idx=None)method
Add a Tighten or Evolve layer. Optional idx to insert at a specific position.
drop_layer(idx)method
Remove a layer by index.
clear_layers()method
Remove all pending layers.
fit(x, y, subset=[])method
Fit the model and training data to the network. subset is an optional list of column names for feature-level optimization.
optimise(callback=None)method
Sequentially run all pending layers.

Attributes​

future_layerslist
List of layers yet to be executed.
completed_layerslist
List of layers that have been executed.
checkpoint_scorefloat
The best score achieved after the last completed layer.

Optimization Layers​

Tighten​

A leaf boosting algorithm that iteratively identifies the single leaf node change that will have the greatest impact on the overall model score, then incrementally adjusts that leaf weight.

1from xplainable.core.optimisation.layers import Tighten
2
3tighten_layer = Tighten(
4 iterations=100, # Number of iterations to run
5 learning_rate=0.03, # How fast the model learns (0.001 to 1)
6 early_stopping=None # Stop if no improvement after n iterations
7)
8

Parameters:

  • iterations (int): Number of optimization iterations. Defaults to 100.
  • learning_rate (float): Controls the step size of each weight adjustment. Between 0.001 and 1. Defaults to 0.03.
  • early_stopping (int, optional): Stop early if no improvement after n iterations. Defaults to None.

Evolve​

A genetic algorithm that generates mutations of the leaf weight chromosome, selects superior mutations, reproduces, and evolves toward better solutions.

1from xplainable.core.optimisation.layers import Evolve
2
3evolve_layer = Evolve(
4 mutations=100, # Number of mutations per generation
5 generations=50, # Number of generations to run
6 max_generation_depth=10, # Maximum depth of reproduction within a generation
7 max_severity=0.5, # Maximum severity of a mutation
8 max_leaves=20, # Maximum number of leaves to mutate at once
9 early_stopping=None # Stop if no improvement after n generations
10)
11

Parameters:

  • mutations (int): Number of mutations to generate per generation. Defaults to 100.
  • generations (int): Number of generations to run. Defaults to 50.
  • max_generation_depth (int): Maximum depth of reproduction cycles within a generation. Defaults to 10.
  • max_severity (float): Maximum severity of a mutation (proportion of original value). Defaults to 0.5.
  • max_leaves (int): Maximum number of leaves to mutate in a single mutation. Defaults to 20.
  • early_stopping (int, optional): Stop early if no improvement after n generations. Defaults to None.

Both layers optimize using either MAE or MSE (controlled by the metric attribute inherited from BaseLayer, defaulting to 'mae').

Basic Usage​

Complete Workflow​

1from xplainable.core.models import XRegressor
2from xplainable.core.optimisation.genetic import XEvolutionaryNetwork
3from xplainable.core.optimisation.layers import Tighten, Evolve
4import pandas as pd
5from sklearn.model_selection import train_test_split
6
7# Load data
8data = pd.read_csv('regression_data.csv')
9X = data.drop('target', axis=1)
10y = data['target']
11X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
12
13# Step 1: Train the base XRegressor model
14model = XRegressor(max_depth=8, min_info_gain=0.01)
15model.fit(X_train, y_train)
16
17# Optional: Optimize tail sensitivity first
18model.optimise_tail_sensitivity(X_train, y_train)
19
20# Evaluate baseline performance
21baseline_metrics = model.evaluate(X_test, y_test)
22print(f"Baseline MAE: {baseline_metrics['MAE']}")
23
24# Step 2: Create the evolutionary network
25network = XEvolutionaryNetwork(model)
26
27# Step 3: Add optimization layers
28network.add_layer(Tighten(iterations=200, learning_rate=0.05, early_stopping=20))
29network.add_layer(Evolve(mutations=100, generations=50, early_stopping=10))
30network.add_layer(Tighten(iterations=100, learning_rate=0.01, early_stopping=15))
31
32# Step 4: Fit the network with training data
33network.fit(X_train, y_train)
34
35# Step 5: Run the optimization
36network.optimise()
37
38# Evaluate optimized performance (model is modified in-place)
39optimized_metrics = model.evaluate(X_test, y_test)
40print(f"Optimized MAE: {optimized_metrics['MAE']}")
41print(f"Optimized R2: {optimized_metrics['R2 Score']}")
42

Feature-Level Optimization​

You can optimize weights for specific features only using the subset parameter:

1# Optimize only specific features
2network = XEvolutionaryNetwork(model)
3network.add_layer(Tighten(iterations=100, learning_rate=0.03))
4network.fit(X_train, y_train, subset=['feature_a', 'feature_b', 'feature_c'])
5network.optimise()
6

Progressive Optimization​

Build up optimization in stages, checking results between layers:

1network = XEvolutionaryNetwork(model)
2
3# Stage 1: Broad tightening
4network.add_layer(Tighten(iterations=200, learning_rate=0.05, early_stopping=20))
5network.fit(X_train, y_train)
6network.optimise()
7
8print(f"After Tighten: {network.checkpoint_score}")
9
10# Stage 2: Add evolutionary refinement and continue
11network.add_layer(Evolve(mutations=150, generations=80, max_severity=0.3, early_stopping=15))
12network.optimise()
13
14print(f"After Evolve: {network.checkpoint_score}")
15
16# Stage 3: Fine-tune with a slow learning rate
17network.add_layer(Tighten(iterations=100, learning_rate=0.01, early_stopping=10))
18network.optimise()
19
20print(f"After fine-tuning: {network.checkpoint_score}")
21

Using with Prediction Range​

When your regression model has bounded predictions, use apply_range=True to clip predictions during optimization:

1model = XRegressor(
2 max_depth=8,
3 prediction_range=(0, 1000) # Predictions bounded between 0 and 1000
4)
5model.fit(X_train, y_train)
6
7# Apply range during optimization
8network = XEvolutionaryNetwork(model, apply_range=True)
9network.add_layer(Tighten(iterations=200, learning_rate=0.05))
10network.add_layer(Evolve(mutations=100, generations=50))
11network.fit(X_train, y_train)
12network.optimise()
13

XParamOptimiser​

XParamOptimiser is a separate Bayesian optimization tool for finding optimal hyperparameters for XClassifier models. It is not a layer in XEvolutionaryNetwork.

1from xplainable.core.optimisation.bayesian import XParamOptimiser
2

Constructor​

1XParamOptimiser(
2 metric='roc-auc',
3 n_trials=30,
4 n_folds=5,
5 early_stopping=30,
6 shuffle=False,
7 subsample=1,
8 alpha=0.01,
9 max_depth_space=[4, 10, 2],
10 min_leaf_size_space=[0.005, 0.05, 0.005],
11 min_info_gain_space=[0.005, 0.05, 0.005],
12 ignore_nan_space=[False, True],
13 weight_space=[0, 1.2, 0.05],
14 power_degree_space=[1, 3, 2],
15 sigmoid_exponent_space=[0.5, 1, 0.1],
16 verbose=True,
17 random_state=1
18)
19

Key Parameters:

  • metric (str): Optimization metric. Supported metrics include 'roc-auc', 'accuracy', 'macro-f1', 'weighted-f1', 'positive-f1', 'negative-f1', 'brier-loss', 'log-loss', and various precision/recall variants. Defaults to 'roc-auc'.
  • n_trials (int): Number of Bayesian optimization trials. Defaults to 30.
  • n_folds (int): Number of cross-validation folds. Defaults to 5.
  • early_stopping (int): Stop if no improvement after n trials. Defaults to 30.
  • alpha (float): Sets the alpha of the model. Defaults to 0.01.
  • *_space parameters: Each is a list of [start, stop, step] values defining the search space. Pass a single value instead of a list to fix that parameter.

Usage​

1from xplainable.core.optimisation.bayesian import XParamOptimiser
2
3opt = XParamOptimiser(
4 metric='roc-auc',
5 n_trials=50,
6 n_folds=5,
7 early_stopping=15
8)
9
10# Returns the best parameter dictionary
11best_params = opt.optimise(X_train, y_train)
12print(f"Best parameters: {best_params}")
13
14# Use the optimized parameters to create a model
15from xplainable.core.models import XClassifier
16
17model = XClassifier(**best_params)
18model.fit(X_train, y_train)
19
20# Evaluate
21evaluation = model.evaluate(X_test, y_test)
22print(f"ROC AUC: {evaluation['roc_auc']}")
23

Target​

The Target class finds model leaf nodes that achieve a specific target prediction value. It is not a layer in XEvolutionaryNetwork.

1from xplainable.core.optimisation.targeting import Target
2
3target = Target(model, tolerance=0.005)
4
5# Find leaf nodes to achieve a target score of 0.62
6# Optionally lock specific feature values
7nodes = target.run(
8 target=0.62,
9 iterations=1000,
10 locked={'Age': 32, 'Balance': 20000, 'IsActiveMember': 1}
11)
12

Constructor:

  • model: A fitted xplainable model.
  • tolerance (float): Acceptable difference from the target value. Defaults to 0.005.

run() method:

  • target (float): The target prediction value to achieve.
  • iterations (int): Number of search iterations. Defaults to 1000.
  • locked (dict): Dictionary of feature names to values that should be fixed during search.

Returns a dictionary mapping feature names to their optimal leaf nodes.

NLPOptimiser​

NLPOptimiser performs Bayesian optimization for NLP preprocessing parameters. It is not a layer in XEvolutionaryNetwork.

1from xplainable.core.optimisation.nlp import NLPOptimiser
2
3nlp_opt = NLPOptimiser(
4 nlp=nlp_instance, # Your NLP preprocessing object
5 drop_cols=[], # Columns to drop after NLP transformation
6 metric='weighted-f1', # Optimization metric
7 early_stopping=100,
8 n_trials=30,
9 n_folds=5,
10 shuffle=False,
11 subsample=1,
12 random_state=1,
13 min_word_freq_space=[0.0002, 0.005, 0.0002],
14 max_word_freq_space=[0.005, 0.5, 0.005],
15 min_ngram_freq_space=[0.0002, 0.005, 0.0002],
16 max_ngram_freq_space=[0.005, 0.5, 0.005]
17)
18
19# Optimize NLP parameters
20best_nlp_params = nlp_opt.optimise(
21 x=X_train,
22 y=y_train,
23 x_val=X_val,
24 y_val=y_val
25)
26

Best Practices​

Designing Effective Optimization
  1. Always train and evaluate the base model first before applying XEvolutionaryNetwork.
  2. Use optimise_tail_sensitivity() on XRegressor before creating the evolutionary network.
  3. Start with Tighten layers for initial improvement, then add Evolve layers for further refinement.
  4. Use early stopping to prevent overfitting and save computation time.
  5. Validate on held-out data -- the model is modified in place, so evaluate on test data after optimization.
  6. Be cautious with update_feature_params() after optimization -- it overwrites the optimized weights.

Next Steps​

Ready for More?