{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "21658c23-2ef7-42e3-bf53-75fba223fcdd", "deepnote_cell_height": 202, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 1891, "execution_start": 1665161862284, "source_hash": "9500ca25" }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "\n", "from odtlearn.flow_oct import FlowOCT, BendersOCT\n", "from odtlearn.utils.binarize import Binarizer" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "cell_id": "22bfbe5ce3fb479c900abc4c0443c38a", "deepnote_cell_height": 84, "deepnote_cell_type": "markdown", "tags": [] }, "source": [ "# `FlowOCT` Examples" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "cell_id": "1d288c39a9ae485b822dbf5eccbd89e3", "deepnote_cell_height": 108, "deepnote_cell_type": "markdown", "tags": [] }, "source": [ "## Example 0: Binarization\n", "\n", "The following example shows how to binarize a dataset with categorical, integer, and continuous features using the built-in `Binarizer` class. This class follows the scikit-learn fit-transform paradigm, making it easy to integrate into your preprocessing pipeline.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "4ed40d742b8d40c697c0ab3d765eaf45", "deepnote_cell_height": 166, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 2, "execution_start": 1665161870277, "source_hash": "c7737b5a", "tags": [] }, "outputs": [], "source": [ "number_of_child_list = [1, 2, 4, 3, 1, 2, 4, 3, 2, 1]\n", "age_list = [10, 20, 40, 30, 10, 20, 40, 30, 20, 10]\n", "race_list = [\n", " \"Black\",\n", " \"White\",\n", " \"Hispanic\",\n", " \"Black\",\n", " \"White\",\n", " \"Black\",\n", " \"White\",\n", " \"Hispanic\",\n", " \"Black\",\n", " \"White\",\n", "]\n", "sex_list = [\"M\", \"F\", \"M\", \"M\", \"F\", \"M\", \"F\", \"M\", \"M\", \"F\"]\n", "income_list = [50000, 75000, 100000, 60000, 80000, 55000, 90000, 70000, 85000, 65000]\n", "\n", "df = pd.DataFrame(\n", " list(zip(sex_list, race_list, number_of_child_list, age_list, income_list)),\n", " columns=[\"sex\", \"race\", \"num_child\", \"age\", \"income\"],\n", ")\n", "\n", "print(df)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "7abc1f79d5044501a954f66ddda8a2d7", "deepnote_cell_height": 628, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 67, "execution_start": 1664770046804, "source_hash": "10849c45", "tags": [] }, "outputs": [], "source": [ "binarizer = Binarizer(\n", " categorical_cols=[\"sex\", \"race\"],\n", " integer_cols=[\"num_child\", \"age\"],\n", " real_cols=[\"income\"],\n", " n_bins=5 # Number of bins for continuous features\n", ")\n", "\n", "# Fit and transform the data\n", "df_enc = binarizer.fit_transform(df)\n", "\n", "print(df_enc)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `Binarizer` class follows the scikit-learn fit-transform paradigm:\n", "\n", "We initialize the `Binarizer` with our desired parameters, specifying which columns are categorical, integer, and real-valued.\n", "We call the fit_transform method, which first fits the binarizer to our data (learning the necessary encoding schemes) and then transforms the data using those learned encodings.\n", "\n", "The resulting `df_enc` DataFrame contains the binarized version of our original data:\n", "\n", "Categorical columns (sex, race) are one-hot encoded.\n", "Integer columns (num_child, age) are binary encoded, where each column represents \"greater than or equal to\" a certain value.\n", "The continuous column (income) is first discretized into 5 bins, then binary encoded similar to the integer columns." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(df_enc.columns)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This binarized data is now ready to be used with any of the models in ODTlearn, which require binary input features.\n", "If you need to transform new data using the same encoding scheme, you can use the transform method of the fitted binarizer:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "new_data = pd.DataFrame({\n", " \"sex\": [\"F\", \"M\"],\n", " \"race\": [\"Hispanic\", \"White\"],\n", " \"num_child\": [2, 3],\n", " \"age\": [20, 30],\n", " \"income\": [70000, 80000]\n", "})\n", "\n", "new_data_enc = binarizer.transform(new_data)\n", "print(new_data_enc)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that for categorical features, if the new data contains categories not seen during fitting, the transform method will raise an error. In such cases, you might need to refit the binarizer on a dataset that includes all possible categories." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "cell_id": "00002-e9b87ac4-673e-49e4-9b9e-b65b2cea9708", "deepnote_cell_height": 188, "deepnote_cell_type": "markdown" }, "source": [ "## Example 1: Varying `depth` and `_lambda`\n", "In this part, we study a simple example and investigate different parameter combinations to provide intuition on how they affect the structure of the tree.\n", "\n", "First we generate the data for our example. The diagram within the code block shows the training dataset. Our dataset has two binary features (X1 and X2) and two class labels (+1 and -1)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "00003-848cadcc-11d1-4e7f-b476-6b0aa6fc5b44", "deepnote_cell_height": 310, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 3, "execution_start": 1665161893704, "owner_user_id": "fe086183-1334-4d34-b4e4-e2e52f4c0652", "source_hash": "75060998" }, "outputs": [], "source": [ "from odtlearn.datasets import flow_oct_example\n", "\n", "\"\"\"\n", " X2\n", " | |\n", " | |\n", " 1 + + | -\n", " | | \n", " |---------------|-------------\n", " | |\n", " 0 - - - - | + + +\n", " | - - - |\n", " |______0________|_______1_______X1\n", "\"\"\"\n", "\n", "\n", "X, y = flow_oct_example()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "cell_id": "00004-08d3e617-af87-4be1-ae2d-e7894cfe2beb", "deepnote_cell_height": 101, "deepnote_cell_type": "markdown" }, "source": [ "### Tree with `depth = 1`\n", "\n", "In the following, we fit a classification tree of depth 1, i.e., a tree with a single branching node and two leaf nodes." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "00005-3fdf503d-31bf-499d-a332-db805ec8b030", "deepnote_cell_height": 230, "deepnote_cell_type": "code", "deepnote_output_heights": [ null, 20 ], "deepnote_to_be_reexecuted": false, "execution_millis": 773, "execution_start": 1665161919210, "source_hash": "9ef0a950" }, "outputs": [], "source": [ "stcl = FlowOCT(depth=1, solver=\"gurobi\", time_limit=100)\n", "stcl.fit(X, y)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "00006-3528e5a2-503d-41bd-b914-933e4b53d8a4", "deepnote_cell_height": 163, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 928, "execution_start": 1665161928297, "source_hash": "ab843413" }, "outputs": [], "source": [ "predictions = stcl.predict(X)\n", "print(f'Optimality gap is {stcl.optim_gap}')\n", "print(f\"In-sample accuracy is {np.sum(predictions==y)/y.shape[0]}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Users can access statistics from the optimization run such as optimality gap, number of nodes, number of constraints, etc. Directly as properties of the initialized class or through the `_solver` object. For example, one can access the optimality gap and the number of solutions after fitting the optimal decision tree through the `optim_gap` and `num_solutions` properties." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(f'Optimality gap is {stcl.optim_gap}')\n", "print(f'Number of solutions {stcl.num_solutions}')" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "cell_id": "00007-ff875aa5-d595-4038-9c0c-db76394cb64e", "deepnote_cell_height": 110, "deepnote_cell_type": "markdown" }, "source": [ "As we can see above, we find the optimal tree and the in-sample accuracy is 76%.\n", "\n", "ODTlearn provides two different ways of visualizing the structure of the tree. The first method prints the structure of the tree in the console:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "00008-7810b10f-28cc-4cfa-bcce-b3c6fe18be24", "deepnote_cell_height": 207, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 3, "execution_start": 1665161943547, "source_hash": "2f43d041" }, "outputs": [], "source": [ "stcl.print_tree()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "cell_id": "00009-90bfb8df-67ac-4e97-abe9-8076249db2b8", "deepnote_cell_height": 52, "deepnote_cell_type": "markdown" }, "source": [ "The second method plots the structure of the tree using `matplotlib`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "00010-c3d86f40-82ba-45c3-aa11-b8fddb6bead1", "deepnote_cell_height": 534, "deepnote_cell_type": "code", "deepnote_output_heights": [ 406 ], "deepnote_to_be_reexecuted": false, "execution_millis": 270, "execution_start": 1665161948808, "source_hash": "d2159fb3" }, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(5, 5))\n", "stcl.plot_tree(ax=ax)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "stcl_progress_log = FlowOCT(depth=1, solver=\"gurobi\", time_limit=100)\n", "stcl_progress_log.fit(X, y)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "cell_id": "00011-8ac47211-d4df-494b-9423-fccb70be09ca", "deepnote_cell_height": 101, "deepnote_cell_type": "markdown" }, "source": [ "### Tree with `depth = 2`\n", "\n", "Now we increase the depth of the tree to achieve higher accuracy." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "00012-6fd1f710-63dc-45bc-aea9-fb6544a3e13d", "deepnote_cell_height": 184, "deepnote_cell_type": "code", "deepnote_output_heights": [ 20 ], "deepnote_to_be_reexecuted": false, "execution_millis": 7, "execution_start": 1665161961681, "source_hash": "1c3c8b1c" }, "outputs": [], "source": [ "stcl = FlowOCT(depth=2, solver=\"gurobi\")\n", "stcl.fit(X, y)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "00013-c3674981-c2c8-4b4b-b429-57a230ae5e93", "deepnote_cell_height": 163, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 619, "execution_start": 1665161968547, "source_hash": "6b45f8a6" }, "outputs": [], "source": [ "predictions = stcl.predict(X)\n", "print(f\"In-sample accuracy is {np.sum(predictions==y)/y.shape[0]}\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "cell_id": "9ee768479e2a4cbd84a0fa6352d26365", "deepnote_cell_height": 52, "deepnote_cell_type": "markdown", "tags": [] }, "source": [ "As we can see, with depth 2, we can achieve 100% in-sample accuracy." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "a9fba11c28df4a56a1a0a3102d2b0314", "deepnote_cell_height": 534, "deepnote_cell_type": "code", "deepnote_output_heights": [ 406 ], "deepnote_to_be_reexecuted": false, "execution_millis": 366, "execution_start": 1665161976627, "scrolled": true, "source_hash": "702243f5", "tags": [] }, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(10, 5))\n", "stcl.plot_tree(ax=ax, fontsize=20)\n", "plt.show()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "cell_id": "00015-cea69e08-5d82-438f-804e-422f3dfe1528", "deepnote_cell_height": 167, "deepnote_cell_type": "markdown" }, "source": [ "### Tree with `depth=2` and Positive `_lambda`\n", "\n", "As we saw in the above example, with depth 2, we can fully classify the training data. However if we add a regularization term with a high enough value of `_lambda`, we can justify pruning one of the branching nodes to get a sparser tree. In the following, we observe that as we increase `_lambda` from 0 to 0.51, one of the branching nodes gets pruned and as a result, the in-sample accuracy drops to 92%." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "00016-9718d2e8-23aa-4565-b40a-373802ebc9ec", "deepnote_cell_height": 202, "deepnote_cell_type": "code", "deepnote_output_heights": [ 20 ], "deepnote_to_be_reexecuted": false, "execution_millis": 205, "execution_start": 1664770189871, "source_hash": "346b639f" }, "outputs": [], "source": [ "stcl = FlowOCT(solver=\"gurobi\", depth=2, _lambda=0.51)\n", "stcl.fit(X, y)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "00017-b7172e7a-d5a1-471a-9f16-51ef1e101a81", "deepnote_cell_height": 125, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 899, "execution_start": 1664770192379, "source_hash": "f3e73368" }, "outputs": [], "source": [ "predictions = stcl.predict(X)\n", "print(f\"In-sample accuracy is {np.sum(predictions==y)/y.shape[0]}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "allow_embed": false, "cell_id": "914dac0037c14b73a80834a07cc31dc5", "deepnote_cell_height": 534, "deepnote_cell_type": "code", "deepnote_output_heights": [ 406 ], "deepnote_to_be_reexecuted": false, "execution_millis": 419, "execution_start": 1664770193471, "source_hash": "702243f5" }, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(10, 5))\n", "stcl.plot_tree(ax=ax, fontsize=20)\n", "plt.show()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "cell_id": "00020-f0f1660b-8b7f-452c-99b7-0f8f23ac89ce", "deepnote_cell_height": 108, "deepnote_cell_type": "markdown" }, "source": [ "## Example 2: Different Objective Functions\n", "\n", "In the following, we have a toy example with an imbalanced data, with the positive class being the minority class." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "00021-8827aefe-7e77-4cfc-a85e-7273ca7b30e5", "deepnote_cell_height": 454, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 0, "execution_start": 1665162085607, "source_hash": "a00d4bf4" }, "outputs": [], "source": [ "'''\n", " X2\n", " | | \n", " | |\n", " 1 + - - | -\n", " | | \n", " |---------------|--------------\n", " | |\n", " 0 - - - + | - - -\n", " | - - - - |\n", " |______0________|_______1_______X1\n", "'''\n", "X = np.array([[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],[0,0],\n", " [1,0],[1,0],[1,0],\n", " [1,1],\n", " [0,1],[0,1],[0,1]])\n", "y = np.array([0,0,0,0,0,0,0,1,\n", " 0,0,0,\n", " 0,\n", " 1,0,0])" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "cell_id": "00022-f6484385-2f43-43ba-83dc-4bec33baee04", "deepnote_cell_height": 62, "deepnote_cell_type": "markdown" }, "source": [ "### Tree with classification accuracy objective" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "00023-e2f878ab-966d-46dd-920e-f0de749caf1f", "deepnote_cell_height": 822, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 54, "execution_start": 1665162091305, "source_hash": "4a372af8" }, "outputs": [], "source": [ "stcl_acc = FlowOCT(solver=\"gurobi\", depth=2, obj_mode=\"acc\")\n", "stcl_acc.fit(X, y)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "predictions = stcl_acc.predict(X)\n", "print(f\"In-sample accuracy is {np.sum(predictions==y)/y.shape[0]}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "09c53f1895bb455e9794f92c03c45ae6", "deepnote_cell_height": 367, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 4, "execution_start": 1665162109464, "source_hash": "a6f4f17", "tags": [] }, "outputs": [], "source": [ "stcl_acc.print_tree()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "00025-de161a83-c201-4c2f-9a41-b2c0051b1834", "deepnote_cell_height": 534, "deepnote_cell_type": "code", "deepnote_output_heights": [ 406 ], "deepnote_to_be_reexecuted": false, "execution_millis": 611, "execution_start": 1665162118253, "source_hash": "84dfb7f3" }, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(10, 5)) \n", "stcl_acc.plot_tree(ax=ax, fontsize=20)\n", "plt.show()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "cell_id": "00026-9028f384-3bb7-4a5a-bf50-d97ddb984dd3", "deepnote_cell_height": 62, "deepnote_cell_type": "markdown" }, "source": [ "### Tree with Balanced Classification Accuracy Objective" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "00027-d4bdf0ac-d06e-4c2a-a8f1-e85669bce65a", "deepnote_cell_height": 840, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 41, "execution_start": 1665162304727, "source_hash": "ea7a2823" }, "outputs": [], "source": [ "stcl_balance = FlowOCT(\n", " solver=\"gurobi\",\n", " depth=2,\n", " obj_mode=\"balance\",\n", " _lambda=0,\n", " verbose=False,\n", ")\n", "stcl_balance.fit(X, y)\n", "predictions = stcl_balance.predict(X)\n", "print(f\"In-sample accuracy is {np.sum(predictions==y)/y.shape[0]}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "00029-fcfc9499-cd0f-45a2-8fbb-dd0be6e2e6b2", "deepnote_cell_height": 534, "deepnote_cell_type": "code", "deepnote_output_heights": [ 406 ], "deepnote_to_be_reexecuted": false, "execution_millis": 339, "execution_start": 1665162313454, "source_hash": "172a869a" }, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(10, 5)) \n", "stcl_balance.plot_tree(ax=ax, fontsize=20)\n", "plt.show()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "cell_id": "865b4df0febc481c969fa433a07f1bce", "deepnote_cell_height": 96, "deepnote_cell_type": "markdown", "tags": [] }, "source": [ "As we can see, when we maximize accuracy, i.e., when `obj_mode = 'acc'`, the optimal tree is just a single node without branching, predicting the majority class for the whole dataset. But when we change the objective mode to balanced accuracy, we account for the minority class by sacrificing the overal accuracy." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "cell_id": "00030-39a05aae-4523-463e-9663-653630a031c9", "deepnote_cell_height": 130, "deepnote_cell_type": "markdown", "deepnote_to_be_reexecuted": false, "execution_millis": 4, "execution_start": 1652396285618, "source_hash": "55e6949" }, "source": [ "## Example 3: UCI Data Example\n", "\n", "In this section, we fit a tree of depth 3 on a real world dataset called the [`balance` dataset](https://archive.ics.uci.edu/ml/datasets/Balance+Scale) from the UCI Machine Learning repository. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "ae66c5fdc7fe4dee9509176ae7738b03", "deepnote_cell_height": 112, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 9, "execution_start": 1663084101706, "source_hash": "459cf272", "tags": [] }, "outputs": [], "source": [ "import pandas as pd\n", "from sklearn.model_selection import train_test_split\n", "from odtlearn.datasets import balance_scale_data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "e586d484fc394bd6ae9e1c9c7423ad77", "deepnote_cell_height": 269, "deepnote_cell_type": "code", "deepnote_output_heights": [ null, 77 ], "deepnote_to_be_reexecuted": false, "execution_millis": 4, "execution_start": 1663084101759, "source_hash": "f97169ac", "tags": [] }, "outputs": [], "source": [ "# read data\n", "data = balance_scale_data()\n", "print(f\"shape{data.shape}\")\n", "data.columns" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "80dbfabf721c4008b103b519b2323c01", "deepnote_cell_height": 148, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 0, "execution_start": 1663084101760, "source_hash": "5e03a259", "tags": [] }, "outputs": [], "source": [ "y = data.pop(\"target\")\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " data, y, test_size=0.33, random_state=42\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "d4144975f0c94c40853a81dff20c53fa", "deepnote_cell_height": 256, "deepnote_cell_type": "code", "deepnote_output_heights": [ 20 ], "deepnote_to_be_reexecuted": false, "execution_millis": 60948, "execution_start": 1663084101761, "source_hash": "a44a256e", "tags": [] }, "outputs": [], "source": [ "stcl = BendersOCT(solver=\"gurobi\", depth=3, time_limit=200, obj_mode=\"acc\", verbose=True)\n", "stcl.store_search_progress_log = True\n", "stcl.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "3d2dd737745047ee8dc047326a93975b", "deepnote_cell_height": 687, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 907, "execution_start": 1663084161863, "source_hash": "2f43d041", "tags": [] }, "outputs": [], "source": [ "stcl.print_tree()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "2e6ec3df59ad4d34894820b67bffa4a1", "deepnote_cell_height": 548.234375, "deepnote_cell_type": "code", "deepnote_output_heights": [ 420.234375 ], "deepnote_to_be_reexecuted": false, "execution_millis": 926, "execution_start": 1663084161864, "source_hash": "2b611b71", "tags": [] }, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(20, 10))\n", "stcl.plot_tree(ax=ax, fontsize=20, color_dict={\"node\": None, \"leaves\": []})\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cell_id": "bea1c5b861094c5fa731c1b08beb4bd2", "deepnote_cell_height": 125, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 371, "execution_start": 1663084162420, "source_hash": "73804030", "tags": [] }, "outputs": [], "source": [ "test_pred = stcl.predict(X_test)\n", "print('The out-of-sample accuracy is {}'.format(np.sum(test_pred==y_test)/y_test.shape[0]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We also provide a simple function allowing users to plot the search progress log over time. Note that you must set the attribute `store_search_progress_log` to `True` before calling the `fit` method to ensure that the bound information is stored. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "stcl.plot_search_progress()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example 4: User-defined Weights\n", "\n", "In this example, we'll demonstrate how to use user-defined weights with FlowOCT and BendersOCT. We'll use a small binary classification dataset and show how user-defined weights can affect the learned tree.\n", "\n", "First, let's create our small binary classification dataset:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "np.random.seed(42)\n", "X = np.random.randint(0, 2, size=(20, 5))\n", "y = np.random.randint(0, 2, size=20)\n", "\n", "print(\"Dataset shape:\", X.shape)\n", "print(\"Class distribution:\", np.bincount(y))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's create weights that heavily favor class 1:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "weights = np.ones_like(y)\n", "weights[y == 1] = 10\n", "\n", "print(\"Weight distribution:\")\n", "print(\"Class 0:\", weights[y == 0].mean())\n", "print(\"Class 1:\", weights[y == 1].mean())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### FlowOCT with User-defined Weights\n", "Let's fit a FlowOCT model with user-defined weights and compare it to a model without the accuracy objective:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# FlowOCT without custom weights\n", "flow_oct_default = FlowOCT(solver=\"gurobi\", obj_mode=\"acc\", depth=2, time_limit=10)\n", "flow_oct_default.fit(X, y)\n", "\n", "# FlowOCT with custom weights\n", "flow_oct_custom = FlowOCT(solver=\"gurobi\", obj_mode=\"weighted\", depth=2, time_limit=10)\n", "flow_oct_custom.fit(X, y, weights=weights)\n", "\n", "print(\"Default FlowOCT predictions:\", flow_oct_default.predict(X))\n", "print(\"User-defined weights FlowOCT predictions:\", flow_oct_custom.predict(X))\n", "\n", "print(\"Default FlowOCT accuracy:\", (flow_oct_default.predict(X) == y).mean())\n", "print(\"User-defined weights FlowOCT accuracy:\", (flow_oct_custom.predict(X) == y).mean())\n", "print(\"User-defined weights FlowOCT weighted accuracy:\", np.average(flow_oct_custom.predict(X) == y, weights=weights))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's visualize both trees:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\n", "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))\n", "\n", "flow_oct_default.plot_tree(ax=ax1, fontsize=10)\n", "ax1.set_title(\"Default FlowOCT\")\n", "\n", "flow_oct_custom.plot_tree(ax=ax2, fontsize=10)\n", "ax2.set_title(\"User-defined Weights FlowOCT\")\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### BendersOCT with User-defined Weights\n", "Now let's do the same with BendersOCT:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# BendersOCT without custom weights\n", "benders_oct_default = BendersOCT(solver=\"gurobi\", obj_mode=\"acc\", depth=2, time_limit=10)\n", "benders_oct_default.fit(X, y)\n", "\n", "# BendersOCT with custom weights\n", "benders_oct_custom = BendersOCT(solver=\"gurobi\", obj_mode=\"weighted\", depth=2, time_limit=10, verbose=False)\n", "benders_oct_custom.fit(X, y, weights=weights)\n", "\n", "print(\"Default BendersOCT predictions:\", benders_oct_default.predict(X))\n", "print(\"User-defined weights BendersOCT predictions:\", benders_oct_custom.predict(X))\n", "\n", "print(\"Default BendersOCT accuracy:\", (benders_oct_default.predict(X) == y).mean())\n", "print(\"User-defined weights BendersOCT accuracy:\", (benders_oct_custom.predict(X) == y).mean())\n", "print(\"User-defined weights BendersOCT weighted accuracy:\", np.average(benders_oct_custom.predict(X) == y, weights=weights))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))\n", "\n", "benders_oct_default.plot_tree(ax=ax1, fontsize=10)\n", "ax1.set_title(\"Default BendersOCT\")\n", "\n", "benders_oct_custom.plot_tree(ax=ax2, fontsize=10)\n", "ax2.set_title(\"User-defined Weights BendersOCT\")\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this example, we've demonstrated how to use user-defined weights with both FlowOCT and BendersOCT. By setting `obj_mode=\"weighted\"` and providing weights during the `fit` method call, we can influence the importance of different samples in the training process.\n", "The user-defined weights in this example heavily favor class 1, which may result in trees that are more likely to predict class 1, potentially at the cost of overall accuracy. However, this can be useful in scenarios where misclassifying one class is more costly than misclassifying the other, or when dealing with imbalanced datasets.\n", "Note that the actual results may vary due to the random nature of the dataset and the optimization process. You may want to run the code multiple times or with different random seeds to get a better understanding of the effects of user-defined weights." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "cell_id": "2181319eebc9479299ad724c6909ad78", "deepnote_cell_height": 203, "deepnote_cell_type": "markdown", "tags": [] }, "source": [ "## References\n", "* Dua, D. and Graff, C. (2019). [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml). Irvine, CA: University of California, School of Information and Computer Science.\n", "* Aghaei, S., Gómez, A., & Vayanos, P. (2025). Strong optimal classification trees. *Operations Research*, 73(4), 2223-2241." ] } ], "metadata": { "deepnote": {}, "deepnote_execution_queue": [], "deepnote_notebook_id": "a83b9a97-2562-44d2-acd9-6ff55c94ce73", "interpreter": { "hash": "dfaf93ad87348b32221474fd3c800e01f580d105683f49be3d64b58d8896a56c" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.2" } }, "nbformat": 4, "nbformat_minor": 4 }