{ "cells": [ { "cell_type": "code", "execution_count": null, "id": "ce4ea63c-d979-4e70-91cb-c4b3da8a2756", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.metrics import accuracy_score\n", "from math import log\n", "from copy import deepcopy\n", "\n", "from odtlearn.datasets import robust_example\n", "from odtlearn.robust_oct import RobustOCT" ] }, { "attachments": {}, "cell_type": "markdown", "id": "373fd5ec", "metadata": { "cell_id": "dc6f466fd2994dbc80cf4e4da3be362f", "deepnote_cell_height": 83.5, "deepnote_cell_type": "markdown", "tags": [] }, "source": [ "# `RobustOCT` Examples" ] }, { "attachments": {}, "cell_type": "markdown", "id": "65b403f6", "metadata": { "cell_id": "00001-5aa34266-f09d-41a6-aea8-7d1b5d36257a", "deepnote_cell_height": 144.796875, "deepnote_cell_type": "markdown", "owner_user_id": "3c75c737-6c37-465a-aa9b-4b298c75816b" }, "source": [ "\n", "\n", "## Example 1: Synthetic Data Without Specified Shifts\n", "If costs and/or budget is not specified, then we will produce the same result as an optimal strong classification tree.\n", "\n", "As an example, say that we are given this training set:" ] }, { "cell_type": "code", "execution_count": null, "id": "22254eb6", "metadata": { "cell_id": "00002-ec1b8b46-8347-44be-88eb-d2c0ff104528", "deepnote_cell_height": 508, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 1198, "execution_start": 1664769662078, "source_hash": "986acf91" }, "outputs": [], "source": [ "\"\"\"\n", " X2\n", " | |\n", " | |\n", " 1 + + | -\n", " | | \n", " |---------------|-------------\n", " | |\n", " 0 - - - - | + + +\n", " | - - - |\n", " |______0________|_______1_______X1\n", "\"\"\"\n", "X = np.array(\n", " [\n", " [0, 0],\n", " [0, 0],\n", " [0, 0],\n", " [0, 0],\n", " [0, 0],\n", " [0, 0],\n", " [0, 0],\n", " [1, 0],\n", " [1, 0],\n", " [1, 0],\n", " [1, 1],\n", " [0, 1],\n", " [0, 1],\n", " ]\n", ")\n", "X = pd.DataFrame(X, columns=[\"X1\", \"X2\"])\n", "\n", "y = np.array([0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1])" ] }, { "attachments": {}, "cell_type": "markdown", "id": "87b66e70", "metadata": { "cell_id": "f5d869ce3ae0415ca6b86aab044f6460", "deepnote_cell_height": 74.796875, "deepnote_cell_type": "markdown", "owner_user_id": "2bc00ffa-8da0-4406-b26d-9986ca85f208", "tags": [] }, "source": [ "If either `costs` or `budget` is not specified, the optimal classification tree will be produced (i.e., a tree that does not account for distribution shifts)." ] }, { "cell_type": "code", "execution_count": null, "id": "bccf620c", "metadata": { "cell_id": "00003-ba6613f0-07d0-4855-badb-8ab0fcfd487c", "deepnote_cell_height": 250.6875, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 565, "execution_start": 1664769682643, "scrolled": true, "source_hash": "7db50d93" }, "outputs": [], "source": [ "robust_classifier = RobustOCT(\n", " solver=\"gurobi\",\n", " depth = 2, \n", " time_limit = 100,\n", " )\n", "robust_classifier.fit(X, y)\n", "predictions = robust_classifier.predict(X)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "4f7e4e06", "metadata": { "cell_id": "8504ebdbd679412689f83492cd8f7a4f", "deepnote_cell_height": 362.734375, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 357, "execution_start": 1664769689116, "source_hash": "a3f1115a", "tags": [] }, "outputs": [], "source": [ "robust_classifier.print_tree()" ] }, { "cell_type": "code", "execution_count": null, "id": "95997b6a", "metadata": { "cell_id": "272908eee5d346a3b82cf3a0aa62c4e1", "deepnote_cell_height": 552, "deepnote_cell_type": "code", "deepnote_output_heights": [ 406 ], "deepnote_to_be_reexecuted": false, "execution_millis": 1265, "execution_start": 1664769691141, "source_hash": "f06c109e", "tags": [] }, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(10, 5))\n", "robust_classifier.plot_tree()\n", "plt.show()" ] }, { "attachments": {}, "cell_type": "markdown", "id": "b73ea5f5", "metadata": { "cell_id": "00004-c066bdea-caf9-4089-9fa1-ee2e8d323ad6", "deepnote_cell_height": 231.984375, "deepnote_cell_type": "markdown" }, "source": [ "## Example 2: synthetic data with specified shifts\n", "\n", "We take the same synthetic data from Example 1, but now add distribution shifts with the following schema:\n", "- For 5 samples at $[0,0]$, pay a cost of 1 to perturb $X_1$ and get $[1,0]$\n", "- For the 1 sample at $[1,1]$, pay a cost of 1 to perturb $X_2$ to get $[1,0]$\n", "- All other perturbations are not allowed\n", "\n", "First, define these costs, which have the same shape and features as your input sample." ] }, { "cell_type": "code", "execution_count": null, "id": "5816b3a0", "metadata": { "cell_id": "00005-ec8f4aac-e1c1-486b-a079-6d46a8a9ef3b", "deepnote_cell_height": 166, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 1, "execution_start": 1664769701276, "source_hash": "92e9e3b9" }, "outputs": [], "source": [ "# Note: 10 is a proxy for infinite cost, as it is over the allowed budgets we will specify\n", "costs = np.array([[1,10],[1,10],[1,10],[1,10],[1,10],[10,10],[10,10],\n", " [10,10],[10,10],[10,10],\n", " [10,1],\n", " [10,10],[10,10]])\n", "costs = pd.DataFrame(costs, columns=['X1', 'X2'])" ] }, { "attachments": {}, "cell_type": "markdown", "id": "e7a8b62b", "metadata": { "cell_id": "00006-17d04a3f-d99a-4770-8dd8-6a188182b3cb", "deepnote_cell_height": 74.796875, "deepnote_cell_type": "markdown" }, "source": [ "\n", "When the budget is 2 (corresponding to the variable ε), we don't see a change in the tree from Example 1 since for this dataset, the budget is small and thus the level of robustness is small." ] }, { "cell_type": "code", "execution_count": null, "id": "1a79ac9d", "metadata": { "cell_id": "00007-e8bcdb8a-f74b-45a4-acd3-a7cb8592fb0a", "deepnote_cell_height": 400, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 308, "execution_start": 1664769713072, "scrolled": true, "source_hash": "9face3f" }, "outputs": [], "source": [ "# Same data as Example 1\n", "X = np.array(\n", " [\n", " [0, 0],\n", " [0, 0],\n", " [0, 0],\n", " [0, 0],\n", " [0, 0],\n", " [0, 0],\n", " [0, 0],\n", " [1, 0],\n", " [1, 0],\n", " [1, 0],\n", " [1, 1],\n", " [0, 1],\n", " [0, 1],\n", " ]\n", ")\n", "X = pd.DataFrame(X, columns=[\"X1\", \"X2\"])\n", "\n", "y = np.array([0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1])\n" ] }, { "cell_type": "code", "execution_count": null, "id": "326b256a", "metadata": {}, "outputs": [], "source": [ "robust_classifier = RobustOCT(\n", " solver=\"gurobi\",\n", " depth=2,\n", " time_limit=100\n", ")\n", "robust_classifier.fit(X, y, costs=costs, budget=2)\n", "predictions = robust_classifier.predict(X)" ] }, { "cell_type": "code", "execution_count": null, "id": "f92977d3", "metadata": { "cell_id": "9116a32b75b94436bfeedf91c1f77815", "deepnote_cell_height": 362.734375, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 2, "execution_start": 1664769714733, "source_hash": "a3f1115a", "tags": [] }, "outputs": [], "source": [ "robust_classifier.print_tree()" ] }, { "cell_type": "code", "execution_count": null, "id": "0d6acd42", "metadata": { "cell_id": "4da674ada20e4400a2a3511f53d341b0", "deepnote_cell_height": 534, "deepnote_cell_type": "code", "deepnote_output_heights": [ 406 ], "deepnote_to_be_reexecuted": false, "execution_millis": 983, "execution_start": 1664769715734, "source_hash": "fbc9df0d", "tags": [] }, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(10, 5)) \n", "robust_classifier.plot_tree()\n", "plt.show()" ] }, { "attachments": {}, "cell_type": "markdown", "id": "0905099c", "metadata": { "cell_id": "00008-73938505-0bb7-4f8e-a9ef-b6b4e57515e0", "deepnote_cell_height": 52.390625, "deepnote_cell_type": "markdown" }, "source": [ "\n", "\n", "But when the budget is increased to 5 (adding more robustness), we see a change in the tree." ] }, { "cell_type": "code", "execution_count": null, "id": "f8d8aa20", "metadata": { "cell_id": "00009-84267e8d-61ab-4b67-96a3-267ef2c17411", "deepnote_cell_height": 112, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 246, "execution_start": 1664769729224, "scrolled": true, "source_hash": "99f32af8" }, "outputs": [], "source": [ "robust_classifier = RobustOCT(\n", " solver=\"gurobi\",\n", " depth=2,\n", " time_limit=100\n", ")\n", "robust_classifier.fit(X, y, costs=costs, budget=5)\n", "predictions = robust_classifier.predict(X)" ] }, { "cell_type": "code", "execution_count": null, "id": "66797dea", "metadata": {}, "outputs": [], "source": [ "robust_classifier.print_tree()" ] }, { "cell_type": "code", "execution_count": null, "id": "7dbda0fa", "metadata": {}, "outputs": [], "source": [ "robust_classifier.plot_tree()\n", "plt.show()" ] }, { "attachments": {}, "cell_type": "markdown", "id": "d451b6c3", "metadata": { "cell_id": "00010-a64753d8-356d-4c57-8ad0-03c7c908edf9", "deepnote_cell_height": 130.796875, "deepnote_cell_type": "markdown" }, "source": [ "## Example 3: UCI data example\n", "Here, we'll see the benefits of using robust optimization by perturbing the test set. We will use the MONK's Problems dataset from the UCI Machine Learning Repository.\n", "\n", "\n" ] }, { "attachments": {}, "cell_type": "markdown", "id": "d8632b13", "metadata": { "cell_id": "41f21d35d18f40339ee6a1475ca5b04d", "deepnote_cell_height": 52.390625, "deepnote_cell_type": "markdown", "tags": [] }, "source": [ "Fetch data and split to train and test" ] }, { "cell_type": "code", "execution_count": null, "id": "44b9d106", "metadata": { "cell_id": "00012-ca63f712-e49e-423a-97e6-ef5b95341980", "deepnote_cell_height": 268.6875, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 508, "execution_start": 1664769746611, "scrolled": true, "source_hash": "6f3fb272" }, "outputs": [], "source": [ "\"\"\"Fetch data and split to train and test\"\"\"\n", "data, y = robust_example()\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " data, y, test_size=0.25, random_state=2\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "id": "cc714a7c", "metadata": { "cell_id": "00013-6684517e-c0ae-464f-8632-05eacb7895eb", "deepnote_cell_height": 52.390625, "deepnote_cell_type": "markdown" }, "source": [ "For sake of comparison, train a classification tree that does not consider the scenario where there is a distribution shift:" ] }, { "cell_type": "code", "execution_count": null, "id": "bdf7635f", "metadata": { "cell_id": "00014-6bf44759-2856-4316-81b4-1f9491d09c83", "deepnote_cell_height": 950.1875, "deepnote_cell_type": "code", "deepnote_output_heights": [ null, 20.1875 ], "deepnote_to_be_reexecuted": false, "execution_millis": 17487, "execution_start": 1664769775115, "scrolled": true, "source_hash": "ec914ba0" }, "outputs": [], "source": [ "\"\"\"Train a non-robust tree for comparison\"\"\"\n", "\n", "# If you define no uncertainty, you get an optimal tree without regularization that maximizes accuracy\n", "non_robust_classifier = RobustOCT(solver=\"gurobi\", depth=2, time_limit=300)\n", "non_robust_classifier.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": null, "id": "885385b9", "metadata": { "cell_id": "cede983182b840ca830bf41e836d1abb", "deepnote_cell_height": 362.734375, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 680, "execution_start": 1664769792015, "source_hash": "388e377f", "tags": [] }, "outputs": [], "source": [ "non_robust_classifier.print_tree()" ] }, { "cell_type": "code", "execution_count": null, "id": "f122efea", "metadata": { "cell_id": "d2858316c41d4fa0b972e1dd756b0fd1", "deepnote_cell_height": 534, "deepnote_cell_type": "code", "deepnote_output_heights": [ 406 ], "deepnote_to_be_reexecuted": false, "execution_millis": 747, "execution_start": 1664769792016, "source_hash": "3d490653", "tags": [] }, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(10, 5)) \n", "non_robust_classifier.plot_tree()\n", "plt.show()" ] }, { "attachments": {}, "cell_type": "markdown", "id": "77bf18eb", "metadata": { "cell_id": "00015-511ab703-4b9f-4c99-8b26-95f27e2ee80e", "deepnote_cell_height": 97.1875, "deepnote_cell_type": "markdown" }, "source": [ "Train a robust tree. First, define the uncertainty. Here, we will generate a probability of certainty for each feature randomly (in practice, you would need to use some guess from domain knowledge). For simplicity, we will not change this probability by data sample $i$. We also define $\\lambda = 0.9$, which in practice must be tuned.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "b641c408", "metadata": { "cell_id": "00016-14b459f8-9786-4b71-9007-d9a68d7c69aa", "deepnote_cell_height": 257.390625, "deepnote_cell_type": "code", "deepnote_output_heights": [ 39.390625 ], "deepnote_to_be_reexecuted": false, "execution_millis": 680, "execution_start": 1664769798155, "scrolled": true, "source_hash": "5f369a82" }, "outputs": [], "source": [ "\"\"\"Generate q_f values for each feature (i.e. probability of certainty for feature f)\"\"\"\n", "np.random.seed(42)\n", "q_f = np.random.normal(loc=0.9, scale=0.1, size=len(X_train.columns))\n", "# Snap q_f to range [0,1]\n", "q_f[q_f <= 0] = np.nextafter(np.float32(0), np.float32(1))\n", "q_f[q_f > 1] = 1.0\n", "\n", "q_f" ] }, { "attachments": {}, "cell_type": "markdown", "id": "d8118dcd", "metadata": { "cell_id": "00017-e2c44d63-a2a7-40c3-873b-173219fa4020", "deepnote_cell_height": 52.390625, "deepnote_cell_type": "markdown" }, "source": [ "Calibrate the `costs` and `budget` parameters for the `fit` function." ] }, { "cell_type": "code", "execution_count": null, "id": "7a563a91", "metadata": { "cell_id": "00018-8348cb86-a857-4d9d-a5ee-44cda4446659", "deepnote_cell_height": 184.1875, "deepnote_cell_type": "code", "deepnote_output_heights": [ 20.1875 ], "deepnote_to_be_reexecuted": false, "execution_millis": 334, "execution_start": 1664769801529, "source_hash": "8febf87c" }, "outputs": [], "source": [ "\"\"\"Define budget of uncertainty\"\"\"\n", "\n", "l = 0.9 # Lambda value between 0 and 1\n", "budget = -1 * X_train.shape[0] * log(l)\n", "budget" ] }, { "cell_type": "code", "execution_count": null, "id": "68ef855d", "metadata": { "cell_id": "00019-a1add7b6-8e21-46a1-87b6-e6a2d74ef071", "deepnote_cell_height": 793, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 28, "execution_start": 1664769805562, "scrolled": true, "source_hash": "7f3949b" }, "outputs": [], "source": [ "\"\"\"Based on q_f values, create costs of uncertainty\"\"\"\n", "costs = deepcopy(X_train)\n", "costs = costs.astype(\"float\")\n", "for f in range(len(q_f)):\n", " if q_f[f] == 1:\n", " costs[costs.columns[f]] = budget + 1 # no uncertainty = \"infinite\" cost\n", " else:\n", " costs[costs.columns[f]] = -1 * log(1 - q_f[f])\n", "\n", "costs" ] }, { "attachments": {}, "cell_type": "markdown", "id": "873b1077", "metadata": { "cell_id": "873b53227f2847c48fd956daa42241d6", "deepnote_cell_height": 52.390625, "deepnote_cell_type": "markdown", "tags": [] }, "source": [ "Train the robust tree using the costs and budget." ] }, { "cell_type": "code", "execution_count": null, "id": "8dc237ab", "metadata": { "cell_id": "00020-17db6a05-404c-4fb0-9b35-d58abc3aa541", "deepnote_cell_height": 184.1875, "deepnote_cell_type": "code", "deepnote_output_heights": [ 20.1875 ], "deepnote_to_be_reexecuted": false, "execution_millis": 115497, "execution_start": 1664769815612, "scrolled": true, "source_hash": "ae6f9fa2" }, "outputs": [], "source": [ "robust_classifier = RobustOCT(\n", " solver=\"gurobi\",\n", " depth=2,\n", " time_limit=200,\n", ")\n", "robust_classifier.fit(X_train, y_train, costs=costs, budget=budget)" ] }, { "cell_type": "code", "execution_count": null, "id": "5ee04af2", "metadata": { "cell_id": "6ea60076b5ff4d4db35fef0d61992e3c", "deepnote_cell_height": 362.734375, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 2, "execution_start": 1664769931108, "source_hash": "a3f1115a", "tags": [] }, "outputs": [], "source": [ "robust_classifier.print_tree()" ] }, { "cell_type": "code", "execution_count": null, "id": "986d5b8f", "metadata": { "cell_id": "02b308f6ab99429d8ffe4d3f4ae0ada7", "deepnote_cell_height": 534, "deepnote_cell_type": "code", "deepnote_output_heights": [ 406 ], "deepnote_to_be_reexecuted": false, "execution_millis": 389, "execution_start": 1664769931109, "source_hash": "fbc9df0d", "tags": [] }, "outputs": [], "source": [ "robust_classifier.plot_tree()\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "cf5d1de9", "metadata": { "cell_id": "00021-3cc7fda4-c89b-43f7-bbf4-7e6ca8a7298e", "deepnote_cell_height": 260.78125, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 7, "execution_start": 1664769931564, "source_hash": "e4bb380a" }, "outputs": [], "source": [ "print(\n", " \"Non-robust training accuracy: \",\n", " accuracy_score(y_train, non_robust_classifier.predict(X_train)),\n", ")\n", "print(\n", " \"Robust training accuracy: \",\n", " accuracy_score(y_train, robust_classifier.predict(X_train)),\n", ")\n", "print(\n", " \"Non-robust test accuracy: \",\n", " accuracy_score(y_test, non_robust_classifier.predict(X_test)),\n", ")\n", "print(\n", " \"Robust test accuracy: \",\n", " accuracy_score(y_test, robust_classifier.predict(X_test)),\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "id": "10f24f30", "metadata": { "cell_id": "00022-17f6f113-6d42-41de-97a4-425263e19c21", "deepnote_cell_height": 74.796875, "deepnote_cell_type": "markdown" }, "source": [ "To measure the performance of the trained models, perturb the test data based off of our known certainties of each feature (to simulate a distribtion shift), and then see how well each tree performs against the perturbed data\n" ] }, { "cell_type": "code", "execution_count": null, "id": "785bec50", "metadata": { "cell_id": "00023-6873bd0d-ac31-4d00-9fd8-74d79f46f931", "deepnote_cell_height": 454, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 5400, "execution_start": 1664769931576, "source_hash": "fa02f8dd" }, "outputs": [], "source": [ "def perturb(data, q_f, seed):\n", " \"\"\"Perturb X given q_f based off of the symmetric geometric distribution\"\"\"\n", " new_data = deepcopy(data)\n", " np.random.seed(seed)\n", " # Perturbation of features\n", " for f in range(len(new_data.columns)):\n", " perturbations = np.random.geometric(q_f[f], size=new_data.shape[0])\n", " perturbations = perturbations - 1 # Support should be 0,1,2,...\n", " signs = (2 * np.random.binomial(1, 0.5, size=new_data.shape[0])) - 1\n", " perturbations = perturbations * signs\n", " new_data[new_data.columns[f]] = new_data[new_data.columns[f]] + perturbations\n", " return new_data\n", "\n", "\n", "\"\"\"Obtain 1000 different perturbed test sets, and record accuracies\"\"\"\n", "non_robust_acc = []\n", "robust_acc = []\n", "for s in range(1, 1001):\n", " X_test_perturbed = perturb(X_test, q_f, s)\n", " non_robust_pred = non_robust_classifier.predict(X_test_perturbed)\n", " robust_pred = robust_classifier.predict(X_test_perturbed)\n", " non_robust_acc += [accuracy_score(y_test, non_robust_pred)]\n", " robust_acc += [accuracy_score(y_test, robust_pred)]" ] }, { "cell_type": "code", "execution_count": null, "id": "0b6a4289", "metadata": { "cell_id": "00024-32351bf0-26fb-4ff1-9892-f9e0b484ca12", "deepnote_cell_height": 219.78125, "deepnote_cell_type": "code", "deepnote_to_be_reexecuted": false, "execution_millis": 4, "execution_start": 1664769936979, "source_hash": "8fcb1c0a" }, "outputs": [], "source": [ "print(\"Worst-case accuracy (Non-Robust Tree): \", min(non_robust_acc))\n", "print(\"Worst-case accuracy (Robust Tree): \", min(robust_acc))\n", "print(\n", " \"Average accuracy (Non-Robust Tree): \", sum(non_robust_acc) / len(non_robust_acc)\n", ")\n", "print(\"Average accuracy (Robust Tree): \", sum(robust_acc) / len(robust_acc))" ] }, { "attachments": {}, "cell_type": "markdown", "id": "117c0143", "metadata": { "cell_id": "6e94893cd87747d48562368a596e05cc", "deepnote_cell_height": 341.59375, "deepnote_cell_type": "markdown", "tags": [] }, "source": [ "## References\n", "* Justin, N., Aghaei, S., Gómez, A., & Vayanos, P. (2021). Optimal robust classification trees. *The AAAI-2022 Workshop on Adversarial Machine Learning and Beyond*. https://openreview.net/pdf?id=HbasA9ysA3\n", "* Justin, N., Aghaei, S., Gómez, A., & Vayanos, P. (2023). Learning optimal classification trees robust to distribution shifts. *arXiv preprint* arXiv:2310.17772.\n", "* Dua, D. and Graff, C. (2019). [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml). Irvine, CA: University of California, School of Information and Computer Science." ] }, { "cell_type": "code", "execution_count": null, "id": "7b0066de-a945-4aa4-b21c-affc595e9931", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "deepnote": {}, "deepnote_execution_queue": [], "deepnote_notebook_id": "48fbaca3-de19-4cc4-b8bd-5da9f4b14110", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.2" } }, "nbformat": 4, "nbformat_minor": 5 }