{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Supervised learning using scikit-learn"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The goal of this tutorial is to introduce you to the scikit libraries for classification. We will also cover feature selection, and evaluation."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"import numpy as np\n",
"import scipy.sparse as sp_sparse\n",
"\n",
"import matplotlib.pyplot as plt\n",
"\n",
"import sklearn as sk\n",
"import sklearn.datasets as sk_data\n",
"import sklearn.metrics as metrics\n",
"\n",
"import seaborn as sns\n",
"\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Feature Selection\n",
"\n",
"Feature selection is about finding the best features for your classifier. This may be important if you do not have enough training data. The idea is to find metrics that either characterize the features by themselves, or with respect to the class we want to predict, or with respect to other features.\n",
"\n",
"http://scikit-learn.org/stable/modules/feature_selection.html"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"#### Variance Threshold\n",
"\n",
"The **VarianceThreshold** selection drops features whose variance is below some threshold. If we have binary features we can estimate the treshold exactly so as to guarantee a specific ratio of 0's and 1's"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[0 0 1]\n",
" [0 1 0]\n",
" [1 0 0]\n",
" [0 1 1]\n",
" [0 1 0]\n",
" [0 1 1]]\n"
]
},
{
"data": {
"text/plain": [
"array([[0, 1],\n",
" [1, 0],\n",
" [0, 0],\n",
" [1, 1],\n",
" [1, 0],\n",
" [1, 1]])"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.feature_selection import VarianceThreshold\n",
"X = [[0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 1, 1], [0, 1, 0], [0, 1, 1]]\n",
"print(np.array(X))\n",
"sel = VarianceThreshold(threshold=(.8 * (1 - .8)))\n",
"sel.fit_transform(X)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[4.9 3. 1.4 0.2]\n",
" [4.7 3.2 1.3 0.2]\n",
" [4.6 3.1 1.5 0.2]\n",
" [5. 3.6 1.4 0.2]\n",
" [5.4 3.9 1.7 0.4]\n",
" [4.6 3.4 1.4 0.3]\n",
" [5. 3.4 1.5 0.2]\n",
" [4.4 2.9 1.4 0.2]\n",
" [4.9 3.1 1.5 0.1]]\n",
"[0.68112222 0.18871289 3.09550267 0.57713289]\n"
]
},
{
"data": {
"text/plain": [
"array([[4.9, 1.4, 0.2],\n",
" [4.7, 1.3, 0.2],\n",
" [4.6, 1.5, 0.2],\n",
" [5. , 1.4, 0.2],\n",
" [5.4, 1.7, 0.4],\n",
" [4.6, 1.4, 0.3],\n",
" [5. , 1.5, 0.2],\n",
" [4.4, 1.4, 0.2],\n",
" [4.9, 1.5, 0.1]])"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import sklearn.datasets as sk_data\n",
"iris = sk_data.load_iris()\n",
"X = iris.data\n",
"print(X[1:10,:])\n",
"print(X.var(axis = 0))\n",
"sel = VarianceThreshold(threshold=0.2)\n",
"sel.fit_transform(X)[1:10]"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"#### Univariate Feature Selection ####\n",
"\n",
"A more sophisticated feature selection technique uses a test to determine if a feature and the class label are independent. An example of such a test is the [chi-square test](https://en.wikipedia.org/wiki/Chi-squared_test) (there are more)\n",
"\n",
"In this case we keep the features with high chi-square score and low p-value. \n",
"\n",
"The features with the lowest scores and highest p-values are rejected.\n",
"\n",
"The chi-square test is usually applied on categorical data."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(150, 4)\n",
"Features:\n",
"[[4.9 3. 1.4 0.2]\n",
" [4.7 3.2 1.3 0.2]\n",
" [4.6 3.1 1.5 0.2]\n",
" [5. 3.6 1.4 0.2]\n",
" [5.4 3.9 1.7 0.4]\n",
" [4.6 3.4 1.4 0.3]\n",
" [5. 3.4 1.5 0.2]\n",
" [4.4 2.9 1.4 0.2]\n",
" [4.9 3.1 1.5 0.1]]\n",
"Labels:\n",
"[0 0 0 0 0 0 0 0 0]\n",
"Selected Features:\n",
"[[1.4 0.2]\n",
" [1.3 0.2]\n",
" [1.5 0.2]\n",
" [1.4 0.2]\n",
" [1.7 0.4]\n",
" [1.4 0.3]\n",
" [1.5 0.2]\n",
" [1.4 0.2]\n",
" [1.5 0.1]]\n"
]
}
],
"source": [
"from sklearn.feature_selection import SelectKBest\n",
"from sklearn.feature_selection import chi2\n",
"iris = sk_data.load_iris()\n",
"X, y = iris.data, iris.target\n",
"print(X.shape)\n",
"print('Features:')\n",
"print(X[1:10,:])\n",
"print('Labels:')\n",
"print(y[1:10])\n",
"sel = SelectKBest(chi2, k=2)\n",
"X_new = sel.fit_transform(X, y)\n",
"print('Selected Features:')\n",
"print(X_new[1:10])"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"The chi-square values and the p-values between features and target variable (X columns and y)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Chi2 values\n",
"[ 10.81782088 3.7107283 116.31261309 67.0483602 ]\n",
"Chi2 values\n",
"[ 10.81782088 3.7107283 116.31261309 67.0483602 ]\n",
"p-values\n",
"[4.47651499e-03 1.56395980e-01 5.53397228e-26 2.75824965e-15]\n"
]
}
],
"source": [
"print('Chi2 values')\n",
"print(sel.scores_)\n",
"c,p = sk.feature_selection.chi2(X, y)\n",
"print('Chi2 values')\n",
"print(c) #The chi-square value between X columns and y\n",
"print('p-values')\n",
"print(p) #The p-value for the test"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Supervised Learning\n",
"\n",
"Python has several classes and objects for implementing different supervised learning techniques such as Regression and Classification. \n",
"\n",
"Regardless of the model being implemented, the following methods are implemented:\n",
"\n",
"The method **fit()** takes the training data and labels/values, and trains the model\n",
"\n",
"The method **predict()** takes as input the test data and applies the model. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Preparing the data\n",
"\n",
"To perform classification we first need to prepare the data into train and test datasets."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"sample of data\n",
"[[5.1 3.5 1.4 0.2]\n",
" [4.9 3. 1.4 0.2]\n",
" [4.7 3.2 1.3 0.2]\n",
" [4.6 3.1 1.5 0.2]\n",
" [5. 3.6 1.4 0.2]]\n",
"the class labels vector\n",
"[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
" 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
" 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2\n",
" 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
" 2 2]\n",
"the names of the classes: ['setosa' 'versicolor' 'virginica']\n",
"['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']\n"
]
}
],
"source": [
"from sklearn.datasets import load_iris\n",
"import sklearn.utils as utils\n",
"\n",
"iris = load_iris()\n",
"print(\"sample of data\")\n",
"print(iris.data[:5,:])\n",
"print(\"the class labels vector\")\n",
"print(iris.target)\n",
"print(\"the names of the classes:\",iris.target_names)\n",
"print(iris.feature_names)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Randomly shuffle the data. This is useful to know that the data is in random order"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(150, 4)\n",
"(150,)\n",
"[0 1 1 0 2 1 2 0 0 2 1 0 2 1 1 0 1 1 0 0 1 1 1 0 2 1 0 0 1 2 1 2 1 2 2 0 1\n",
" 0 1 2 2 0 2 2 1 2 0 0 0 1 0 0 2 2 2 2 2 1 2 1 0 2 2 0 0 2 0 2 2 1 1 2 2 0\n",
" 1 1 2 1 2 1 0 0 0 2 0 1 2 2 0 0 1 0 2 1 2 2 1 2 2 1 0 1 0 1 1 0 1 0 0 2 2\n",
" 2 0 0 1 0 2 0 2 2 0 2 0 1 0 1 1 0 0 1 0 1 1 0 1 1 1 1 2 0 0 2 1 2 1 2 2 1\n",
" 2 0]\n"
]
}
],
"source": [
"X, y = utils.shuffle(iris.data, iris.target, random_state=1) #shuffle the data\n",
"print(X.shape)\n",
"print(y.shape)\n",
"print(y)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Select a subset for training and a subset for testing"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(100, 4) (100,)\n",
"(50, 4) (50,)\n"
]
}
],
"source": [
"train_set_size = 100\n",
"X_train = X[:train_set_size] # selects first 100 rows (examples) for train set\n",
"y_train = y[:train_set_size]\n",
"X_test = X[train_set_size:] # selects from row 100 until the last one for test set\n",
"y_test = y[train_set_size:]\n",
"print(X_train.shape, y_train.shape)\n",
"print(X_test.shape, y_test.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"We can also use the train_test_split function of python for splitting the data into train and test sets. In this case you do not need the random shuffling (but it does not hurt)."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=0)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Classification models\n",
"\n",
"http://scikit-learn.org/stable/supervised_learning.html#supervised-learning\n",
"\n",
"Python has classes and objects that implement the different classification techniques that we described in class. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Decision Trees\n",
"http://scikit-learn.org/stable/modules/tree.html\n",
"\n",
"Train and apply a decision tree classifier. The default score computed in the classifier object is the accuracy. Decision trees can also give us probabilities"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"classifier accuracy: 0.9\n",
"classifier predictions: [2 2 2 0 0 0 2 2 2 2]\n",
"ground truth labels : [1 2 2 0 0 0 2 2 2 2]\n",
"[[0. 0. 1.]\n",
" [0. 0. 1.]\n",
" [0. 0. 1.]\n",
" [1. 0. 0.]\n",
" [1. 0. 0.]\n",
" [1. 0. 0.]\n",
" [0. 0. 1.]\n",
" [0. 0. 1.]\n",
" [0. 0. 1.]\n",
" [0. 0. 1.]]\n"
]
}
],
"source": [
"from sklearn import tree\n",
"\n",
"dtree = tree.DecisionTreeClassifier()\n",
"dtree = dtree.fit(X_train, y_train)\n",
"\n",
"print(\"classifier accuracy:\",dtree.score(X_test,y_test))\n",
"\n",
"y_pred = dtree.predict(X_test)\n",
"y_prob = dtree.predict_proba(X_test)\n",
"print(\"classifier predictions:\",y_pred[:10])\n",
"print(\"ground truth labels :\",y_test[:10])\n",
"print(y_prob[:10])"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Compute some more metrics"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"accuracy: 0.9\n",
"\n",
"Confusion matrix\n",
"[[20 0 0]\n",
" [ 0 15 4]\n",
" [ 0 2 19]]\n",
"\n",
"Precision Score per class\n",
"[1. 0.88235294 0.82608696]\n",
"\n",
"Average Precision Score\n",
"0.9018755328218244\n",
"\n",
"Recall Score per class\n",
"[1. 0.78947368 0.9047619 ]\n",
"\n",
"Average Recall Score\n",
"0.9\n",
"\n",
"F1-score Score per class\n",
"[1. 0.83333333 0.86363636]\n",
"\n",
"Average F1 Score\n",
"0.8994949494949495\n"
]
}
],
"source": [
"print(\"accuracy:\",metrics.accuracy_score(y_test,y_pred))\n",
"print(\"\\nConfusion matrix\")\n",
"print(metrics.confusion_matrix(y_test,y_pred))\n",
"print(\"\\nPrecision Score per class\")\n",
"print(metrics.precision_score(y_test,y_pred,average=None))\n",
"print(\"\\nAverage Precision Score\")\n",
"print(metrics.precision_score(y_test,y_pred,average='weighted'))\n",
"print(\"\\nRecall Score per class\")\n",
"print(metrics.recall_score(y_test,y_pred,average=None))\n",
"print(\"\\nAverage Recall Score\")\n",
"print(metrics.recall_score(y_test,y_pred,average='weighted'))\n",
"print(\"\\nF1-score Score per class\")\n",
"print(metrics.f1_score(y_test,y_pred,average=None))\n",
"print(\"\\nAverage F1 Score\")\n",
"print(metrics.f1_score(y_test,y_pred,average='weighted'))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Visualize the decision tree.\n",
"\n",
"For this you will need to install the package python-graphviz"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']\n"
]
},
{
"data": {
"image/svg+xml": [
"\r\n",
"\r\n",
"\r\n",
"\r\n",
"\r\n"
],
"text/plain": [
""
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#conda install python-graphviz\n",
"import graphviz \n",
"print(iris.feature_names)\n",
"dot_data = tree.export_graphviz(dtree,out_file=None)\n",
"graph = graphviz.Source(dot_data)\n",
"graph"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.9166666666666666\n"
]
},
{
"data": {
"image/svg+xml": [
"\r\n",
"\r\n",
"\r\n",
"\r\n",
"\r\n"
],
"text/plain": [
""
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dtree2 = tree.DecisionTreeClassifier(max_depth=2)\n",
"dtree2 = dtree2.fit(X_train, y_train)\n",
"print(dtree2.score(X_test,y_test))\n",
"dot_data2 = tree.export_graphviz(dtree2,out_file=None)\n",
"graph2 = graphviz.Source(dot_data2)\n",
"graph2"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"\n",
"### k-NN Classification ###\n",
"\n",
"https://scikit-learn.org/stable/modules/neighbors.html#classification"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"classifier score: 0.9333333333333333\n",
"\n",
"accuracy: 0.9333333333333333\n",
"\n",
"Confusion matrix\n",
"[[20 0 0]\n",
" [ 0 16 3]\n",
" [ 0 1 20]]\n",
"\n",
"Precision Score per class\n",
"[1. 0.94117647 0.86956522]\n",
"\n",
"Average Precision Score\n",
"0.9357203751065644\n",
"\n",
"Recall Score per class\n",
"[1. 0.84210526 0.95238095]\n",
"\n",
"Average Recall Score\n",
"0.9333333333333333\n",
"\n",
"F1-score Score per class\n",
"[1. 0.88888889 0.90909091]\n",
"\n",
"Average F1 Score\n",
"0.9329966329966328\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\neighbors\\_classification.py:228: FutureWarning: Unlike other reduction functions (e.g. `skew`, `kurtosis`), the default behavior of `mode` typically preserves the axis it acts along. In SciPy 1.11.0, this behavior will change: the default value of `keepdims` will become False, the `axis` over which the statistic is taken will be eliminated, and the value None will no longer be accepted. Set `keepdims` to True or False to avoid this warning.\n",
" mode, _ = stats.mode(_y[neigh_ind, k], axis=1)\n",
"C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\neighbors\\_classification.py:228: FutureWarning: Unlike other reduction functions (e.g. `skew`, `kurtosis`), the default behavior of `mode` typically preserves the axis it acts along. In SciPy 1.11.0, this behavior will change: the default value of `keepdims` will become False, the `axis` over which the statistic is taken will be eliminated, and the value None will no longer be accepted. Set `keepdims` to True or False to avoid this warning.\n",
" mode, _ = stats.mode(_y[neigh_ind, k], axis=1)\n"
]
}
],
"source": [
"from sklearn.neighbors import KNeighborsClassifier\n",
"\n",
"knn = KNeighborsClassifier(n_neighbors=3)\n",
"knn.fit(X_train,y_train)\n",
"print(\"classifier score:\", knn.score(X_test,y_test))\n",
"\n",
"y_pred = knn.predict(X_test)\n",
"\n",
"print(\"\\naccuracy:\",metrics.accuracy_score(y_test,y_pred))\n",
"print(\"\\nConfusion matrix\")\n",
"print(metrics.confusion_matrix(y_test,y_pred))\n",
"print(\"\\nPrecision Score per class\")\n",
"print(metrics.precision_score(y_test,y_pred,average=None))\n",
"print(\"\\nAverage Precision Score\")\n",
"print(metrics.precision_score(y_test,y_pred,average='weighted'))\n",
"print(\"\\nRecall Score per class\")\n",
"print(metrics.recall_score(y_test,y_pred,average=None))\n",
"print(\"\\nAverage Recall Score\")\n",
"print(metrics.recall_score(y_test,y_pred,average='weighted'))\n",
"print(\"\\nF1-score Score per class\")\n",
"print(metrics.f1_score(y_test,y_pred,average=None))\n",
"print(\"\\nAverage F1 Score\")\n",
"print(metrics.f1_score(y_test,y_pred,average='weighted'))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### SVM Classification\n",
"\n",
"http://scikit-learn.org/stable/modules/svm.html\n",
"\n",
"http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"classifier score: 0.95\n",
"\n",
"accuracy: 0.95\n",
"\n",
"Confusion matrix\n",
"[[20 0 0]\n",
" [ 0 18 1]\n",
" [ 0 2 19]]\n",
"\n",
"Precision Score per class\n",
"[1. 0.9 0.95]\n",
"\n",
"Average Precision Score\n",
"0.9508333333333333\n",
"\n",
"Recall Score per class\n",
"[1. 0.94736842 0.9047619 ]\n",
"\n",
"Average Recall Score\n",
"0.95\n",
"\n",
"F1-score Score per class\n",
"[1. 0.92307692 0.92682927]\n",
"\n",
"Average F1 Score\n",
"0.9500312695434646\n"
]
}
],
"source": [
"from sklearn import svm\n",
"\n",
"#svm_clf = svm.LinearSVC()\n",
"#svm_clf = svm.SVC(kernel = 'poly')\n",
"svm_clf = svm.SVC()\n",
"svm_clf.fit(X_train,y_train)\n",
"print(\"classifier score:\",svm_clf.score(X_test,y_test))\n",
"y_pred = svm_clf.predict(X_test)\n",
"print(\"\\naccuracy:\",metrics.accuracy_score(y_test,y_pred))\n",
"print(\"\\nConfusion matrix\")\n",
"print(metrics.confusion_matrix(y_test,y_pred))\n",
"print(\"\\nPrecision Score per class\")\n",
"print(metrics.precision_score(y_test,y_pred,average=None))\n",
"print(\"\\nAverage Precision Score\")\n",
"print(metrics.precision_score(y_test,y_pred,average='weighted'))\n",
"print(\"\\nRecall Score per class\")\n",
"print(metrics.recall_score(y_test,y_pred,average=None))\n",
"print(\"\\nAverage Recall Score\")\n",
"print(metrics.recall_score(y_test,y_pred,average='weighted'))\n",
"print(\"\\nF1-score Score per class\")\n",
"print(metrics.f1_score(y_test,y_pred,average=None))\n",
"print(\"\\nAverage F1 Score\")\n",
"print(metrics.f1_score(y_test,y_pred,average='weighted'))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Logistic Regression ###\n",
"\n",
"http://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
"\n",
"http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"classifier score: 0.95\n",
"\n",
"accuracy: 0.95\n",
"\n",
"Confusion matrix\n",
"[[20 0 0]\n",
" [ 0 17 2]\n",
" [ 0 1 20]]\n",
"\n",
"Precision Score per class\n",
"[1. 0.94444444 0.90909091]\n",
"\n",
"Average Precision Score\n",
"0.9505892255892257\n",
"\n",
"Recall Score per class\n",
"[1. 0.89473684 0.95238095]\n",
"\n",
"Average Recall Score\n",
"0.95\n",
"\n",
"F1-score Score per class\n",
"[1. 0.91891892 0.93023256]\n",
"\n",
"Average F1 Score\n",
"0.9499057196731615\n"
]
}
],
"source": [
"import sklearn.linear_model as linear_model\n",
"\n",
"lr_clf = linear_model.LogisticRegression(solver='lbfgs')\n",
"lr_clf.fit(X_train, y_train)\n",
"print(\"classifier score:\",lr_clf.score(X_test,y_test))\n",
"y_pred = lr_clf.predict(X_test)\n",
"print(\"\\naccuracy:\",metrics.accuracy_score(y_test,y_pred))\n",
"print(\"\\nConfusion matrix\")\n",
"print(metrics.confusion_matrix(y_test,y_pred))\n",
"print(\"\\nPrecision Score per class\")\n",
"print(metrics.precision_score(y_test,y_pred,average=None))\n",
"print(\"\\nAverage Precision Score\")\n",
"print(metrics.precision_score(y_test,y_pred,average='weighted'))\n",
"print(\"\\nRecall Score per class\")\n",
"print(metrics.recall_score(y_test,y_pred,average=None))\n",
"print(\"\\nAverage Recall Score\")\n",
"print(metrics.recall_score(y_test,y_pred,average='weighted'))\n",
"print(\"\\nF1-score Score per class\")\n",
"print(metrics.f1_score(y_test,y_pred,average=None))\n",
"print(\"\\nAverage F1 Score\")\n",
"print(metrics.f1_score(y_test,y_pred,average='weighted'))\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"For Logistic Regression we can also obtain the probabilities for the different classes"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Class Probabilities (first 10):\n",
"[[3.57734848e-03 4.78993829e-01 5.17428823e-01]\n",
" [1.14793557e-03 4.05790308e-01 5.93061756e-01]\n",
" [3.93057726e-05 6.43403533e-02 9.35620341e-01]\n",
" [9.57992494e-01 4.20057847e-02 1.72111135e-06]\n",
" [9.43314566e-01 5.66824755e-02 2.95895565e-06]\n",
" [9.82884159e-01 1.71154037e-02 4.37041484e-07]\n",
" [5.56184067e-05 7.14905187e-02 9.28453863e-01]\n",
" [1.95627597e-04 1.77447474e-01 8.22356898e-01]\n",
" [1.39827861e-04 1.43545432e-01 8.56314741e-01]\n",
" [1.27971563e-05 1.91614584e-02 9.80825744e-01]]\n",
"[1 2 2 0 0 0 2 2 2 2]\n",
"[2 2 2 0 0 0 2 2 2 2]\n",
"[0.51742882 0.59306176 0.93562034 0.95799249 0.94331457 0.98288416\n",
" 0.92845386 0.8223569 0.85631474 0.98082574]\n"
]
}
],
"source": [
"probs = lr_clf.predict_proba(X_test)\n",
"print(\"Class Probabilities (first 10):\")\n",
"print (probs[:10])\n",
"print(y_test[:10])\n",
"print(probs.argmax(axis = 1)[:10])\n",
"print(probs.max(axis = 1)[:10])"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"And the coeffients of the logistic regression model"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[-0.41392169 0.72653317 -2.19127669 -0.95006913]\n",
" [ 0.08012925 -0.38613281 -0.02275503 -0.69474957]\n",
" [ 0.33379244 -0.34040036 2.21403172 1.6448187 ]]\n"
]
}
],
"source": [
"print(lr_clf.coef_)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Neural Networks: Mutli-Layer Perceptron\n",
"\n",
"https://scikit-learn.org/stable/modules/neural_networks_supervised.html\n",
"https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"classifier score: 0.95\n",
"\n",
"accuracy: 0.95\n",
"\n",
"Confusion matrix\n",
"[[20 0 0]\n",
" [ 0 18 1]\n",
" [ 0 2 19]]\n",
"\n",
"Precision Score per class\n",
"[1. 0.9 0.95]\n",
"\n",
"Average Precision Score\n",
"0.9508333333333333\n",
"\n",
"Recall Score per class\n",
"[1. 0.94736842 0.9047619 ]\n",
"\n",
"Average Recall Score\n",
"0.95\n",
"\n",
"F1-score Score per class\n",
"[1. 0.92307692 0.92682927]\n",
"\n",
"Average F1 Score\n",
"0.9500312695434646\n"
]
}
],
"source": [
"from sklearn.neural_network import MLPClassifier\n",
"\n",
"mlp_clf = MLPClassifier(solver='lbfgs')\n",
"mlp_clf.fit(X_train, y_train)\n",
"print(\"classifier score:\",mlp_clf.score(X_test,y_test))\n",
"y_pred = mlp_clf.predict(X_test)\n",
"print(\"\\naccuracy:\",metrics.accuracy_score(y_test,y_pred))\n",
"print(\"\\nConfusion matrix\")\n",
"print(metrics.confusion_matrix(y_test,y_pred))\n",
"print(\"\\nPrecision Score per class\")\n",
"print(metrics.precision_score(y_test,y_pred,average=None))\n",
"print(\"\\nAverage Precision Score\")\n",
"print(metrics.precision_score(y_test,y_pred,average='weighted'))\n",
"print(\"\\nRecall Score per class\")\n",
"print(metrics.recall_score(y_test,y_pred,average=None))\n",
"print(\"\\nAverage Recall Score\")\n",
"print(metrics.recall_score(y_test,y_pred,average='weighted'))\n",
"print(\"\\nF1-score Score per class\")\n",
"print(metrics.f1_score(y_test,y_pred,average=None))\n",
"print(\"\\nAverage F1 Score\")\n",
"print(metrics.f1_score(y_test,y_pred,average='weighted'))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Linear Regression\n",
"\n",
"Linear Regression is implemented in the library sklearn.linear_model.LinearRegression: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html\n"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"from sklearn.linear_model import LinearRegression\n",
"X_reg = np.array([[1, 1], [1, 2], [2, 2], [2, 3]])\n",
"\n",
"# y = 1 * x_0 + 2 * x_1 + 3\n",
"y_reg = np.dot(X_reg, np.array([1, 2])) + 3\n",
"\n",
"reg = LinearRegression().fit(X_reg, y_reg)\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1. 2.]\n",
"3.000000000000001\n"
]
}
],
"source": [
"#Obtain the function coefficients\n",
"print(reg.coef_)\n",
"#and the intercept\n",
"print(reg.intercept_)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"The [$R^2$ score](https://en.wikipedia.org/wiki/Coefficient_of_determination) computes the \"explained variance\"\n",
"\n",
"$R^2 = 1-\\frac{\\sum_i (y_i -\\hat y_i)^2}{\\sum_i (y_i -\\bar y)^2}$\n",
"\n",
"where $\\hat y_i$ is the prediction for point $x_i$ and $\\bar y$ is the mean value of the target variable"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1.0\n"
]
}
],
"source": [
"print(reg.score(X_reg, y_reg))"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([16.])"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#Predict for a new point\n",
"reg.predict(np.array([[3, 5]]))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"A more complex example with the [diabetes dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_diabetes.html#sklearn.datasets.load_diabetes)"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Coefficients: \n",
" [ -8.80343059 -238.68845774 515.45209151 329.26528155 -878.18276219\n",
" 530.03363161 126.03912568 213.28475276 734.46021416 67.32526032]\n",
"Mean squared error: 2304.97\n",
"Coefficient of determination: 0.68\n",
"Predictions:\n",
"[149.75303117 199.7656287 248.11294766 182.95040528 98.34540804\n",
" 96.66271486 248.59757565 64.84343648 234.52373522 209.30957394\n",
" 179.26665684 85.95716856 70.54292903 197.93453267 100.34630781\n",
" 116.81521079 134.97372936 64.08572743 178.32873088 155.32247369]\n",
"True values:\n",
"[168. 221. 310. 283. 81. 94. 277. 72. 270. 268. 174. 96. 83. 222.\n",
" 69. 153. 202. 43. 124. 276.]\n"
]
}
],
"source": [
"diabetes_X, diabetes_y = sk_data.load_diabetes(return_X_y=True)\n",
"\n",
"# Shuffle the data\n",
"diabetes_X, diabetes_y = utils.shuffle(diabetes_X, diabetes_y, random_state=1)\n",
"\n",
"# Split the data into training/testing sets\n",
"diabetes_X_train = diabetes_X[:-20]\n",
"diabetes_X_test = diabetes_X[-20:]\n",
"\n",
"# Split the targets into training/testing sets\n",
"diabetes_y_train = diabetes_y[:-20]\n",
"diabetes_y_test = diabetes_y[-20:]\n",
"\n",
"# Create linear regression object\n",
"regr = linear_model.LinearRegression()\n",
"\n",
"# Train the model using the training sets\n",
"regr.fit(diabetes_X_train, diabetes_y_train)\n",
"\n",
"# Make predictions using the testing set\n",
"diabetes_y_pred = regr.predict(diabetes_X_test)\n",
"\n",
"# The coefficients\n",
"print('Coefficients: \\n', regr.coef_)\n",
"# The mean squared error\n",
"print('Mean squared error: %.2f'\n",
" % metrics.mean_squared_error(diabetes_y_test, diabetes_y_pred))\n",
"# The coefficient of determination: 1 is perfect prediction\n",
"# Computed over the *test* data\n",
"print('Coefficient of determination: %.2f'\n",
" % metrics.r2_score(diabetes_y_test, diabetes_y_pred))\n",
"print('Predictions:')\n",
"print(diabetes_y_pred)\n",
"print('True values:')\n",
"print(diabetes_y_test)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## More Evaluation\n",
"\n",
"http://scikit-learn.org/stable/model_selection.html#model-selection"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Computing Scores"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1. 0.9 0.95]\n",
"[1. 0.94736842 0.9047619 ]\n",
"[1. 0.92307692 0.92682927]\n",
"[20 19 21]\n"
]
}
],
"source": [
"p,r,f,s = metrics.precision_recall_fscore_support(y_test,y_pred)\n",
"print(p)\n",
"print(r)\n",
"print(f)\n",
"print(s)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" precision recall f1-score support\n",
"\n",
" 0 1.00 1.00 1.00 20\n",
" 1 0.90 0.95 0.92 19\n",
" 2 0.95 0.90 0.93 21\n",
"\n",
" accuracy 0.95 60\n",
" macro avg 0.95 0.95 0.95 60\n",
"weighted avg 0.95 0.95 0.95 60\n",
"\n"
]
}
],
"source": [
"report = metrics.classification_report(y_test,y_pred)\n",
"print(report)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"The [breast cancer dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html#sklearn.datasets.load_breast_cancer)"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"classifier score: 0.9565217391304348\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
"STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
"\n",
"Increase the number of iterations (max_iter) or scale the data as shown in:\n",
" https://scikit-learn.org/stable/modules/preprocessing.html\n",
"Please also refer to the documentation for alternative solver options:\n",
" https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
" n_iter_i = _check_optimize_result(\n"
]
}
],
"source": [
"cancer_data = sk_data.load_breast_cancer()\n",
"X_cancer,y_cancer = utils.shuffle(cancer_data.data, cancer_data.target, random_state=1)\n",
"X_cancer_train = X_cancer[:500]\n",
"y_cancer_train = y_cancer[:500]\n",
"X_cancer_test = X_cancer[500:]\n",
"y_cancer_test = y_cancer[500:]\n",
"lr_clf.fit(X_cancer_train, y_cancer_train)\n",
"print(\"classifier score:\",lr_clf.score(X_cancer_test,y_cancer_test))"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Class Probabilities (first 10):\n",
"[[5.60422369e-01 4.39577631e-01]\n",
" [7.03281401e-03 9.92967186e-01]\n",
" [9.99988304e-01 1.16956041e-05]\n",
" [9.99894278e-01 1.05722247e-04]\n",
" [2.39455667e-02 9.76054433e-01]\n",
" [9.99983718e-01 1.62819123e-05]\n",
" [4.47497687e-03 9.95525023e-01]\n",
" [1.62579095e-03 9.98374209e-01]\n",
" [1.78903213e-03 9.98210968e-01]\n",
" [6.40995488e-02 9.35900451e-01]]\n",
"[1. 0.97619048 0.97619048 0.97619048 0.95238095 0.92857143\n",
" 0.9047619 0.88095238 0.85714286 0.83333333 0.80952381 0.78571429\n",
" 0.76190476 0.76190476 0.73809524 0.71428571 0.69047619 0.66666667\n",
" 0.64285714 0.61904762 0.5952381 0.57142857 0.54761905 0.52380952\n",
" 0.5 0.47619048 0.45238095 0.42857143 0.4047619 0.38095238\n",
" 0.35714286 0.33333333 0.30952381 0.28571429 0.26190476 0.23809524\n",
" 0.21428571 0.19047619 0.16666667 0.14285714 0.11904762 0.0952381\n",
" 0.07142857 0.04761905 0.02380952 0. ]\n",
"[0.93333333 0.93181818 0.95348837 0.97619048 0.97560976 0.975\n",
" 0.97435897 0.97368421 0.97297297 0.97222222 0.97142857 0.97058824\n",
" 0.96969697 1. 1. 1. 1. 1.\n",
" 1. 1. 1. 1. 1. 1.\n",
" 1. 1. 1. 1. 1. 1.\n",
" 1. 1. 1. 1. 1. 1.\n",
" 1. 1. 1. 1. 1. 1.\n",
" 1. 1. 1. 1. ]\n",
"[0.43957763 0.46342134 0.64685966 0.77398617 0.83286525 0.84559913\n",
" 0.86486531 0.92093819 0.93082139 0.93590045 0.9369014 0.9461342\n",
" 0.94742559 0.95278082 0.96068092 0.97605443 0.97869647 0.98329802\n",
" 0.98857067 0.98917395 0.99072987 0.99166743 0.99240833 0.99296719\n",
" 0.99298497 0.99347899 0.99520593 0.99525018 0.99552502 0.9957985\n",
" 0.99620429 0.99738804 0.99745872 0.99804553 0.99821097 0.99837421\n",
" 0.99923191 0.99936541 0.99947042 0.99960289 0.99969407 0.99972281\n",
" 0.99978423 0.99978497 0.9998579 ]\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAGdCAYAAAAxCSikAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAu60lEQVR4nO3dfVSU553/8c8wDMxEgRhJiAoCWqMkNj6AQSEP6zbBEjWa02xI27g1v8QNW3Miod1WVm2i2YRf1p9sairEJxJN0mA3z01pIs1uEgwmU6h2azGQxBhQoR6ogkoFhPv3xyyTTgeNgwrX4Pt1zn3KXHPdM9/rOp7cn94P19gsy7IEAABgsJCBLgAAAOCrEFgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYLHegCzpfu7m4dOnRIERERstlsA10OAAA4C5Zl6dixYxo5cqRCQk5/HmXQBJZDhw4pLi5uoMsAAAB9UF9fr9jY2NO+P2gCS0REhCTPgCMjIwe4GgAAcDZaW1sVFxfnPY6fzqAJLD2XgSIjIwksAAAEma+6nYObbgEAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QIOLO+//77mzp2rkSNHymaz6bXXXvvKfd577z0lJyfL6XRqzJgxevrpp/36vPzyy7r66qsVHh6uq6++Wq+++mqgpQEAgEEq4MBy4sQJTZo0ST/72c/Oqv/nn3+uW2+9VTfccIN27dqlf/3Xf9WDDz6ol19+2dtn586dysrK0oIFC/T73/9eCxYs0J133qmPPvoo0PIAAMAgZLMsy+rzzjabXn31Vc2fP/+0fX784x/rjTfe0N69e71t2dnZ+v3vf6+dO3dKkrKystTa2qpf//rX3j7f/OY3NWzYML344otnVUtra6uioqLU0tLCbwkBABAkzvb4fcF//HDnzp3KyMjwaZs1a5Y2b96szs5OORwO7dy5Uw899JBfnyeffPK0n9ve3q729nbv69bW1vNat+dD66UXUs7/5wIAJNmkcbdL3yiUvuKH74ALHlgaGxsVExPj0xYTE6NTp06pqalJI0aMOG2fxsbG035ufn6+Vq5ceUFq/lK31Hb4An8HAFzEfv+0NOVBaXjSQFcCw13wwCL5/2R0z1Wov27vrc+Zfmo6Ly9Pubm53tetra2Ki4s7H+V+yRYqRSWe388EAEh/aZI6jnn+7jw+sLUgKFzwwHLllVf6nSk5fPiwQkNDNXz48DP2+duzLn8tPDxc4eHh57/gvxYxSrpv34X9DgC4GP3Xg9Kupwa6CgSRC74Oy4wZM1RWVubTtn37dqWkpMjhcJyxT1pa2oUuDwAABIGAz7AcP35cn376qff1559/rt27d+uyyy7T6NGjlZeXp4MHD2rr1q2SPE8E/exnP1Nubq4WLVqknTt3avPmzT5P/yxZskQ33nijnnjiCc2bN0+vv/66fvOb32jHjh3nYYgAACDYBXyGpbKyUlOmTNGUKVMkSbm5uZoyZYp+8pOfSJIaGhpUV1fn7Z+YmKjS0lK9++67mjx5sh599FGtXbtW3/rWt7x90tLSVFJSomeeeUbXXnutnn32WW3btk2pqannOj4AADAInNM6LCZhHRYACCJ/fQ/Ld93SldMGth4MmLM9fvNbQgAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADj9SmwFBYWKjExUU6nU8nJySovLz9j/3Xr1ikpKUkul0vjx4/X1q1bfd7v7OzUqlWrNHbsWDmdTk2aNElvvfVWX0oDAACDUMCBZdu2bcrJydGyZcu0a9cu3XDDDcrMzFRdXV2v/YuKipSXl6dHHnlEf/zjH7Vy5UotXrxYv/zlL719li9frvXr1+upp55SdXW1srOzdfvtt2vXrl19HxkAABg0bJZlWYHskJqaqqlTp6qoqMjblpSUpPnz5ys/P9+vf1pamtLT07V69WpvW05OjiorK7Vjxw5J0siRI7Vs2TItXrzY22f+/PkaOnSonn/++bOqq7W1VVFRUWppaVFkZGQgQwIA9Lf/elDa9ZTn7++6pSunDWw9GDBne/wO6AxLR0eHqqqqlJGR4dOekZGhioqKXvdpb2+X0+n0aXO5XHK73ers7Dxjn55Ac7rPbW1t9dkAAMDgFFBgaWpqUldXl2JiYnzaY2Ji1NjY2Os+s2bN0qZNm1RVVSXLslRZWani4mJ1dnaqqanJ26egoECffPKJuru7VVZWptdff10NDQ2nrSU/P19RUVHeLS4uLpChAACAINKnm25tNpvPa8uy/Np6rFixQpmZmZo+fbocDofmzZunhQsXSpLsdrsk6ac//anGjRunCRMmKCwsTA888IDuuece7/u9ycvLU0tLi3err6/vy1AAAEAQCCiwREdHy263+51NOXz4sN9Zlx4ul0vFxcVqa2vT/v37VVdXp4SEBEVERCg6OlqSdPnll+u1117TiRMn9MUXX+jjjz/W0KFDlZiYeNpawsPDFRkZ6bMBAIDBKaDAEhYWpuTkZJWVlfm0l5WVKS0t7Yz7OhwOxcbGym63q6SkRHPmzFFIiO/XO51OjRo1SqdOndLLL7+sefPmBVIeAAAYpEID3SE3N1cLFixQSkqKZsyYoQ0bNqiurk7Z2dmSPJdqDh486F1rpba2Vm63W6mpqTpy5IgKCgq0Z88ebdmyxfuZH330kQ4ePKjJkyfr4MGDeuSRR9Td3a0f/ehH52mYAAAgmAUcWLKystTc3KxVq1apoaFBEydOVGlpqeLj4yVJDQ0NPmuydHV1ac2aNaqpqZHD4dDMmTNVUVGhhIQEb5+TJ09q+fLl2rdvn4YOHapbb71Vzz33nC699NJzHiAAAAh+Aa/DYirWYQGAIMI6LPhfF2QdFgAAgIFAYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgvD4FlsLCQiUmJsrpdCo5OVnl5eVn7L9u3TolJSXJ5XJp/Pjx2rp1q1+fJ598UuPHj5fL5VJcXJweeughnTx5si/lAQCAQSY00B22bdumnJwcFRYWKj09XevXr1dmZqaqq6s1evRov/5FRUXKy8vTxo0bNW3aNLndbi1atEjDhg3T3LlzJUkvvPCCli5dquLiYqWlpam2tlYLFy6UJP3Hf/zHuY0QAAAEPZtlWVYgO6Smpmrq1KkqKirytiUlJWn+/PnKz8/365+Wlqb09HStXr3a25aTk6PKykrt2LFDkvTAAw9o7969euedd7x9fvCDH8jtdn/l2Zsera2tioqKUktLiyIjIwMZEgCgv/3Xg9Kupzx/f9ctXTltYOvBgDnb43dAl4Q6OjpUVVWljIwMn/aMjAxVVFT0uk97e7ucTqdPm8vlktvtVmdnpyTp+uuvV1VVldxutyRp3759Ki0t1ezZs09bS3t7u1pbW302AAAwOAUUWJqamtTV1aWYmBif9piYGDU2Nva6z6xZs7Rp0yZVVVXJsixVVlaquLhYnZ2dampqkiTdddddevTRR3X99dfL4XBo7NixmjlzppYuXXraWvLz8xUVFeXd4uLiAhkKAAAIIn266dZms/m8tizLr63HihUrlJmZqenTp8vhcGjevHne+1Psdrsk6d1339Vjjz2mwsJC/e53v9Mrr7yiN998U48++uhpa8jLy1NLS4t3q6+v78tQAABAEAgosERHR8tut/udTTl8+LDfWZceLpdLxcXFamtr0/79+1VXV6eEhARFREQoOjpakifULFiwQPfdd5++/vWv6/bbb9fjjz+u/Px8dXd39/q54eHhioyM9NkAAMDgFFBgCQsLU3JyssrKynzay8rKlJaWdsZ9HQ6HYmNjZbfbVVJSojlz5igkxPP1bW1t3r972O12WZalAO8JBgAAg1DAjzXn5uZqwYIFSklJ0YwZM7RhwwbV1dUpOztbkudSzcGDB71rrdTW1srtdis1NVVHjhxRQUGB9uzZoy1btng/c+7cuSooKNCUKVOUmpqqTz/9VCtWrNBtt93mvWwEAAAuXgEHlqysLDU3N2vVqlVqaGjQxIkTVVpaqvj4eElSQ0OD6urqvP27urq0Zs0a1dTUyOFwaObMmaqoqFBCQoK3z/Lly2Wz2bR8+XIdPHhQl19+uebOnavHHnvs3EcIAACCXsDrsJiKdVgAIIiwDgv+1wVZhwUAAGAgEFgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEC/i0hAAAQ5FrrpbrfSC37pK/fJ0XGD3RFX4nAAgDAYHfyqFT/riekfFEmHan98r2Gj6Q7tg9QYWePwAIAwGBzql1q+PDLgNL4W8nq7r3v8UP9W1sfEVgAAAh2liX9uUba/5a0/23pwPvSqbbe+4aESiOmS4d2SlZX/9Z5DggsAAAEo/ZWqe4dT0DZ/5bU+sXp+w6/Roq/WRp9sxR3kxQWIa2NkDqP91+954jAAgBAMLC6pcO7vzyLcqhC6j7Ve9+hIz3hJP5mafQ3PK+DHIEFAABTnTzqCSef/8rzv22He+9nD5NG3SglflNKmOU5o2Kz9WupFxqBBQAAU1iW9Oe90r5fSfvelA5+cPr7TIZd5QknCd/0XOZxDOnfWvsZgQUAgIF06qTnkeOekNK6v/d+jqGeyzsJszzbpWP6sciBR2ABAKC/nWiUPvulJ6B88ZvTP9EzbJw0Zo6UOFuKvcFz6eciRWABAKA/NH8sffa69OnrnjVSZPn3CXFIsTd6QsqY2Z7AAkkEFgAALgyrWzr04Zch5UhN7/0uiZESb/UElPhbpPDI/q0zSBBYAAA4X06d9Fzi+ex1zyWftj/13m/4NdLX5klj50lXpkg2fov4qxBYAAA4F51t0ue/lj552RNSeluMzRYijbreE1C+Nk+6dGz/1xnkCCwAAASq45jnqZ5PXpb2lfZ+02yoy/M0z9h5nntSLonu/zoHEQILAABn4+RRad8vpdqXPIu4dbX793EO+9+zKLd7Vpl1XNLvZQ5WBBYAAE6nvcVzw2zNNs+vHnd3+vdxXS6Nu10a9y0pbqZkd/R/nRcBAgsAAH+ts82zPsrHJdLnpb2fSRlypSegjPuWZ32UEA6nFxozDADAqXbpi+2ekPLZ61LnCf8+Q2Olq+7wbCNn8GRPPyOwAAAuTt2nPEvif/yi9MkrUvtR/z6XxEjj75QmfFsakUpIGUAEFgDAxcOypMO7pOrnPEGlt3VSnMOkcXdIE+6SYm+SQuz9Xyf8EFgAAIPfsQPS3hek6q1Sc7X/+46h0tfme0JK/C0X9W/2mIrAAgAYnDqOSbUvS3ufk+r+W36/3WMP86yPMuE7nqXxHa4BKRNnh8ACABg8uk95lsavfk769FXp1F/8+4y6Xrp6gXTVP3gu/yAoEFgAAMHvyCfSnmLpj1ukEw3+71/6NU9ISbpbunRM/9eHc0ZgAQAEp84Tnks+ezZLB973f995mTQ+S7r6H//3CR9b/9eI84bAAgAIHpYlNf7WE1I+ftFzn8pfCwmVEmdL1yyUxtzKzbODCIEFAGC+tiZp7/OeoNK0x//9YeOlr9/rOZsyJKb/68MF16cVcAoLC5WYmCin06nk5GSVl5efsf+6deuUlJQkl8ul8ePHa+vWrT7v/93f/Z1sNpvfNnv27L6UBwAYDCxLqn9PevPb0vqR0rsP+YYVxxBp4v+R7vpAumevNO1fCCuDWMBnWLZt26acnBwVFhYqPT1d69evV2ZmpqqrqzV69Gi//kVFRcrLy9PGjRs1bdo0ud1uLVq0SMOGDdPcuXMlSa+88oo6Ojq8+zQ3N2vSpEn6h3/4h3MYGgAgKJ086nnK53+e7n3NlJFpnqAy/k4pLKLfy8PACDiwFBQU6N5779V9990nSXryySf19ttvq6ioSPn5+X79n3vuOd1///3KysqSJI0ZM0YffvihnnjiCW9gueyyy3z2KSkp0SWXXEJgAYCLyZ+qpN1FnntTTrX5vueK9tyXMvH/SMOTBqQ8DKyAAktHR4eqqqq0dOlSn/aMjAxVVFT0uk97e7ucTqdPm8vlktvtVmdnpxwO/5/h3rx5s+666y4NGTLktLW0t7ervf3LX9BsbW0NZCgAAFPU/EJ6Z7HnZtq/Nep6adI/e34VOTS8/2uDMQK6h6WpqUldXV2KifG9RhgTE6PGxsZe95k1a5Y2bdqkqqoqWZalyspKFRcXq7OzU01NTX793W639uzZ4z2Dczr5+fmKiorybnFxcYEMBQBgisr/5xtWwiKkyYul7/1BuqtcSvoOYQV9u+nW9jfPsluW5dfWY8WKFcrMzNT06dPlcDg0b948LVy4UJJkt/v/oNTmzZs1ceJEXXfddWesIS8vTy0tLd6tvr6+L0MBAAyIXo4ZV0yRbtkg3X9I+sbPpOiJ/V8WjBVQYImOjpbdbvc7m3L48GG/sy49XC6XiouL1dbWpv3796uurk4JCQmKiIhQdHS0T9+2tjaVlJR85dkVSQoPD1dkZKTPBgAIEgmzJJtdCnVK13xP+s6H0t1V0rWLpLChA10dDBTQPSxhYWFKTk5WWVmZbr/9dm97WVmZ5s2bd8Z9HQ6HYmNjJXluqp0zZ45CQnzz0i9+8Qu1t7fr7rvvDqQsAECwGXOrdP8Bz6PJPOmDsxDwU0K5ublasGCBUlJSNGPGDG3YsEF1dXXKzs6W5LlUc/DgQe9aK7W1tXK73UpNTdWRI0dUUFCgPXv2aMuWLX6fvXnzZs2fP1/Dhw8/x2EBAIw35MqBrgBBJODAkpWVpebmZq1atUoNDQ2aOHGiSktLFR8fL0lqaGhQXV2dt39XV5fWrFmjmpoaORwOzZw5UxUVFUpISPD53NraWu3YsUPbt28/txEBAIBBx2ZZljXQRZwPra2tioqKUktLC/ezAADwVdZGSJ3HpeHXSAt7+bmDfnK2x+8+PSUEAADQnwgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIzXp8BSWFioxMREOZ1OJScnq7y8/Iz9161bp6SkJLlcLo0fP15bt27163P06FEtXrxYI0aMkNPpVFJSkkpLS/tSHgAAGGRCA91h27ZtysnJUWFhodLT07V+/XplZmaqurpao0eP9utfVFSkvLw8bdy4UdOmTZPb7daiRYs0bNgwzZ07V5LU0dGhW265RVdccYVeeuklxcbGqr6+XhEREec+QgAAEPRslmVZgeyQmpqqqVOnqqioyNuWlJSk+fPnKz8/369/Wlqa0tPTtXr1am9bTk6OKisrtWPHDknS008/rdWrV+vjjz+Ww+Ho00BaW1sVFRWllpYWRUZG9ukzAAC4aKyNkDqPS8OvkRbuGbAyzvb4HdAloY6ODlVVVSkjI8OnPSMjQxUVFb3u097eLqfT6dPmcrnkdrvV2dkpSXrjjTc0Y8YMLV68WDExMZo4caIef/xxdXV1BVIeAAAYpAIKLE1NTerq6lJMTIxPe0xMjBobG3vdZ9asWdq0aZOqqqpkWZYqKytVXFyszs5ONTU1SZL27dunl156SV1dXSotLdXy5cu1Zs0aPfbYY6etpb29Xa2trT4bAAAYnPp0063NZvN5bVmWX1uPFStWKDMzU9OnT5fD4dC8efO0cOFCSZLdbpckdXd364orrtCGDRuUnJysu+66S8uWLfO57PS38vPzFRUV5d3i4uL6MhQAABAEAgos0dHRstvtfmdTDh8+7HfWpYfL5VJxcbHa2tq0f/9+1dXVKSEhQREREYqOjpYkjRgxQldddZU3wEie+2IaGxvV0dHR6+fm5eWppaXFu9XX1wcyFAAAEEQCCixhYWFKTk5WWVmZT3tZWZnS0tLOuK/D4VBsbKzsdrtKSko0Z84chYR4vj49PV2ffvqpuru7vf1ra2s1YsQIhYWF9fp54eHhioyM9NkAAMDgFPAlodzcXG3atEnFxcXau3evHnroIdXV1Sk7O1uS58zHP/7jP3r719bW6vnnn9cnn3wit9utu+66S3v27NHjjz/u7fPP//zPam5u1pIlS1RbW6tf/epXevzxx7V48eLzMEQAABDsAl6HJSsrS83NzVq1apUaGho0ceJElZaWKj4+XpLU0NCguro6b/+uri6tWbNGNTU1cjgcmjlzpioqKpSQkODtExcXp+3bt+uhhx7Stddeq1GjRmnJkiX68Y9/fO4jBAAAQS/gdVhMxTosAAAEYDCvwwIAADAQCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxutTYCksLFRiYqKcTqeSk5NVXl5+xv7r1q1TUlKSXC6Xxo8fr61bt/q8/+yzz8pms/ltJ0+e7Et5AABgkAkNdIdt27YpJydHhYWFSk9P1/r165WZmanq6mqNHj3ar39RUZHy8vK0ceNGTZs2TW63W4sWLdKwYcM0d+5cb7/IyEjV1NT47Ot0OvswJAAAMNgEHFgKCgp077336r777pMkPfnkk3r77bdVVFSk/Px8v/7PPfec7r//fmVlZUmSxowZow8//FBPPPGET2Cx2Wy68sor+zoOAAAwiAV0Saijo0NVVVXKyMjwac/IyFBFRUWv+7S3t/udKXG5XHK73ers7PS2HT9+XPHx8YqNjdWcOXO0a9euM9bS3t6u1tZWnw0AAAxOAQWWpqYmdXV1KSYmxqc9JiZGjY2Nve4za9Ysbdq0SVVVVbIsS5WVlSouLlZnZ6eampokSRMmTNCzzz6rN954Qy+++KKcTqfS09P1ySefnLaW/Px8RUVFebe4uLhAhgIAAIJIn266tdlsPq8ty/Jr67FixQplZmZq+vTpcjgcmjdvnhYuXChJstvtkqTp06fr7rvv1qRJk3TDDTfoF7/4ha666io99dRTp60hLy9PLS0t3q2+vr4vQwEAAEEgoMASHR0tu93udzbl8OHDfmdderhcLhUXF6utrU379+9XXV2dEhISFBERoejo6N6LCgnRtGnTzniGJTw8XJGRkT4bAAAYnAIKLGFhYUpOTlZZWZlPe1lZmdLS0s64r8PhUGxsrOx2u0pKSjRnzhyFhPT+9ZZlaffu3RoxYkQg5QEAgEEq4KeEcnNztWDBAqWkpGjGjBnasGGD6urqlJ2dLclzqebgwYPetVZqa2vldruVmpqqI0eOqKCgQHv27NGWLVu8n7ly5UpNnz5d48aNU2trq9auXavdu3dr3bp152mYAAAgmAUcWLKystTc3KxVq1apoaFBEydOVGlpqeLj4yVJDQ0Nqqur8/bv6urSmjVrVFNTI4fDoZkzZ6qiokIJCQnePkePHtU//dM/qbGxUVFRUZoyZYref/99XXfddec+QgAAEPRslmVZA13E+dDa2qqoqCi1tLRwPwsAAF9lbYTUeVwafo20cM+AlXG2x29+SwgAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeH0KLIWFhUpMTJTT6VRycrLKy8vP2H/dunVKSkqSy+XS+PHjtXXr1tP2LSkpkc1m0/z58/tSGgAAGIRCA91h27ZtysnJUWFhodLT07V+/XplZmaqurpao0eP9utfVFSkvLw8bdy4UdOmTZPb7daiRYs0bNgwzZ0716fvF198oR/+8Ie64YYb+j4iAAAw6Ngsy7IC2SE1NVVTp05VUVGRty0pKUnz589Xfn6+X/+0tDSlp6dr9erV3racnBxVVlZqx44d3rauri7ddNNNuueee1ReXq6jR4/qtddeO+u6WltbFRUVpZaWFkVGRgYyJAAALj5rI6TO49Lwa6SFewasjLM9fgd0Saijo0NVVVXKyMjwac/IyFBFRUWv+7S3t8vpdPq0uVwuud1udXZ2ettWrVqlyy+/XPfee+9Z1dLe3q7W1lafDQAADE4BBZampiZ1dXUpJibGpz0mJkaNjY297jNr1ixt2rRJVVVVsixLlZWVKi4uVmdnp5qamiRJH3zwgTZv3qyNGzeedS35+fmKiorybnFxcYEMBQAABJE+3XRrs9l8XluW5dfWY8WKFcrMzNT06dPlcDg0b948LVy4UJJkt9t17Ngx3X333dq4caOio6PPuoa8vDy1tLR4t/r6+r4MBQAABIGAbrqNjo6W3W73O5ty+PBhv7MuPVwul4qLi7V+/Xr96U9/0ogRI7RhwwZFREQoOjpa//M//6P9+/f73IDb3d3tKS40VDU1NRo7dqzf54aHhys8PDyQ8gEAQJAK6AxLWFiYkpOTVVZW5tNeVlamtLS0M+7rcDgUGxsru92ukpISzZkzRyEhIZowYYL+8Ic/aPfu3d7ttttu08yZM7V7924u9QAAgMAfa87NzdWCBQuUkpKiGTNmaMOGDaqrq1N2drYkz6WagwcPetdaqa2tldvtVmpqqo4cOaKCggLt2bNHW7ZskSQ5nU5NnDjR5zsuvfRSSfJrBwAAF6eAA0tWVpaam5u1atUqNTQ0aOLEiSotLVV8fLwkqaGhQXV1dd7+XV1dWrNmjWpqauRwODRz5kxVVFQoISHhvA0CAAAMbgGvw2Iq1mEBACAAg3kdFgAAgIFAYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAM7sYIX0yzulro4BKyF0wL4ZAACYratD2rlScv9fyeqWLhsvpT86IKUQWAAAgL/maqn0bunwri/b6t+Tuk9JIf0fHwgsAADgS1a3tOtnUvmPpVMnPW0hoVLaSmnajwYkrEgEFgAA0OPYAemte6S633zZdlmSdOvzUszUgatLBBYAACBJe1+U3vm+1H70y7apS6Tr8yWHa8DK6kFgAQDgYtZ5Qnrz21JNyZdtQ0dJ33xWir95wMr6WwQWAAAuZq37PVuPCd+WvrFOcg4bqIp6RWABAABS+KXSNwqlpG8PdCW9IrAAAHAxCouQOo97/h79Dc8loIjYAS3pTAgsAABcjG58Qvr9055LQJO/L9nMXvyewAIAwMXo6gWeLUiYHacAAABEYAEAAEGgT4GlsLBQiYmJcjqdSk5OVnl5+Rn7r1u3TklJSXK5XBo/fry2bt3q8/4rr7yilJQUXXrppRoyZIgmT56s5557ri+lAQCAQSjge1i2bdumnJwcFRYWKj09XevXr1dmZqaqq6s1evRov/5FRUXKy8vTxo0bNW3aNLndbi1atEjDhg3T3LlzJUmXXXaZli1bpgkTJigsLExvvvmm7rnnHl1xxRWaNWvWuY8SAAAENZtlWVYgO6Smpmrq1KkqKirytiUlJWn+/PnKz8/365+Wlqb09HStXr3a25aTk6PKykrt2LHjtN8zdepUzZ49W48+enY/Y93a2qqoqCi1tLQoMjIygBEBAICBcrbH74AuCXV0dKiqqkoZGRk+7RkZGaqoqOh1n/b2djmdTp82l8slt9utzs5Ov/6WZemdd95RTU2NbrzxxtPW0t7ertbWVp8NAAAMTgEFlqamJnV1dSkmJsanPSYmRo2Njb3uM2vWLG3atElVVVWyLEuVlZUqLi5WZ2enmpqavP1aWlo0dOhQhYWFafbs2Xrqqad0yy23nLaW/Px8RUVFebe4uLhAhgIAAIJIn266tdlsPq8ty/Jr67FixQplZmZq+vTpcjgcmjdvnhYuXChJstvt3n4RERHavXu3fvvb3+qxxx5Tbm6u3n333dPWkJeXp5aWFu9WX1/fl6EAAIAgEFBgiY6Olt1u9zubcvjwYb+zLj1cLpeKi4vV1tam/fv3q66uTgkJCYqIiFB0dPSXhYSE6Gtf+5omT56sH/zgB7rjjjt6vSemR3h4uCIjI302AAAwOAUUWMLCwpScnKyysjKf9rKyMqWlpZ1xX4fDodjYWNntdpWUlGjOnDkKCTn911uWpfb29kDKAwAAg1TAjzXn5uZqwYIFSklJ0YwZM7RhwwbV1dUpOztbkudSzcGDB71rrdTW1srtdis1NVVHjhxRQUGB9uzZoy1btng/Mz8/XykpKRo7dqw6OjpUWlqqrVu3+jyJBAAALl4BB5asrCw1Nzdr1apVamho0MSJE1VaWqr4+HhJUkNDg+rq6rz9u7q6tGbNGtXU1MjhcGjmzJmqqKhQQkKCt8+JEyf0/e9/XwcOHJDL5dKECRP0/PPPKysr69xHCAAAgl7A67CYinVYAAAIPmd7/B40v9bck7tYjwUAgODRc9z+qvMngyawHDt2TJJYjwUAgCB07NgxRUVFnfb9QXNJqLu7W4cOHVJERMRp14Tpi9bWVsXFxam+vp5LTRcQ89x/mOv+wTz3D+a5f1zIebYsS8eOHdPIkSPP+PTwoDnDEhISotjY2Av2+az10j+Y5/7DXPcP5rl/MM/940LN85nOrPTo00q3AAAA/YnAAgAAjEdg+Qrh4eF6+OGHFR4ePtClDGrMc/9hrvsH89w/mOf+YcI8D5qbbgEAwODFGRYAAGA8AgsAADAegQUAABiPwAIAAIxHYJFUWFioxMREOZ1OJScnq7y8/Iz933vvPSUnJ8vpdGrMmDF6+umn+6nS4BbIPL/yyiu65ZZbdPnllysyMlIzZszQ22+/3Y/VBq9A/z33+OCDDxQaGqrJkydf2AIHkUDnur29XcuWLVN8fLzCw8M1duxYFRcX91O1wSvQeX7hhRc0adIkXXLJJRoxYoTuueceNTc391O1wen999/X3LlzNXLkSNlsNr322mtfuU+/Hwuti1xJSYnlcDisjRs3WtXV1daSJUusIUOGWF988UWv/fft22ddcskl1pIlS6zq6mpr48aNlsPhsF566aV+rjy4BDrPS5YssZ544gnL7XZbtbW1Vl5enuVwOKzf/e53/Vx5cAl0nnscPXrUGjNmjJWRkWFNmjSpf4oNcn2Z69tuu81KTU21ysrKrM8//9z66KOPrA8++KAfqw4+gc5zeXm5FRISYv30pz+19u3bZ5WXl1vXXHONNX/+/H6uPLiUlpZay5Yts15++WVLkvXqq6+esf9AHAsv+sBy3XXXWdnZ2T5tEyZMsJYuXdpr/x/96EfWhAkTfNruv/9+a/r06ResxsEg0HnuzdVXX22tXLnyfJc2qPR1nrOysqzly5dbDz/8MIHlLAU617/+9a+tqKgoq7m5uT/KGzQCnefVq1dbY8aM8Wlbu3atFRsbe8FqHGzOJrAMxLHwor4k1NHRoaqqKmVkZPi0Z2RkqKKiotd9du7c6dd/1qxZqqysVGdn5wWrNZj1ZZ7/Vnd3t44dO6bLLrvsQpQ4KPR1np955hl99tlnevjhhy90iYNGX+b6jTfeUEpKiv793/9do0aN0lVXXaUf/vCH+stf/tIfJQelvsxzWlqaDhw4oNLSUlmWpT/96U966aWXNHv27P4o+aIxEMfCQfPjh33R1NSkrq4uxcTE+LTHxMSosbGx130aGxt77X/q1Ck1NTVpxIgRF6zeYNWXef5ba9as0YkTJ3TnnXdeiBIHhb7M8yeffKKlS5eqvLxcoaEX9X8OAtKXud63b5927Nghp9OpV199VU1NTfr+97+vP//5z9zHchp9mee0tDS98MILysrK0smTJ3Xq1Cnddttteuqpp/qj5IvGQBwLL+ozLD1sNpvPa8uy/Nq+qn9v7fAV6Dz3ePHFF/XII49o27ZtuuKKKy5UeYPG2c5zV1eXvvOd72jlypW66qqr+qu8QSWQf9Pd3d2y2Wx64YUXdN111+nWW29VQUGBnn32Wc6yfIVA5rm6uloPPvigfvKTn6iqqkpvvfWWPv/8c2VnZ/dHqReV/j4WXtT/lyo6Olp2u90vqR8+fNgvOfa48sore+0fGhqq4cOHX7Bag1lf5rnHtm3bdO+99+o///M/dfPNN1/IMoNeoPN87NgxVVZWateuXXrggQckeQ6qlmUpNDRU27dv19///d/3S+3Bpi//pkeMGKFRo0YpKirK25aUlCTLsnTgwAGNGzfugtYcjPoyz/n5+UpPT9e//Mu/SJKuvfZaDRkyRDfccIP+7d/+jbPg58lAHAsv6jMsYWFhSk5OVllZmU97WVmZ0tLSet1nxowZfv23b9+ulJQUORyOC1ZrMOvLPEueMysLFy7Uz3/+c64/n4VA5zkyMlJ/+MMftHv3bu+WnZ2t8ePHa/fu3UpNTe2v0oNOX/5Np6en69ChQzp+/Li3rba2ViEhIYqNjb2g9QarvsxzW1ubQkJ8D212u13Sl2cAcO4G5Fh4wW7nDRI9j8xt3rzZqq6utnJycqwhQ4ZY+/fvtyzLspYuXWotWLDA27/nUa6HHnrIqq6utjZv3sxjzWch0Hn++c9/boWGhlrr1q2zGhoavNvRo0cHaghBIdB5/ls8JXT2Ap3rY8eOWbGxsdYdd9xh/fGPf7Tee+89a9y4cdZ99903UEMICoHO8zPPPGOFhoZahYWF1meffWbt2LHDSklJsa677rqBGkJQOHbsmLVr1y5r165dliSroKDA2rVrl/fxcROOhRd9YLEsy1q3bp0VHx9vhYWFWVOnTrXee+8973vf+973rJtuusmn/7vvvmtNmTLFCgsLsxISEqyioqJ+rjg4BTLPN910kyXJb/ve977X/4UHmUD/Pf81AktgAp3rvXv3WjfffLPlcrms2NhYKzc312pra+vnqoNPoPO8du1a6+qrr7ZcLpc1YsQI67vf/a514MCBfq46uPz3f//3Gf+ba8Kx0GZZnCMDAABmu6jvYQEAAMGBwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4/1/+C1243BpDGgAAAAASUVORK5CYII=\n",
"text/plain": [
"