Introduction to Classification

The goal of this tutorial is to introduce you to the scikit libraries for classification. We will also cover the topic of feature normalization, and evaluation.

In [51]:
import numpy as np
import scipy.sparse as sp_sparse

import matplotlib.pyplot as plt

import sklearn as sk
import sklearn.datasets as sk_data
import sklearn.metrics as metrics
from sklearn import preprocessing

import seaborn as sns

%matplotlib inline

Feature normalization

Python provides some functionality for normalizing and standardizing the data. Be careful though, some operations work only with dense data.

http://scikit-learn.org/stable/modules/preprocessing.html#preprocessing

Normalize by removing the mean and dividing by the standard deviation. This is done per feature, that is, per column of the dataset.

In [52]:
X = np.array([[ 1., -1.,  2.],
              [ 2.,  0.,  1.],
              [ 0.,  1., -1.]])
print(X.mean(axis = 0))
print(X.std(axis = 0))
X_scaled = preprocessing.scale(X)
print(X_scaled)
print(X_scaled.mean(axis=0))
print(X_scaled.var(axis = 0))
[ 1.          0.          0.66666667]
[ 0.81649658  0.81649658  1.24721913]
[[ 0.         -1.22474487  1.06904497]
 [ 1.22474487  0.          0.26726124]
 [-1.22474487  1.22474487 -1.33630621]]
[  0.00000000e+00   0.00000000e+00   1.48029737e-16]
[ 1.  1.  1.]

The same can be done with the StandardScaler.

The function fit() computes the parameters for scaling, and transform() applies the scaling

In [53]:
std_scaler = preprocessing.StandardScaler().fit(X)
print(std_scaler.mean_)
print(std_scaler.scale_)
X_std = std_scaler.transform(X)
print(X_std)
[ 1.          0.          0.66666667]
[ 0.81649658  0.81649658  1.24721913]
[[ 0.         -1.22474487  1.06904497]
 [ 1.22474487  0.          0.26726124]
 [-1.22474487  1.22474487 -1.33630621]]

The advantage is the we can now apply the transform to new data.

For example, we compute the parameters for the training data and we apply the scaling to the test data.

In [54]:
y = np.array([[2.,3.,1.],
              [1.,2.,1.]])
print(std_scaler.transform(y))
[[ 1.22474487  3.67423461  0.26726124]
 [ 0.          2.44948974  0.26726124]]

The MinMaxScaler subbtracts from each column the minimum and then divides by the max-min.

In [56]:
min_max_scaler = preprocessing.MinMaxScaler()
X_minmax = min_max_scaler.fit_transform(X)
print(X_minmax)
print(min_max_scaler.transform(y))
[[ 0.5         0.          1.        ]
 [ 1.          0.5         0.66666667]
 [ 0.          1.          0.        ]]
[[ 1.          2.          0.66666667]
 [ 0.5         1.5         0.66666667]]

The MaxAbsScaler divides with the maximum absolute value.

The MaxAbsScaler can work with sparse data, since it does not destroy the data sparseness. For the other datasets, removing the mean (or min) can destroy the sparseness of the data.

Sometimes we may choose to normalize only the non-zero values. This should be done manually.

In [57]:
# works with sparse data
max_abs_scaler = preprocessing.MaxAbsScaler()
X_maxabs = max_abs_scaler.fit_transform(X)
X_maxabs
Out[57]:
array([[ 0.5, -1. ,  1. ],
       [ 1. ,  0. ,  0.5],
       [ 0. ,  1. , -0.5]])

The normalize function normalizes the rows so that they become unit vectors in some norm that we specify. It can be applied to sparse matrices without destroying the sparsity.

In [59]:
#works with sparse data

X_normalized = preprocessing.normalize(X, norm='l2')

X_normalized                                      
Out[59]:
array([[ 0.25      , -0.25      ,  0.5       ],
       [ 0.66666667,  0.        ,  0.33333333],
       [ 0.        ,  0.5       , -0.5       ]])

The OneHotEncoder can be used for categorical data to transform them into binary, where for each attribute value we have 0 or 1 depending on whether this value appears in the feature vector. It works with numerical categorical values.

In [60]:
X = [[0,1,2],
     [1,2,3],
     [0,1,4]]
enc = preprocessing.OneHotEncoder()
enc.fit(X)
enc.transform([[0,2,4],[1,1,2]]).toarray()
Out[60]:
array([[ 1.,  0.,  0.,  1.,  0.,  0.,  1.],
       [ 0.,  1.,  1.,  0.,  1.,  0.,  0.]])

We can also apply it selectively to some columns of the data

In [81]:
#works with sparse data

X = [[0, 10, 45100],
     [1, 20, 45221],
     [0, 20, 45212]]
enc = preprocessing.OneHotEncoder(categorical_features=[2]) #only the third column is categorical
enc.fit(X)
enc.transform([[5,13,45212],[4,12,45221]]).toarray()
Out[81]:
array([[  0.,   1.,   0.,   5.,  13.],
       [  0.,   0.,   1.,   4.,  12.]])

Feature Selection

Feature selection is about finding the best features for your classifier. This may be important if you do not have enough training data. The idea is to find metrics that either characterize the features by themselves, or with respect to the class we want to predict, or with respect to other features.

http://scikit-learn.org/stable/modules/feature_selection.html

The VarianceThreshold selection drops features whose variance is below some threshold. If we have binary features we can estimate the treshold exactly so as to guarantee a specific ratio of 0's and 1's

In [70]:
from sklearn.feature_selection import VarianceThreshold
X = [[0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 1, 1], [0, 1, 0], [0, 1, 1]]
print(np.array(X))
sel = VarianceThreshold(threshold=(.8 * (1 - .8)))
sel.fit_transform(X)
[[0 0 1]
 [0 1 0]
 [1 0 0]
 [0 1 1]
 [0 1 0]
 [0 1 1]]
Out[70]:
array([[0, 1],
       [1, 0],
       [0, 0],
       [1, 1],
       [1, 0],
       [1, 1]])

A more sophisticated feature selection technique uses the chi-square test to determine if a feature and the class label are independent.

https://en.wikipedia.org/wiki/Chi-squared_test

In this case the feature will have high score and low p-value. The features with the lowest values are rejected.

The chi-square test is usually applied on categorical data.

In [71]:
from sklearn.datasets import load_iris
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
iris = load_iris()
X, y = iris.data, iris.target
print(X.shape)
print(X[1:10,:],y[1:10])
sel = SelectKBest(chi2, k=2)
X_new = sel.fit_transform(X, y)
print(X_new[1:10])
print(sel.scores_)
c,p = sk.feature_selection.chi2(X, y)
print(c)
print(p)
(150, 4)
[[ 4.9  3.   1.4  0.2]
 [ 4.7  3.2  1.3  0.2]
 [ 4.6  3.1  1.5  0.2]
 [ 5.   3.6  1.4  0.2]
 [ 5.4  3.9  1.7  0.4]
 [ 4.6  3.4  1.4  0.3]
 [ 5.   3.4  1.5  0.2]
 [ 4.4  2.9  1.4  0.2]
 [ 4.9  3.1  1.5  0.1]] [0 0 0 0 0 0 0 0 0]
[[ 1.4  0.2]
 [ 1.3  0.2]
 [ 1.5  0.2]
 [ 1.4  0.2]
 [ 1.7  0.4]
 [ 1.4  0.3]
 [ 1.5  0.2]
 [ 1.4  0.2]
 [ 1.5  0.1]]
[  10.81782088    3.59449902  116.16984746   67.24482759]
[  10.81782088    3.59449902  116.16984746   67.24482759]
[  4.47651499e-03   1.65754167e-01   5.94344354e-26   2.50017968e-15]

Classification models

http://scikit-learn.org/stable/supervised_learning.html#supervised-learning

Python has classes and objects that implement the different classification techniques that we described in class.

Regardless of the choice of classifier, each classifier has two methods:

The method fit() takes the training data and labels and trains the model

The method predict() takes as input the test data and applies the model.

Preparing the data

Here we load the data, randomly shuffle it, and select a part of the data for training and a part for testing

In [72]:
from sklearn.datasets import load_iris
import sklearn.utils as utils

iris = load_iris()
print(iris.data[:5,:])
print(iris.target)
print(iris.target_names)
X, y = utils.shuffle(iris.data, iris.target, random_state=1)
print(X.shape)
print(y.shape)
print(y)
[[ 5.1  3.5  1.4  0.2]
 [ 4.9  3.   1.4  0.2]
 [ 4.7  3.2  1.3  0.2]
 [ 4.6  3.1  1.5  0.2]
 [ 5.   3.6  1.4  0.2]]
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
 2 2]
['setosa' 'versicolor' 'virginica']
(150, 4)
(150,)
[0 1 1 0 2 1 2 0 0 2 1 0 2 1 1 0 1 1 0 0 1 1 1 0 2 1 0 0 1 2 1 2 1 2 2 0 1
 0 1 2 2 0 2 2 1 2 0 0 0 1 0 0 2 2 2 2 2 1 2 1 0 2 2 0 0 2 0 2 2 1 1 2 2 0
 1 1 2 1 2 1 0 0 0 2 0 1 2 2 0 0 1 0 2 1 2 2 1 2 2 1 0 1 0 1 1 0 1 0 0 2 2
 2 0 0 1 0 2 0 2 2 0 2 0 1 0 1 1 0 0 1 0 1 1 0 1 1 1 1 2 0 0 2 1 2 1 2 2 1
 2 0]
In [73]:
train_set_size = 100
X_train = X[:train_set_size]  # selects first 100 rows (examples) for train set
y_train = y[:train_set_size]
X_test = X[train_set_size:]   # selects from row 100 until the last one for test set
y_test = y[train_set_size:]
print(X_train.shape, y_train.shape)
print(X_test.shape, y_test.shape)
(100, 4) (100,)
(50, 4) (50,)
In [74]:
from sklearn import tree

dtree = tree.DecisionTreeClassifier()
dtree = dtree.fit(X_train, y_train)
print(dtree.score(X_test,y_test))

y_pred = dtree.predict(X_test)
y_prob = dtree.predict_proba(X_test)
print(y_pred[:10])
print(y_prob[:10])
print(metrics.accuracy_score(y_test,y_pred))
print(metrics.confusion_matrix(y_test,y_pred))
0.9
[0 1 0 1 1 0 1 0 0 2]
[[ 1.  0.  0.]
 [ 0.  1.  0.]
 [ 1.  0.  0.]
 [ 0.  1.  0.]
 [ 0.  1.  0.]
 [ 1.  0.  0.]
 [ 0.  1.  0.]
 [ 1.  0.  0.]
 [ 1.  0.  0.]
 [ 0.  0.  1.]]
0.9
[[19  0  0]
 [ 0 17  1]
 [ 0  4  9]]
In [75]:
from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_train,y_train)
print(knn.score(X_test,y_test))

y_pred = knn.predict(X_test)
print(metrics.confusion_matrix(y_test,y_pred))
0.94
[[19  0  0]
 [ 0 16  2]
 [ 0  1 12]]
In [76]:
from sklearn import svm

#svm_clf = svm.LinearSVC()
#svm_clf = svm.SVC(kernel = 'poly')
svm_clf = svm.SVC()
svm_clf.fit(X_train,y_train)
print(svm_clf.score(X_test,y_test))
y_pred = svm_clf.predict(X_test)
print(metrics.confusion_matrix(y_test,y_pred))
0.98
[[19  0  0]
 [ 0 18  0]
 [ 0  1 12]]
In [77]:
import sklearn.linear_model as linear_model

lr_clf = linear_model.LogisticRegression()
lr_clf.fit(X_train, y_train)
print(lr_clf.score(X_test,y_test))
y_pred = lr_clf.predict(X_test)
print(metrics.confusion_matrix(y_test,y_pred))
probs = lr_clf.predict_proba(X_test)
print (probs)
print (probs.argmax(axis = 1))
print (probs.max(axis = 1))
0.96
[[19  0  0]
 [ 0 16  2]
 [ 0  0 13]]
[[  9.42848537e-01   5.71375773e-02   1.38860506e-05]
 [  7.36901098e-02   7.22694123e-01   2.03615768e-01]
 [  7.11546818e-01   2.87755584e-01   6.97598209e-04]
 [  4.89825361e-02   7.17197031e-01   2.33820433e-01]
 [  1.81245540e-02   6.43677875e-01   3.38197571e-01]
 [  8.56222465e-01   1.43598825e-01   1.78709592e-04]
 [  1.32661219e-02   6.69340523e-01   3.17393355e-01]
 [  8.90737869e-01   1.09004746e-01   2.57384935e-04]
 [  7.99367919e-01   2.00379203e-01   2.52878017e-04]
 [  1.26409293e-03   1.57157352e-01   8.41578555e-01]
 [  2.49910290e-03   2.46064318e-01   7.51436579e-01]
 [  6.39100774e-04   2.24939995e-01   7.74420904e-01]
 [  8.33706116e-01   1.66210803e-01   8.30810191e-05]
 [  8.94412871e-01   1.05503577e-01   8.35520171e-05]
 [  9.72020748e-03   3.00444130e-01   6.89835663e-01]
 [  7.98905301e-01   2.00868235e-01   2.26463968e-04]
 [  1.60403736e-03   2.95399378e-01   7.02996584e-01]
 [  7.87969440e-01   2.11818459e-01   2.12101017e-04]
 [  2.72568465e-03   2.82704683e-01   7.14569632e-01]
 [  3.64574157e-03   3.04211789e-01   6.92142469e-01]
 [  8.22276156e-01   1.77562266e-01   1.61577503e-04]
 [  2.09344504e-03   1.84118529e-01   8.13788026e-01]
 [  8.09401345e-01   1.90462524e-01   1.36131352e-04]
 [  6.59873831e-02   6.19755173e-01   3.14257444e-01]
 [  7.80727909e-01   2.18951841e-01   3.20249733e-04]
 [  6.71158547e-02   7.66891891e-01   1.65992254e-01]
 [  4.02285825e-02   6.89671837e-01   2.70099580e-01]
 [  8.56946753e-01   1.42998178e-01   5.50690948e-05]
 [  7.94214334e-01   2.05641671e-01   1.43995870e-04]
 [  1.25434739e-01   6.99690556e-01   1.74874705e-01]
 [  8.95288270e-01   1.04669525e-01   4.22044393e-05]
 [  5.07419345e-02   5.50399848e-01   3.98858218e-01]
 [  1.33216135e-02   5.48832988e-01   4.37845398e-01]
 [  8.37568960e-01   1.62343221e-01   8.78194536e-05]
 [  2.74945520e-02   6.41899518e-01   3.30605930e-01]
 [  4.41253274e-02   6.20573300e-01   3.35301372e-01]
 [  5.42982750e-03   5.05914148e-01   4.88656025e-01]
 [  4.37634497e-02   7.61758107e-01   1.94478443e-01]
 [  1.60403736e-03   2.95399378e-01   7.02996584e-01]
 [  8.18520368e-01   1.81402743e-01   7.68890305e-05]
 [  7.72715352e-01   2.27097263e-01   1.87384606e-04]
 [  7.72125588e-04   4.37082934e-01   5.62144941e-01]
 [  7.65259021e-02   7.19907084e-01   2.03567014e-01]
 [  1.39728181e-03   4.66169653e-01   5.32433065e-01]
 [  1.58140880e-01   7.60607424e-01   8.12516952e-02]
 [  3.77017915e-03   4.60739591e-01   5.35490230e-01]
 [  1.80675792e-03   3.27728808e-01   6.70464434e-01]
 [  3.59953765e-03   4.77448945e-01   5.18951517e-01]
 [  9.40355077e-04   1.96873447e-01   8.02186198e-01]
 [  7.82774102e-01   2.17108428e-01   1.17470682e-04]]
[0 1 0 1 1 0 1 0 0 2 2 2 0 0 2 0 2 0 2 2 0 2 0 1 0 1 1 0 0 1 0 1 1 0 1 1 1
 1 2 0 0 2 1 2 1 2 2 2 2 0]
[ 0.94284854  0.72269412  0.71154682  0.71719703  0.64367787  0.85622247
  0.66934052  0.89073787  0.79936792  0.84157856  0.75143658  0.7744209
  0.83370612  0.89441287  0.68983566  0.7989053   0.70299658  0.78796944
  0.71456963  0.69214247  0.82227616  0.81378803  0.80940134  0.61975517
  0.78072791  0.76689189  0.68967184  0.85694675  0.79421433  0.69969056
  0.89528827  0.55039985  0.54883299  0.83756896  0.64189952  0.6205733
  0.50591415  0.76175811  0.70299658  0.81852037  0.77271535  0.56214494
  0.71990708  0.53243307  0.76060742  0.53549023  0.67046443  0.51895152
  0.8021862   0.7827741 ]

Computing Scores

In [78]:
p,r,f,s = metrics.precision_recall_fscore_support(y_test,y_pred)
print(p)
print(r)
print(f)
report = metrics.classification_report(y_test,y_pred)
print(report)
[ 1.          1.          0.86666667]
[ 1.          0.88888889  1.        ]
[ 1.          0.94117647  0.92857143]
             precision    recall  f1-score   support

          0       1.00      1.00      1.00        19
          1       1.00      0.89      0.94        18
          2       0.87      1.00      0.93        13

avg / total       0.97      0.96      0.96        50

In [79]:
y_true = np.array([0, 0, 1, 1])
y_scores = np.array([0.1, 0.4, 0.35, 0.8])
precision, recall, thresholds = metrics.precision_recall_curve(y_true,y_scores)
plt.scatter(recall,precision)
print(recall)
print(precision)
print(thresholds)
fpr, tpr, ths = metrics.roc_curve(y_true,y_scores)
print(metrics.roc_auc_score(y_true,y_scores))
[ 1.   0.5  0.5  0. ]
[ 0.66666667  0.5         1.          1.        ]
[ 0.35  0.4   0.8 ]
0.75
In [80]:
import sklearn.cross_validation as cross_validation

scores = cross_validation.cross_val_score(#lr_clf,
                                          #svm_clf,
                                          #knn,
                                          dtree,
                                          X,
                                          y,
                                          scoring='accuracy',
                                          cv=5)
print (scores)
print (scores.mean())
[ 1.          0.93333333  0.96666667  0.96666667  0.86666667]
0.946666666667