Supervised learning using scikit-learn¶
The goal of this tutorial is to introduce you to the scikit libraries for classification. We will also cover feature selection, and evaluation.
import numpy as np
import scipy.sparse as sp_sparse
import matplotlib.pyplot as plt
import sklearn as sk
import sklearn.datasets as sk_data
import sklearn.metrics as metrics
import seaborn as sns
%matplotlib inline
Feature Selection¶
Feature selection is about finding the best features for your classifier. This may be important if you do not have enough training data. The idea is to find metrics that either characterize the features by themselves, or with respect to the class we want to predict, or with respect to other features.
http://scikit-learn.org/stable/modules/feature_selection.html
Variance Threshold¶
The VarianceThreshold selection drops features whose variance is below some threshold. If we have binary features we can estimate the treshold exactly so as to guarantee a specific ratio of 0's and 1's
from sklearn.feature_selection import VarianceThreshold
X = [[0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 1, 1], [0, 1, 0], [0, 1, 1]]
print(np.array(X))
sel = VarianceThreshold(threshold=(.8 * (1 - .8)))
sel.fit_transform(X)
import sklearn.datasets as sk_data
iris = sk_data.load_iris()
X = iris.data
print(X[1:10,:])
print(X.var(axis = 0))
sel = VarianceThreshold(threshold=0.2)
sel.fit_transform(X)[1:10]
Univariate Feature Selection¶
A more sophisticated feature selection technique uses test to determine if a feature and the class label are independent. An example of such a test is the chi-square test (there are more)
In this case we keep the features with high chi-square score and low p-value.
The features with the lowest scores and highest values are rejected.
The chi-square test is usually applied on categorical data.
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
iris = sk_data.load_iris()
X, y = iris.data, iris.target
print(X.shape)
print('Features:')
print(X[1:10,:])
print('Labels:')
print(y[1:10])
sel = SelectKBest(chi2, k=2)
X_new = sel.fit_transform(X, y)
print('Selected Features:')
print(X_new[1:10])
The chi-square values and the p-values between features and target variable (X columns and y)
print('Chi2 values')
print(sel.scores_)
c,p = sk.feature_selection.chi2(X, y)
print('Chi2 values')
print(c) #The chi-square value between X columns and y
print('p-values')
print(p) #The p-value for the test
Supervised Learning¶
Python has several classes and objects for implementing different supervised learning techniques such as Regression and Classification.
Regardless of the model being implemented, the following methods are implemented:
The method fit() takes the training data and labels/values, and trains the model
The method predict() takes as input the test data and applies the model.
Preparing the data¶
To perform classification we first need to prepare the data into train and test datasets.
from sklearn.datasets import load_iris
import sklearn.utils as utils
iris = load_iris()
print("sample of data")
print(iris.data[:5,:])
print("the class labels vector")
print(iris.target)
print("the names of the classes:",iris.target_names)
print(iris.feature_names)
Randomly shuffle the data. This is useful to know that the data is in random order
X, y = utils.shuffle(iris.data, iris.target, random_state=1) #shuffle the data
print(X.shape)
print(y.shape)
print(y)
Select a subset for training and a subset for testing
train_set_size = 100
X_train = X[:train_set_size] # selects first 100 rows (examples) for train set
y_train = y[:train_set_size]
X_test = X[train_set_size:] # selects from row 100 until the last one for test set
y_test = y[train_set_size:]
print(X_train.shape, y_train.shape)
print(X_test.shape, y_test.shape)
We can also use the train_test_split function of python for splitting the data into train and test sets. In this case you do not need the random shuffling (but it does not hurt).
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=0)
Classification models¶
http://scikit-learn.org/stable/supervised_learning.html#supervised-learning
Python has classes and objects that implement the different classification techniques that we described in class.
Decision Trees¶
http://scikit-learn.org/stable/modules/tree.html
Train and apply a decision tree classifier. The default score computed in the classifier object is the accuracy. Decision trees can also give us probabilities
from sklearn import tree
dtree = tree.DecisionTreeClassifier()
dtree = dtree.fit(X_train, y_train)
print("classifier accuracy:",dtree.score(X_test,y_test))
y_pred = dtree.predict(X_test)
y_prob = dtree.predict_proba(X_test)
print("classifier predictions:",y_pred[:10])
print("ground truth labels :",y_test[:10])
print(y_prob[:10])
Compute some more metrics
print("accuracy:",metrics.accuracy_score(y_test,y_pred))
print("\nConfusion matrix")
print(metrics.confusion_matrix(y_test,y_pred))
print("\nPrecision Score per class")
print(metrics.precision_score(y_test,y_pred,average=None))
print("\nAverage Precision Score")
print(metrics.precision_score(y_test,y_pred,average='weighted'))
print("\nRecall Score per class")
print(metrics.recall_score(y_test,y_pred,average=None))
print("\nAverage Recall Score")
print(metrics.recall_score(y_test,y_pred,average='weighted'))
print("\nF1-score Score per class")
print(metrics.f1_score(y_test,y_pred,average=None))
print("\nAverage F1 Score")
print(metrics.f1_score(y_test,y_pred,average='weighted'))
Visualize the decision tree.
For this you will need to install the package python-graphviz
#conda install python-graphviz
import graphviz
print(iris.feature_names)
dot_data = tree.export_graphviz(dtree,out_file=None)
graph = graphviz.Source(dot_data)
graph
dtree2 = tree.DecisionTreeClassifier(max_depth=2)
dtree2 = dtree2.fit(X_train, y_train)
print(dtree2.score(X_test,y_test))
dot_data2 = tree.export_graphviz(dtree2,out_file=None)
graph2 = graphviz.Source(dot_data2)
graph2
k-NN Classification¶
https://scikit-learn.org/stable/modules/neighbors.html#classification
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_train,y_train)
print("classifier score:", knn.score(X_test,y_test))
y_pred = knn.predict(X_test)
print("\naccuracy:",metrics.accuracy_score(y_test,y_pred))
print("\nConfusion matrix")
print(metrics.confusion_matrix(y_test,y_pred))
print("\nPrecision Score per class")
print(metrics.precision_score(y_test,y_pred,average=None))
print("\nAverage Precision Score")
print(metrics.precision_score(y_test,y_pred,average='weighted'))
print("\nRecall Score per class")
print(metrics.recall_score(y_test,y_pred,average=None))
print("\nAverage Recall Score")
print(metrics.recall_score(y_test,y_pred,average='weighted'))
print("\nF1-score Score per class")
print(metrics.f1_score(y_test,y_pred,average=None))
print("\nAverage F1 Score")
print(metrics.f1_score(y_test,y_pred,average='weighted'))
from sklearn import svm
#svm_clf = svm.LinearSVC()
#svm_clf = svm.SVC(kernel = 'poly')
svm_clf = svm.SVC()
svm_clf.fit(X_train,y_train)
print("classifier score:",svm_clf.score(X_test,y_test))
y_pred = svm_clf.predict(X_test)
print("\naccuracy:",metrics.accuracy_score(y_test,y_pred))
print("\nConfusion matrix")
print(metrics.confusion_matrix(y_test,y_pred))
print("\nPrecision Score per class")
print(metrics.precision_score(y_test,y_pred,average=None))
print("\nAverage Precision Score")
print(metrics.precision_score(y_test,y_pred,average='weighted'))
print("\nRecall Score per class")
print(metrics.recall_score(y_test,y_pred,average=None))
print("\nAverage Recall Score")
print(metrics.recall_score(y_test,y_pred,average='weighted'))
print("\nF1-score Score per class")
print(metrics.f1_score(y_test,y_pred,average=None))
print("\nAverage F1 Score")
print(metrics.f1_score(y_test,y_pred,average='weighted'))
import sklearn.linear_model as linear_model
lr_clf = linear_model.LogisticRegression(solver='lbfgs')
lr_clf.fit(X_train, y_train)
print("classifier score:",lr_clf.score(X_test,y_test))
y_pred = lr_clf.predict(X_test)
print("\naccuracy:",metrics.accuracy_score(y_test,y_pred))
print("\nConfusion matrix")
print(metrics.confusion_matrix(y_test,y_pred))
print("\nPrecision Score per class")
print(metrics.precision_score(y_test,y_pred,average=None))
print("\nAverage Precision Score")
print(metrics.precision_score(y_test,y_pred,average='weighted'))
print("\nRecall Score per class")
print(metrics.recall_score(y_test,y_pred,average=None))
print("\nAverage Recall Score")
print(metrics.recall_score(y_test,y_pred,average='weighted'))
print("\nF1-score Score per class")
print(metrics.f1_score(y_test,y_pred,average=None))
print("\nAverage F1 Score")
print(metrics.f1_score(y_test,y_pred,average='weighted'))
For Logistic Regression we can also obtain the probabilities for the different classes
probs = lr_clf.predict_proba(X_test)
print("Class Probabilities (first 10):")
print (probs[:10])
print(probs.argmax(axis = 1)[:10])
print(probs.max(axis = 1)[:10])
And the coeffients of the logistic regression model
print(lr_clf.coef_)
Linear Regression¶
Linear Regression is implemented in the library sklearn.linear_model.LinearRegression: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html
from sklearn.linear_model import LinearRegression
X_reg = np.array([[1, 1], [1, 2], [2, 2], [2, 3]])
# y = 1 * x_0 + 2 * x_1 + 3
y_reg = np.dot(X_reg, np.array([1, 2])) + 3
reg = LinearRegression().fit(X_reg, y_reg)
#Obtain the function coefficients
print(reg.coef_)
#and the intercept
print(reg.intercept_)
The $R^2$ score computes the "explained variance"
$R^2 = 1-\frac{\sum_i (y_i -\hat y_i)^2}{\sum_i (y_i -\bar y)^2}$
where $\hat y_i$ is the prediction for point $x_i$ and $\bar y$ is the mean value of the target variable
print(reg.score(X_reg, y_reg))
#Predict for a new point
reg.predict(np.array([[3, 5]]))
A more complex example with the diabetes dataset
diabetes_X, diabetes_y = sk_data.load_diabetes(return_X_y=True)
# Shuffle the data
diabetes_X, diabetes_y = utils.shuffle(diabetes_X, diabetes_y, random_state=1)
# Split the data into training/testing sets
diabetes_X_train = diabetes_X[:-20]
diabetes_X_test = diabetes_X[-20:]
# Split the targets into training/testing sets
diabetes_y_train = diabetes_y[:-20]
diabetes_y_test = diabetes_y[-20:]
# Create linear regression object
regr = linear_model.LinearRegression()
# Train the model using the training sets
regr.fit(diabetes_X_train, diabetes_y_train)
# Make predictions using the testing set
diabetes_y_pred = regr.predict(diabetes_X_test)
# The coefficients
print('Coefficients: \n', regr.coef_)
# The mean squared error
print('Mean squared error: %.2f'
% metrics.mean_squared_error(diabetes_y_test, diabetes_y_pred))
# The coefficient of determination: 1 is perfect prediction
# Computed over the *test* data
print('Coefficient of determination: %.2f'
% metrics.r2_score(diabetes_y_test, diabetes_y_pred))
print('Predictions:')
print(diabetes_y_pred)
print('True values:')
print(diabetes_y_test)
Computing Scores¶
p,r,f,s = metrics.precision_recall_fscore_support(y_test,y_pred)
print(p)
print(r)
print(f)
print(s)
report = metrics.classification_report(y_test,y_pred)
print(report)
cancer_data = sk_data.load_breast_cancer()
X_cancer,y_cancer = utils.shuffle(cancer_data.data, cancer_data.target, random_state=1)
X_cancer_train = X_cancer[:500]
y_cancer_train = y_cancer[:500]
X_cancer_test = X_cancer[500:]
y_cancer_test = y_cancer[500:]
lr_clf.fit(X_cancer_train, y_cancer_train)
print("classifier score:",lr_clf.score(X_cancer_test,y_cancer_test))
y_cancer_pred = lr_clf.predict(X_cancer_test)
cancer_probs = lr_clf.predict_proba(X_cancer_test)
print("Class Probabilities (first 10):")
print (cancer_probs[:10])
y_cancer_scores = cancer_probs[:,1]
precision, recall, thresholds = metrics.precision_recall_curve(y_cancer_test,y_cancer_scores)
#plt.scatter(recall,precision)
plt.plot(recall,precision, color='darkorange',lw=2)
print(recall)
print(precision)
print(thresholds)
fpr, tpr, ths = metrics.roc_curve(y_cancer_test,y_cancer_scores)
plt.plot(fpr,tpr,color='darkorange',lw=2)
print(metrics.roc_auc_score(y_cancer_test,y_cancer_scores))
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic example')
plt.legend(loc="lower right")
plt.show()
(Xtoy,y_toy)=sk_data.make_classification(n_samples=1000)
Xttrain = Xtoy[:800,:]
Xttest = Xtoy[800:,:]
yttrain = y_toy[:800]
yttest = y_toy[800:]
lr_clf.fit(Xttrain, yttrain)
#print(lr_clf.score(Xttest,yttest))
#y_tpred = lr_clf.predict(X_test)
tprobs = lr_clf.predict_proba(Xttest)
print (tprobs[:10])
y_tscores = tprobs[:,1]
precision, recall, thresholds = metrics.precision_recall_curve(yttest,y_tscores)
plt.plot(recall,precision)
k-fold cross validation¶
In k-fold cross validation the data is split into k equal parts, the k-1 are used for training and the last one for testing. k models are trained, each time leaving a different part for testing
https://scikit-learn.org/stable/modules/cross_validation.html
There are two methods for implementing k-fold cross-validation, under the library model selection: cross_val_score, and cross validate. The latter allows multiple metrics to be considered together.
import sklearn.model_selection as model_selection
scores = model_selection.cross_val_score(#lr_clf,
#svm_clf,
#knn,
dtree,
X,
y,
scoring='f1_weighted',
cv=5)
print (scores)
print (scores.mean())
scores = model_selection.cross_validate(#lr_clf,
#svm_clf,
#knn,
dtree,
X,
y,
scoring=['precision_weighted','recall_weighted'],
cv=3)
print (scores)
print (scores['test_precision_weighted'].mean(),scores['test_recall_weighted'].mean())
Creating a pipeline¶
If the same steps are often repeated, you can create a pipeline to perform them all at once:
Text classification Example¶
We will use the 20 newsgroups to do a text classification example
from sklearn.datasets import fetch_20newsgroups
categories = ['sci.space','rec.sport.baseball']
#categories = ['alt.atheism', 'rec.sport.baseball']
news_train = sk_data.fetch_20newsgroups(subset='train',
remove=('headers', 'footers', 'quotes'),
categories=categories)
print (len(news_train.target))
X_news_train_data = news_train.data
y_news_train = news_train.target
news_test = sk_data.fetch_20newsgroups(subset='test',
remove=('headers', 'footers', 'quotes'),
categories=categories)
print (len(news_test.target))
X_news_test_data = news_test.data
y_news_test = news_test.target
from sklearn.linear_model import LinearRegression
import sklearn.linear_model as linear_model
import sklearn.feature_extraction.text as sk_text
vectorizer = sk_text.TfidfVectorizer(stop_words='english',
max_features = 1000,
min_df=4, max_df=0.8)
X_news_train = vectorizer.fit_transform(X_news_train_data)
lr_clf = linear_model.LogisticRegression(solver='lbfgs')
lr_clf.fit(X_news_train, y_news_train)
X_news_test = vectorizer.transform(X_news_test_data)
print("classifier score:",lr_clf.score(X_news_test,y_news_test))
Word embeddings and Text classification¶
We will now see how we can train and use word embeddings. We will also see the NLTK library.
The NLTK libary allows for sohpisticated text processing. It can do stemming, create a parse tree, do PoS (Part-of-Speech) tagging, find Noun Phrases, entities,
import nltk
from nltk.tokenize import sent_tokenize, word_tokenize
nltk.download('punkt')
import string as string
X_news_train_nltk = []
y_news_train_nltk = []
for x,y in zip(X_news_train_data,y_news_train):
wt = word_tokenize(x.lower())
doc = [w for w in wt if w not in string.punctuation]
if len(doc) == 0: continue
X_news_train_nltk.append(doc)
y_news_train_nltk.append(y)
from nltk.corpus import stopwords
nltk.download('stopwords')
english_stop_words = set(stopwords.words('english'))
X_news_test_nltk = []
y_news_test_nltk = []
for x,y in zip(X_news_test_data,y_news_test):
wt = word_tokenize(x.lower())
doc = [w for w in wt if (w not in english_stop_words) and (w not in string.punctuation)]
if len(doc) == 0: continue
X_news_test_nltk.append(doc)
y_news_test_nltk.append(y)
The Gensim library¶
The Gensim library has several NLP models.
You can use existing modules to train a word2vec model: https://radimrehurek.com/gensim/models/word2vec.html
import gensim
from gensim.models import Word2Vec
Train a CBOW embedding on the training data corpus
embedding_size = 100
cbow_model = gensim.models.Word2Vec(X_news_train_nltk, min_count = 1,size = embedding_size, window = 50)
cbow_model.wv['space']
Transform the train and test data
X_news_train_cbow = []
for x in X_news_train_nltk:
vx = np.zeros(embedding_size)
for w in x:
vx += cbow_model.wv[w]
vx /= len(x)
X_news_train_cbow.append(vx)
X_news_test_cbow = []
for x in X_news_test_nltk:
vx = np.zeros(embedding_size)
length = 0;
for w in x:
if (w not in cbow_model.wv): continue
length += 1
vx += cbow_model.wv[w]
if length != 0: vx /= length
X_news_test_cbow.append(vx)
Train a classifier on the emebddings
lr_clf.fit(np.array(X_news_train_cbow), np.array(y_news_train_nltk))
lr_clf.score(np.array(X_news_test_cbow),y_news_test_nltk)
Train a SkipGram embedding on the training data corpus
embedding_size = 100
skipgram_model = gensim.models.Word2Vec(X_news_train_nltk, min_count = 1,size = embedding_size, window = 50, sg = 1)
Transform the train and test data
X_news_train_skipgram = []
for x in X_news_train_nltk:
vx = np.zeros(embedding_size)
for w in x:
vx += skipgram_model.wv[w]
vx /= len(x)
X_news_train_skipgram.append(vx)
X_news_test_skipgram = []
for x in X_news_test_nltk:
vx = np.zeros(embedding_size)
length = 0
for w in x:
if (w not in skipgram_model.wv): continue
length += 1
vx += skipgram_model.wv[w]
if length!= 0: vx /= length
X_news_test_skipgram.append(vx)
Train a classifier on the emebddings
lr_clf.fit(np.array(X_news_train_skipgram), np.array(y_news_train_nltk))
lr_clf.score(np.array(X_news_test_skipgram),y_news_test_nltk)
You can also download the Google word2vec model trained over millions of documents
import gensim.downloader as api
path = api.load("word2vec-google-news-300", return_path=True)
print(path)
path = 'C:\\Users\\tsapa/gensim-data\word2vec-google-news-300\word2vec-google-news-300.gz'
g_model = gensim.models.KeyedVectors.load_word2vec_format(path, binary=True)
print(len(g_model['hello']))
print(g_model.wv['hello'])
Transform the train and test data
X_news_train_gmodel = []
for x in X_news_train_nltk:
vx = np.zeros(300)
length = 0
for w in x:
if w in g_model.wv:
length += 1
vx += g_model[w]
if length != 0: vx /= length
X_news_train_gmodel.append(vx)
X_news_test_gmodel = []
for x in X_news_test_nltk:
vx = np.zeros(300)
length = 0
for w in x:
if (w not in g_model.wv): continue
length += 1
vx += g_model.wv[w]
if length != 0: vx /= length
X_news_test_gmodel.append(vx)
Train a classifier on the emebddings
lr_clf.fit(X_news_train_gmodel, np.array(y_news_train_nltk))
lr_clf.score(np.array(X_news_test_gmodel),y_news_test_nltk)