Some Python reminders¶
I assume that you all know Python. If not you can look back at the course notes of <a href = 'http://ecourse.uoi.gr/course/view.php?id=489'>Introduction to Programming</a>.
There is also a nice introductory <a href = "https://github.com/mcrovella/CS506-Computational-Tools-for-Data-Science/blob/master/01-Intro-to-Python.ipynb">tutorial</a> from Evimaria Terzi and Mark Crovella from whom we will borrow a lot of material.
Here we will just give a few reminders.
Strings¶
String manipulation will be very important for many of the tasks we will do. Therefore let us play around a bit with strings.
#Concatenating strings
a = "Hello" # String
b = " World" # Another string
print (a + b) # Concatenation
# Slicing strings
a = "World"
print (a[0])
print (a[-1])
print ("World"[0:4])
print (a[::-1])
print(a[1:-1])
# Popular string functions
a = "Hello World"
l = list(a)
print(l)
print ("-".join(a))
print ("-".join(l))
print (a.startswith("Wo"))
print (a.endswith("rld"))
print (a.replace("o","0").replace("d","[)").replace("l","1"))
print (a.split())
print (a.split('o'))
Strings are an example of an imutable data type. Once you instantiate a string you cannot change any characters in it's set.
string = "string"
string[-1] = "y" #Here we attempt to assign the last character in the string to "y"
Lists, Tuples, Sets and Dictionaries¶
Number and strings alone are not enough! we need data types that can hold multiple values.
Lists:¶
Lists are mutable or able to be altered. Lists are a collection of data and that data can be of differing types.
groceries = []
# Add to list
groceries.append("oranges")
groceries.append("meat")
groceries.append("asparangus")
# Access by index
print (groceries[2])
print (groceries[0])
# Find number of things in list
print (len(groceries))
# Sort the items in the list
groceries.sort()
print (groceries)
# List Comprehension
veggie = [x for x in groceries if x is not "meat"]
print (veggie)
# Remove from list
groceries.remove("asparangus")
#groceries.pop()
print (groceries)
#The list is mutable
groceries[0] = 2
print (groceries)
groceries.sort()
print (groceries)
L = ['x',1,3,'y']
print(L.pop())
print(L.pop(0))
# lists are objects
L = [2,5,1,4]
X = L
L.sort()
print (X)
L.append(3)
print(X)
L = sorted(L)
print(L)
print(X)
#slicing works for lists as for strings
print (L[1:-1])
print (L[2:])
print(L[:-2])
print(L[1:-1:2])
List Comprehension¶
Recall the mathematical notation:
$$L_1 = \left\{x^2 : x \in \{0\ldots 9\}\right\}$$
$$L_2 = \left(1, 2, 4, 8,\ldots, 2^{12}\right)$$
$$M = \left\{x \mid x \in L_1 \text{ and } x \text{ is even}\right\}$$
L1 = [x**2 for x in range(10)] # range(n): returns an iterator over the numbers 0,...,n-1
L2 = [2**i for i in range(13)]
L3 = [x for x in L1 if x % 2 == 0]
print (L1)
print (L2)
print (L3)
[x for x in [x**2 for x in range(10)] if x % 2 == 0]
words = 'The quick brown fox jumps over the lazy dog'.split()
print(words)
upper = [w.upper() for w in words]
print(upper)
stuff = [[w.upper(), w.lower(), len(w)] for w in words]
print(stuff)
s = input('Give numbers separated by comma: ')
x = [int(n) for n in s.split(',')]
print(x)
y = s.split(',')
print(y)
print(y[0]+y[1])
print(x[0]+x[1])
#create a vector of all 10 zeros
z = [0 for i in range(10)]
print(z)
#create a 10x10 matrix with all 0s
M = [[0 for i in range(10)] for j in range(10)]
#set the diagonal to 1
for i in range(10): M[i][i] = 1
print(M)
#create a list of random integers in [0,99]
import random
R = [random.choice(range(100)) for i in range(10)]
print(R)
# Removing elements from a list while you iterate it can lead to problems
L = [1,2,4,5,6,8]
for x in L:
if x%2 == 0:
L.remove(x)
print(L)
#Another way to do this:
L = [1,2,4,5,6,8]
L = [x for x in L if x%2 == 1] #creates a new list
L[:] = [x for x in L if x%2 == 1]
print(L)
L = [1,2,4,5,6,8]
R =[y for y in L if y%2 == 0]
for x in R: L.remove(x)
print(L)
Tuples:¶
Tuples are an immutable type. Like strings, once you create them, you cannot change them. It is their immutability that allows you to use them as keys in dictionaries. However, they are similar to lists in that they are a collection of data and that data can be of differing types.
# Tuple grocery list
groceries = ('orange', 'meat', 'asparangus', 2.5, True)
print (groceries)
#print(groceries[2])
#groceries[2] = 'milk'
L = [1,2,3]
t = tuple(L)
print(t)
L[1] = 5
print(t)
t[1] = 4
Sets:¶
A set is a sequence of items that cannot contain duplicates. They handle operations like sets in mathematics.
numbers = range(10)
evens = [2, 4, 6, 8]
evens = set(evens)
numbers = set(numbers)
# Use difference to find the odds
odds = numbers - evens
print (odds)
# Note: Set also allows for use of union (|), and intersection (&)
a = [2,1,2,1]
print (a)
a = set(a)
print(a)
Dictionaries:¶
A dictionary is a map of keys to values. This is one of the most useful structures. Keys must be unique and immutable.
# A simple dictionary
simple_dict = {'cse012': 'data mining'}
# Access by key
print (simple_dict['cse012'])
# A longer dictionary
classes = {
'cse012': 'data mining',
'cse205': 'object oriented programming'
}
# Check if item is in dictionary
print ('cse012' in classes)
# Add new item
classes['L14'] = 'social networks'
print (classes['L14'])
# Print just the keys
print (list(classes.keys()))
# Print just the values
print (list(classes.values()))
# Print the items in the dictionary
print (list(classes.items()))
# Print dictionary pairs another way
for key, value in classes.items():
print (key, value)
for key in classes:
print (key, classes[key])
#change values in a dictionary
classes['L14'] = 'graduate social networks'
print (classes['L14'])
# Complex Data structures
# Dictionaries inside a dictionary!
professors = {
"prof1": {
"name": "Panayiotis Tsaparas",
"department": "Computer Science",
"research interests": ["algorithms", "data mining", "machine learning",]
},
"prof2": {
"name": "Yanis Varoufakis",
"department": "Economics",
"interests": ["debt", "game theory", "parallel currency",],
}
}
for prof in professors:
print (professors[prof]["name"])
professors['prof2']['interests'][1]
Depending on the task that we want to perform, it makes a bid difference in efficiency what structure we use. When searching over a structure it is important to use a Set or a Dictionary structure since search is done in constant time in expectation (or O(logn) worst case). This makes a huge difference when dealing with large datasets.
# The importance of using the right structure:
import random
L = [random.choice(range(1000000)) for i in range(1000)]
import time
t = time.clock()
count = 0;
for x in range(1000000):
if x in L:
count += 1
print (time.clock() - t)
S = set(L)
t = time.clock()
count = 0;
for x in range(1000000):
if x in S:
count += 1
print (time.clock() - t)
#What structure should we use for storing the edges of a graph with millions of edges?
L = [[1,2],[1,4],[2,5],[3,7],[5,7],[6,8]] # etc...
S = set([tuple(x) for x in L])
print(S)
#What if we have also a weight on the edges, and we want to get the weight?
L = [[1,2,0.2],[1,4,0.5],[2,5,0.8],[3,7,0.1],[5,7,0.4],[6,8,0.9]]
D = {}
for x in L:
D[tuple(x[0:2])] = float(x[2])
print(D)
#What if we want to be able to get the neighbors of each node and the weight of the edge?
D = {}
for x in L:
v1,v2,w = x
if v1 in D:
D[v1][v2] = w
else:
D[v1] = {}
D[v1][v2] = w
print(D)
print(D[1][2])
Lambda functions¶
Python supports the creation of anonymous functions (i.e. functions that are not bound to a name) at runtime, using a construct called "lambda".
def f (x): return x**2
print (f(8))
g = lambda x: x**2
print (g(8))
The above pieces of code are equivalent to each other! Note that there is no ``return" statement in the lambda function. A lambda function does not need to be assigned to variable, but it can be used within the code wherever a function is expected.
f = lambda x, y : x + y
print (f(2,3))
def multiply (n): return lambda x: x*n
multiply_by_2 = multiply(2)
g = multiply(6)
print (multiply_by_2)
print (multiply_by_2(10), g(10))
multiply(3)(30)
The map() function¶
The advantage of the lambda operator can be seen when it is used in combination with the map() function. map() is a function with two arguments:
r = map(func,s)
func is a function and s is a sequence (e.g., a list). map returns an iterator where we have applied function func to all the elements of s. You need to convert this into a list if you want to use it.
map() and lambda give functionality very similar to that of list comprehension
def dollar2euro(x):
return 0.89*x
def euro2dollar(x):
return 1.12*x
amounts= (100, 200, 300, 400)
dollars = map(dollar2euro, amounts)
print (dollars)
print (list(dollars))
amounts= (100, 200, 300, 400)
euros = map(euro2dollar, amounts)
print (list(euros))
list(map(lambda x: 0.89*x, amounts))
map can also be applied to more than one lists as long as they are of the same size and type
a = [1,2,3,4,5]
b = [-1,-2,-3, -4, -5]
c = [10, 20 , 30, 40, 50]
l1 = list(map(lambda x,y: x+y, a,b))
print (l1)
l2 = list(map (lambda x,y,z: x-y+z, a,b,c))
print (l2)
words = 'The quick brown fox jumps over the lazy dog'.split()
uwords = list(map(lambda w: [w.upper(), w.lower(), len(w)], words))
for t in uwords:
print (t)
The filter() function¶
The function filter(function, list) filters out all the elements of a list, for which the function function returns True. Returns an iterator
nums = [i for i in range(100)]
print (nums)
even = list(filter(lambda x: x%2==0 and x!=0, nums))
print (even)
Command line arguments¶
To read the command line arguments use the sys.argv list. The first element of this list is the program name. More sophisiticated processing can be done using the getopt.getopt method
import sys
print ('Number of arguments:', len(sys.argv), 'arguments.')
print ('Argument List:', str(sys.argv))
Libraries¶
Python is a high-level open-source language. But the Python world is inhabited by many packages or libraries that provide useful things like array operations, plotting functions, and much more. We can (and we should) import libraries of functions to expand the capabilities of Python in our programs.
There are many python libraries for data mining, and we will use many of these libraries in this course.
Example: The random library¶
import random
print (random.choice(range(10))) # generates a random number in the range [0,9]
myList = [2, 109, False, 10, "data", 482, "mining"]
print(random.choice(myList))
from random import shuffle # imports a specific function of random that can be used without the random. prefix
x = [i for i in range(10)]
print (x)
shuffle(x)
print (x)
# some more methods of random
print(random.sample([1,3,4,7,8,9,20],3)) #samples 3 elements uniformly at random
print(random.random()) # a random number in [0,1)
print(random.uniform(-1,1)) #a random number in [-1,1]
print(random.gauss(0,1)) #sample from a gaussian distribution with mean 0 and std 1
Including images¶
from IPython.display import Image
Image(filename = "CSE-UOI-LOGO-EN.jpg",width = 231.4, height = 272.9)
#Image(response.content)
