PandA Python Training 1: Intro to Python¶

Andrew Lau

# this just gets the notebook to print all the output
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

Contents¶

Introduction
- Notebooks and IDE
- What is Python?
- Python vs R vs SAS
Python Crash Course
- Syntax
- Control structures
- Data Types/Structures
- Functions
- Object oriented programming
- Modules

Introduction¶

Notebooks and IDE¶

Before we begin, let's talk a bit about notebooks and IDEs.

Jupyter Notebooks

What you looking at in your web browser (hopefully, if you got it working, otherwise a PDF) is a Jupyter Notebook
It was designed for data science in mind and reproducible research
It acts as a data scientists' notebook, allowing note taking, documentation, mathematical notation as well as most importantly running code and storing results all in the one document
These notes are live and interactive! You can run and edit the code during this session!

IDEs

a 'code editor'. R Studio is an IDE.
Some good IDE (code editors, Integrated Development Environment)
- Spyder (feels like R Studio, made for Data Science, probably easiest to get used to). If you have installed Anaconda, you already have it. Hit your windows key and type 'spyder' and you should see it.
- Visual Studio
- Sublime Text
- Eclipse

What is Python¶

Unlike R or SAS, Python is a general purpose programming language that can do basically anything
Pythons most popular applications are:
- Web development (powering backends using django/flask frameworks). Website that use some Python for their websites include Google, YouTube, Quora, Dropbox, Yahoo!, Reddit, Instagram and Spotify.
- Data science and machine learning. The ultra high tech, users of machine learning such as Google, Amazon will have Python driving a lot of their machine learning.
- Scripting. Python's readibility and speed of development lends well to creating quick scripts to automate tasks
- Education. Python is probably the most popular language to teach computer science with as unis.
Python can be do virtually anything:
- games (not the best for making games though)
- breaching security networks
- developing desktop applications (not the best for this, however Spyder (a desktop app) is written in Python)
- robotics
- raspberry pi
Developed initially to teach children how to program. This is why the Python language has been built to value simplicity and readibility.
- Pythonic: code which is simple, clear, concise, maintainable and uses Python the way it was intended.

Python vs R vs SAS¶

Python:

General purpose language developed by computer scientists/software engineers
More popular with computer scientists and software engineers. More popular on Kaggle.
Has superior and better developed deep learning packages (e.g. tensorflow, keras, pytorch)
Much easier to integrate into systems (e.g. live machine learning models, integrating a model into a website backend)
Because Python is a general purpose programming language, it is easier to build software and systems that have machine learning or AI in it
Lighter footprint, advantageous for cloud computing
Data science modules are far more unifed (a more cohesive and inified system of modules)
The syntax, naming conventions and how things work are far more unified as there are a few key packages basically everyone uses and is worked on by everyone (compared to the plethora of R packages maintained by a few individuals)

R:

Built from ground up for statistical analysis
Better for performing traditional statistical analysis (e.g. significance tests, ANOVA, statistical inference, linear models...)
More popular with those from stats/maths backgrounds
Has niche stats libraries/packages not found in Python
Can be better for plotting / EDA (e.g. ggplot2)

SAS

Closed source, with far, far fewer packages available
Is not really used at big tech companies like Amazon, Google, Facebook or by "high end" data scientists
Much harder to find help on online. There are far more users of Python/R and there is a strong culture of contributing to the open source project so it is far easier to find help online
ML, and AI pales in comparison to Python/R
Graphs and plotting in SAS is not great
Expensive

Overview of the language¶

We will cover the basics of the Python language today
Python standard libaries

Syntax¶

Python uses whitespace instead of brackets to delimit blocks (indicate the start and end of if statements, functions etc)
This was a conscious design choice to encourage clean code through making tab indentation mandatory as well as ditching a lot of brackets.

# tabs and new lines are used to delimit the condition and the action to be executed
# if the condition is true
for i in range(5):
    print(i, "hello")
    
# function and if statement
def double_num(x):
    if x > 5:
        return 2 * x
    return x / 2

# hashes are used to comment. press ctrl + '/' to (un)comment
print("double_num(2):", double_num(2))  # you can also inline comment
print("double_num(2):", double_num(6))
"""
Use three quotation marks to do block commenting.
"""

0 hello
1 hello
2 hello
3 hello
4 hello
double_num(2): 1.0
double_num(2): 12

'\nUse three quotation marks to do block commenting.\n'

Control Structures¶

if statements¶

x = 5
if x > 2:
    print("x is greater than 2")
    
if x == 5:  # note that unlike SAS, the equality operator "==" is different to the assignment operator "="
    print("x is 5")

x is greater than 2
x is 5

for loops¶

# in Python we can loop over many things (we say it is 'iterable')
for i in range(10):  # the range(x) 'generator' allows us to loop over 0 to 1 - x
    print(i)
    
my_list = ['t', 'o', 'n', 'y']

for letter in my_list:  # we can loop over lists
    print(letter)

0
1
2
3
4
5
6
7
8
9
t
o
n
y

while loops¶

x = 0
while x < 10:
    print(x)
    x += 1

Data Types¶

The basic data types in Python are:

# integers
2
3
4

2

3

4

# floats - you can think of these as decimals, (floats are scientific notation in binary to
# allow efficient storage of decimals and very big/small numbers)
2.234
3.2309

2.234

3.2309

# strings - series of characters
"hello"
'bye'  # you can use single or double quotations marks to delimit strings
# if you want single (double) quotation marks in your string, you can use the double (single) 
# quotation marks to delimit
"single 'quotations'"
'double "quotations"'
# there are many useful string operations
[x for x in dir(str) if not x.startswith('_')]  # this is a list comprehension, more on this later

'hello'

'bye'

"single 'quotations'"

'double "quotations"'

['capitalize',
 'casefold',
 'center',
 'count',
 'encode',
 'endswith',
 'expandtabs',
 'find',
 'format',
 'format_map',
 'index',
 'isalnum',
 'isalpha',
 'isascii',
 'isdecimal',
 'isdigit',
 'isidentifier',
 'islower',
 'isnumeric',
 'isprintable',
 'isspace',
 'istitle',
 'isupper',
 'join',
 'ljust',
 'lower',
 'lstrip',
 'maketrans',
 'partition',
 'replace',
 'rfind',
 'rindex',
 'rjust',
 'rpartition',
 'rsplit',
 'rstrip',
 'split',
 'splitlines',
 'startswith',
 'strip',
 'swapcase',
 'title',
 'translate',
 'upper',
 'zfill']

bad = 'bad'
bad.capitalize()
bad.upper()

'Bad'

'BAD'

# booleans
True
False
# data types can be 'cast' into other data types. The boolean True has the value 1 and False 0:
True + 1
50 - False

True

False

2

50

# zero is considered True, and other numbers False
def is_true(number):
    if number:
        print(number, "is True")
    else:
        print(number, "is False")

for number in [-100, -0.5, 0, 0.5, 100, 'Jessie']:
    is_true(number)

-100 is True
-0.5 is True
0 is False
0.5 is True
100 is True
Jessie is True

Data Structures¶

The standard data structures (things that can hold data) in Python are:

lists
tuples
dictionaries
sets

Lists¶

Perhaps the most versatile and commonly used data structure in Python

# lists can hold anything! from basic data types like integers, floats, strings...
[1, 2, 3, 'one', 'two', 'three', 1.0, 2.0, 3.0]

# to other data structures...
[['this is a str in a list'], {'this is a str in a set'}, {'apple':'red', 'strawberry':'red'}]

# to functions...
def add_2(x):
    return x + 2

def add_3(x):
    return x + 2

[add_2, add_3]

# and any other 'object' <- more on this later!

[1, 2, 3, 'one', 'two', 'three', 1.0, 2.0, 3.0]

[['this is a str in a list'],
 {'this is a str in a set'},
 {'apple': 'red', 'strawberry': 'red'}]

[<function __main__.add_2(x)>, <function __main__.add_3(x)>]

# accessing elements of a list
my_list = [1 ,2, 3, 4, 5]
my_list[0]  # note that Python (and a lot of other languages) start their indexing at 0 rather than 1!
my_list[3]

1

4

# 'slicing' - accessing subsets of a list
my_list[0:2]  # the index at the right of the colon operator is not included in the slice
my_list[2:4]
my_list[:2]
my_list[2:]

[1, 2]

[3, 4]

[1, 2]

[3, 4, 5]

# generating lists
# one of the most common ways to build a list is to start with an empty one and build it up with a loop
x = []  # initialise empty list
for i in range(5):
    x.append(i)
x

[0, 1, 2, 3, 4]

# list comprehensions are a more compact way to build a list
y = [i for i in range(5)]

# the dir() function gives you a list of any objects attributes/methods <- more on this later
dir(list)

['__add__',
 '__class__',
 '__contains__',
 '__delattr__',
 '__delitem__',
 '__dir__',
 '__doc__',
 '__eq__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__getitem__',
 '__gt__',
 '__hash__',
 '__iadd__',
 '__imul__',
 '__init__',
 '__init_subclass__',
 '__iter__',
 '__le__',
 '__len__',
 '__lt__',
 '__mul__',
 '__ne__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__reversed__',
 '__rmul__',
 '__setattr__',
 '__setitem__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 'append',
 'clear',
 'copy',
 'count',
 'extend',
 'index',
 'insert',
 'pop',
 'remove',
 'reverse',
 'sort']

# use a 'list comprehension' to find all the list methods
# [DO ACTION TO SOMETHING for THAT SOMETHING in ITERABLE if CONDITION]
['list.' + something for something in dir(list) if not something.startswith('_')]

['list.append',
 'list.clear',
 'list.copy',
 'list.count',
 'list.extend',
 'list.index',
 'list.insert',
 'list.pop',
 'list.remove',
 'list.reverse',
 'list.sort']

[x for x in dir(str) if not x.startswith('_')]

['capitalize',
 'casefold',
 'center',
 'count',
 'encode',
 'endswith',
 'expandtabs',
 'find',
 'format',
 'format_map',
 'index',
 'isalnum',
 'isalpha',
 'isascii',
 'isdecimal',
 'isdigit',
 'isidentifier',
 'islower',
 'isnumeric',
 'isprintable',
 'isspace',
 'istitle',
 'isupper',
 'join',
 'ljust',
 'lower',
 'lstrip',
 'maketrans',
 'partition',
 'replace',
 'rfind',
 'rindex',
 'rjust',
 'rpartition',
 'rsplit',
 'rstrip',
 'split',
 'splitlines',
 'startswith',
 'strip',
 'swapcase',
 'title',
 'translate',
 'upper',
 'zfill']

my_string = 'pandas'
my_string
my_string = my_string.upper()
my_string

'pandas'

'PANDAS'

# useful list methods and the help function
help(list.extend)
help(list.append)
help(list.pop)
help(list.sort)

Help on method_descriptor:

extend(self, iterable, /)
    Extend list by appending elements from the iterable.

Help on method_descriptor:

append(self, object, /)
    Append object to the end of the list.

Help on method_descriptor:

pop(self, index=-1, /)
    Remove and return item at index (default last).
    
    Raises IndexError if list is empty or index is out of range.

Help on method_descriptor:

sort(self, /, *, key=None, reverse=False)
    Stable sort *IN PLACE*.

Tuples¶

# like lists, tuples can hold any object
my_tup = ('p', 'a', 'n', 'd', 'a', 123, 2.3, dir())
# and they can be sliced
my_tup[:5]
# but they are 'immutable', meaning they cannot be altered once they have been defined
# useful if you need to store something that you DO NOT want changed after creation

('p', 'a', 'n', 'd', 'a')

Sets¶

As in maths, a set in Python is an unordered collection with no duplicate elements. Uses of sets:

checking for membership of a set
- you can do this with a list, but it is less efficient
set unions, differences and intersections
when you need a data structure that needs unique values

[x for x in dir(set) if not x.startswith('_')]

['add',
 'clear',
 'copy',
 'difference',
 'difference_update',
 'discard',
 'intersection',
 'intersection_update',
 'isdisjoint',
 'issubset',
 'issuperset',
 'pop',
 'remove',
 'symmetric_difference',
 'symmetric_difference_update',
 'union',
 'update']

stuff_powerlifters_like = {"bench", "squat"}
stuff_weightlifters_like = {"clean and jerk", "snatch", "squat"}
stuff_powerlifters_like
stuff_weightlifters_like
stuff_powerlifters_like.add("deadlift")
stuff_powerlifters_like.add("bench")  # adding an existing element does nothing
# can perform mathematical set operations
stuff_powerlifters_like.intersection(stuff_weightlifters_like)
stuff_powerlifters_like.intersection(stuff_powerlifters_like)

{'bench', 'squat'}

{'clean and jerk', 'snatch', 'squat'}

{'squat'}

{'bench', 'deadlift', 'squat'}

'squat' in stuff_powerlifters_like

True

'foam rolling' in stuff_powerlifters_like

False

Dictionaries¶

A mapping from a unique key to a value (you can think of it like a v-lookup, except the lookup value has to be unique)

ML_models = {'Linear Regression Models': 'Regressors', 'Decision Trees': 'Regressors or Classifiers', 'SVMs':'Regressors or Classifiers',
         'Naive Bayes Models':'Classifiers', 'K Nearest Neighbors Models':'Regressors or Classifiers'}
ML_models

{'Linear Regression Models': 'Regressors',
 'Decision Trees': 'Regressors or Classifiers',
 'SVMs': 'Regressors or Classifiers',
 'Naive Bayes Models': 'Classifiers',
 'K Nearest Neighbors Models': 'Regressors or Classifiers'}

# can access dictionary values using the index notation
ML_models['Naive Bayes Models']
ML_models['K Nearest Neighbors Models']

for model in ML_models:  # can iterate over dictionaries
    print(model, "are", ML_models[model])

'Classifiers'

'Regressors or Classifiers'

Linear Regression Models are Regressors
Decision Trees are Regressors or Classifiers
SVMs are Regressors or Classifiers
Naive Bayes Models are Classifiers
K Nearest Neighbors Models are Regressors or Classifiers

Functions¶

Create reusable code
begins with def func_name(arg_1, arg_2, ...):

# an example of a recursive function
def fibonacci(x=5):  # you can set a default parameter value
    """
    It is good practice to add documentation at the beginning of your function.
    When someone calls help() on your function, they will see the documentation.
    """
    if x <= 1:
        return 1  # the function call ends when a return statement is reached, so an else statement is not needed
    return fibonacci(x - 2) + fibonacci(x - 1)  # if not return statement is specified, None will be returned
    
# list comprehension
[fibonacci(x) for x in range(10)]

# default parameter value
fibonacci()

help(fibonacci)

[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

8

Help on function fibonacci in module __main__:

fibonacci(x=5)
    It is good practice to add documentation at the beginning of your function.
    When someone calls help() on your function, they will see the documentation.

Object Oriented Programming¶

Something to be aware of in Python, is that everything is an object. The full details of object oriented programming are beyond the scope of this training, but there are some things to be aware of:

objects have attributes and methods, accessed with a dot - '.'
- object.attribute - attributes are data about that object, stored in the object
- object.method() - methods are functions associated with an object and as such need a set of brackets to call them object.method(). Like all functions, these methods may take arguments.
classes are types/classes of objects. They can be thought of as the prototype/sketch of a class/type of object. All objects in that class will 'inherit' attributes/methods from that class.
objects are 'instantiated' (brought to life, created), use the 'generator' of a class, which generates an instance of that class

This is all probably a bit confusing, so let's go through some examples.

# everything in Python is an object. For example, strings are objects
my_string = "python"  # this instantiates an instance/object of the class string
my_string_2 = "pandas" # this instantiates another instance/object of the class string

# the below are string attributes and methods. all strings have these.
[x for x in dir(my_string) if not x.startswith("_")]
[x for x in dir(my_string_2) if not x.startswith("_")]
# both the above strings have inherited the same attributes/methods from the overarching string class.

['capitalize',
 'casefold',
 'center',
 'count',
 'encode',
 'endswith',
 'expandtabs',
 'find',
 'format',
 'format_map',
 'index',
 'isalnum',
 'isalpha',
 'isascii',
 'isdecimal',
 'isdigit',
 'isidentifier',
 'islower',
 'isnumeric',
 'isprintable',
 'isspace',
 'istitle',
 'isupper',
 'join',
 'ljust',
 'lower',
 'lstrip',
 'maketrans',
 'partition',
 'replace',
 'rfind',
 'rindex',
 'rjust',
 'rpartition',
 'rsplit',
 'rstrip',
 'split',
 'splitlines',
 'startswith',
 'strip',
 'swapcase',
 'title',
 'translate',
 'upper',
 'zfill']

['capitalize',
 'casefold',
 'center',
 'count',
 'encode',
 'endswith',
 'expandtabs',
 'find',
 'format',
 'format_map',
 'index',
 'isalnum',
 'isalpha',
 'isascii',
 'isdecimal',
 'isdigit',
 'isidentifier',
 'islower',
 'isnumeric',
 'isprintable',
 'isspace',
 'istitle',
 'isupper',
 'join',
 'ljust',
 'lower',
 'lstrip',
 'maketrans',
 'partition',
 'replace',
 'rfind',
 'rindex',
 'rjust',
 'rpartition',
 'rsplit',
 'rstrip',
 'split',
 'splitlines',
 'startswith',
 'strip',
 'swapcase',
 'title',
 'translate',
 'upper',
 'zfill']

# we can access string methods like this
my_string.upper()  # returns the string with everything made into upper case
my_string_2.isdigit()  # returns whether the string is a digit

'PYTHON'

False

# lists are objects as well
higher_level_languages = ['Python', 'Java Script']  # this creates an instance/object of the class string
lower_level_languages = ['C', 'x86 Assembly']
# all lists have the below attributes/methods, including the ones we just made
[x for x in dir(higher_level_languages) if not x.startswith("_")]

['append',
 'clear',
 'copy',
 'count',
 'extend',
 'index',
 'insert',
 'pop',
 'remove',
 'reverse',
 'sort']

higher_level_languages.extend(lower_level_languages)
higher_level_languages

['Python', 'Java Script', 'C', 'x86 Assembly']

You can also create your own classes as well!

We will see next week that machine learning models are objects, and that a basic understanding of how objects work will be needed to use them.

Modules¶

Modules are python scripts
You import modules to access pre-written functions, variables and classes
Similar to R's library() function
For data science, you will need:
- import pandas as pd
- import numpy as np
- import sklearn
You can write your own modules (it is just a .py file) and then import it into your code if it is in the same directory with import your_module

import pandas as pd  # import a module under an alias
from matplotlib import pyplot as plt # you can import specific things from a module with this syntax

# use a dot '.' to access things from that module, in this case the DataFrame class
my_df = pd.DataFrame({'x':[1, 2, 3, 4, 5], 'y':[2, 4, 6, 8, 10]})
# the DataFrame class has a plot method.
my_df.plot()
plt.show()

from math import sqrt  # import a specific thing (sqrt function) from a module. you can now call sqrt().
import math  # import the whole module, you will need to specify what you are calling with math.THING.

print("sqrt(4) is", sqrt(4))
print("math.pi is", math.pi)

<matplotlib.axes._subplots.AxesSubplot at 0x168c35eb4a8>

<Figure size 640x480 with 1 Axes>

sqrt(4) is 2.0
math.pi is 3.141592653589793