Introduction to Statistical Learning Using Python¶
Table of Contents¶
- LAB1: Introduction to Python
- Basic Elements of Python
- Working with Lists
- IF Statement
- FOR and While Loops
- Functions
- LAB2: Dealing With Data
- Importing and Manipulating Data
- Dealing with Missing Values
- Summary Statistics
- Understanding Data
- Data Interpolation
- Visualizing Data
- LAB3: Real World Data
- Logistic Regression
- Linear and Quadratic Discriminant Analysis
- Classification Trees, Random Forest
Getting Started with Pandas¶
Panda is a library, which contains high-level data structure that designed to make data analysis fast and easy in Python.¶
- Main two type structures:
- Series
- DataFrame
Series¶
- Series is a one dimensional array-like object containing an array of data and an associated array of data labels, called
index
In [2]:
from pandas import Series, DataFrame
import pandas as pd
import os
In [4]:
obj = Series ([4,7,-5,3])
In [5]:
print(obj)
In [6]:
obj.values
Out[6]:
In [7]:
obj.index
Out[7]:
Selecting Value by index¶
In [8]:
obj[1]
Out[8]:
In [10]:
obj[[1,3]]
Out[10]:
Operations with Series¶
In [11]:
obj[obj>0]
Out[11]:
In [14]:
obj[obj>3]
Out[14]:
In [15]:
obj*2
Out[15]:
In [19]:
4 in obj.values
Out[19]:
DataFrame¶
- A DataFrame represents spreadsheet -like data structure containing an ordered collection of columns, each of which can be a different value type (numeric, string, etc.)
In [56]:
data= {
'Student': ["Muhammad Abdullah","Kayli Abernathy","Jacqueline Alvarez",
"Daniel Chavez","Aramayis Dallakyan"],
'UIN' : [925024924,925024924,925024924,915140123,925124214],
'Graduation_Year' : [2017,2016,2018,2020,2018]
}
frame = DataFrame(data)
In [57]:
frame
Out[57]:
Seperating First and Last Name¶
In [61]:
frame.loc[frame['Student'].index, 'Last_name'] = frame['Student'].str.split().str[-1]
frame.loc[frame['Student'].index, 'First_name'] = frame['Student'].str.split().str[0]
In [62]:
frame
Out[62]:
In [72]:
frame.drop(["Student"],axis=1,inplace= True)
In [75]:
frame["First_name"]
Out[75]:
In [77]:
frame[frame["First_name"]=="Aramayis"]
Out[77]:
Inserting Data¶
- Define Directory
In [335]:
os.chdir("C:\Users\dallakyan1988\Documents\Python_Presentation")
- Import Data using
read_csv()function
read_csv(filepath, sep=', ', header='infer', names=None, index_col=None, usecols=None,skiprows=None, nrows=None, na_values=None)
In [336]:
data1 = pd.read_csv("Orange_data.csv",sep = ",",na_values=' ')
data2 = pd.read_csv("Income.csv",sep = ",")
In [337]:
data2.set_index(pd.DatetimeIndex(data2['Date']),inplace = True)
In [338]:
data1.head()
Out[338]:
In [339]:
data2.head()
Out[339]:
In [340]:
del data2.index.name
- It is always good idea to check data types of variables
In [341]:
data1.dtypes
Out[341]:
In [342]:
data2.dtypes
Out[342]:
- Dealing with missing values
- Function to count missing values
In [343]:
def missing_count(data):
missing = pd.DataFrame(data.isnull().sum(),columns=['Total_missing'])
return missing
In [344]:
missing_count(data1)
Out[344]:
In [345]:
missing_count(data2)
Out[345]:
In [346]:
data1[data1["Per_Capita_Cons"].isnull()==True]
Out[346]:
- Possible Solutions
- Drop all
NANvalues (Not always the best)
- Drop all
In [347]:
cleaned = data1.dropna()
In [348]:
missing_count(cleaned)
Out[348]:
NOTE:You have several options for dropping. You can drop if¶
* all raw is **`NAN`**
* all column is **`NAN`**
* or by threshholdIn [349]:
from numpy import nan as NA
example = DataFrame([[2.,6.5,3.],[1.,NA,NA],[NA,NA,NA],[NA,4.,4.]])
In [350]:
example
Out[350]:
In [351]:
example.dropna()
Out[351]:
In [352]:
example.dropna(how="all")
Out[352]:
In [353]:
example.dropna(thresh=2)
Out[353]:
Method 2
- Fill
NANvalues with mean
In [354]:
last_item_filled=data1.fillna(method='ffill')
In [355]:
mean_filled=data1.fillna(data1['Per_Capita_Cons'].mean())
In [356]:
data1[data1["Per_Capita_Cons"].isnull()==True]
Out[356]:
In [357]:
last_item_filled.ix[data1[data1["Per_Capita_Cons"].isnull()==True].index,:]
Out[357]:
In [358]:
mean_filled.ix[data1[data1["Per_Capita_Cons"].isnull()==True].index,:]
Out[358]:
Method 3
- Interpolation (Not usable for all variables) Interpolation is method to create a function based on fixed data points which can be evaluated anywhere within the domain defined by the given data using linear, Cubic spline interpolation.
In [359]:
import numpy as np
import matplotlib.pyplot as plt
from scipy import interpolate
from scipy.interpolate import Rbf,InterpolatedUnivariateSpline
In [360]:
x = np.arange(0, 2*np.pi+np.pi/4, 2*np.pi/8)
y = 0.8*np.power(x,5)- 3*np.power(x,2)
tck = interpolate.splrep(x, y, s=0)
xnew = np.arange(0, 2*np.pi, np.pi/50)
ynew = interpolate.splev(xnew, tck, der=0)
plt.figure()
plt.plot(x, y, 'x', xnew, ynew, xnew, 0.8*np.power(xnew,5)- 3* np.power(xnew,2), x, y, 'b')
plt.legend(['Linear', 'Cubic Spline', 'True','RBF'])
plt.axis([-0.05, 3.05, -4.05, 10])
plt.title('Cubic-spline interpolation')
plt.show()
Linear and Cubic Spline for Orange data¶
In [361]:
linear_interpolated= data1['Per_Capita_Cons'].interpolate(method='linear')
spline_interpolated= data1['Per_Capita_Cons'].interpolate(method='spline',order=3)
In [362]:
plt.figure
plt.plot(xi, data1["Per_Capita_Cons"],xi, mean_filled["Per_Capita_Cons"],'g',xi,linear_interpolated,'b',xi,spline_interpolated,'r')
plt.legend(["Real",'mean_filled','Linear',"Spline"])
plt.show()
In [363]:
data1['Per_Capita_Cons'] = spline_interpolated
Notice that Income data and Consumption data have different time frequences.¶
- We want to match the frequences of Income and Consumption
In [365]:
data2.head()
Out[365]:
In [366]:
data2.tail()
Out[366]:
- To start interpolation from Quartely frequency to Monthly frequency, first we need to proper index
In [367]:
from pandas import datetime
period = 7*12 + 9-2
index = pd.date_range('3/1/1995', periods=period, freq='m')
In [368]:
upsampled = data2["Per_Capita_Income"].resample('M')
interpolated = upsampled.interpolate(method='spline', order=3)
print(interpolated.head(10))
interpolated.plot()
plt.show()
In [369]:
interpolated = upsampled.interpolate(method='linear', order=3)
print(interpolated.head(10))
interpolated.plot()
plt.show()
In [370]:
interpolated=pd.DataFrame(interpolated)
interpolated.head()
Out[370]:
Now we want to merge Interpolated income data with Orange consumption data¶
In [371]:
data1.head()
Out[371]:
In [372]:
period = 7*12 + 9
data1index = pd.date_range('1/1/1995', periods=period, freq='m')
In [373]:
data1.set_index(data1index,inplace = True)
In [374]:
data1.head()
Out[374]:
Merge by index¶
In [414]:
Merged =DataFrame(pd.merge(data1,interpolated,left_index=True,right_index=True))
In [415]:
Merged.head()
Out[415]:
In [428]:
Merged=Merged.drop(["Year","Month "],axis=1)
In [429]:
Merged.head()
Out[429]:
In [412]:
Merged.columns
Out[412]:
Other examples of Merging¶
In [379]:
df1 = pd.DataFrame({'A': np.random.uniform(size=4),
'B': np.random.uniform(size=4),
'C': np.random.uniform(size=4),
'D': np.random.uniform(size=4)},
index=[0, 1, 2, 3])
df2 = pd.DataFrame({'A': np.random.normal(size=4),
'B': np.random.normal(size=4),
'C': np.random.normal(size=4),
'D': np.random.normal(size=4)},
index=[4, 5, 6, 7])
df3 = pd.DataFrame({'A':np.random.uniform(size=4),
'B': np.random.uniform(size=4),
'C': np.random.uniform(size=4),
'D': np.random.uniform(size=4)},
index=[8, 9, 10, 11])
In [382]:
df1
Out[382]:
In [383]:
df2
Out[383]:
In [384]:
df3
Out[384]:
In [385]:
frames = [df1, df2, df3]
result = pd.concat(frames)
In [386]:
result
Out[386]:
In [387]:
df4 = pd.DataFrame({'B': np.random.uniform(size=4),
'D':np.random.normal(size=4),
'F': np.random.normal(size=4)},
index=[2, 3, 6, 7])
In [390]:
df4
Out[390]:
In [392]:
df1
Out[392]:
In [388]:
result = pd.concat([df1, df4], axis=1)
In [389]:
result
Out[389]:
In [393]:
result = df1.append(df2)
In [394]:
result
Out[394]:
- Merging based on key
In [396]:
left = pd.DataFrame({'key': ['A', 'G', 'E', 'C'],
'A': np.random.normal(size=4),
'B': np.random.normal(size=4)})
right = pd.DataFrame({'key': ['A', 'G', 'E', 'C'],
'C': np.random.normal(size=4),
'D': np.random.normal(size=4)})
result = pd.merge(left, right, on='key')
In [400]:
left
Out[400]:
In [401]:
right
Out[401]:
In [402]:
result
Out[402]:
Descriptive Statistics¶
In [430]:
Merged.describe()
Out[430]:
Correlation and Covariance Matrix¶
In [440]:
returns= Merged.pct_change()
In [441]:
returns
Out[441]:
In [452]:
plt.figure()
plt.plot(returns["Real_Price"].cumsum(),"b")
plt.xlabel("Time")
plt.ylabel("Percentage change")
plt.title("Cumulatice Percentage Change")
plt.show()
In [450]:
Merged.corr()
Out[450]:
In [444]:
Merged.cov()
Out[444]:
In [460]:
Merged["Per_Capita_Cons"].hist(bins=20)
plt.show()
In [466]:
Merged["Per_Capita_Cons"].hist(bins=20,normed= True)
Merged["Per_Capita_Cons"].plot(kind="kde")
plt.show()
In [469]:
comp1=np.random.normal(Merged["Per_Capita_Cons"].mean(),Merged["Per_Capita_Cons"].std(),size=200)
values = Series(comp1)
In [473]:
Merged["Per_Capita_Cons"].hist(bins=20,normed= True)
values.plot(kind="kde")
plt.show()
In [481]:
pd.scatter_matrix(Merged, diagonal="kde",alpha=0.5)
plt.show()
