import numpy as np55 Numpy and Pandas
If you know a bit of linear algebra, you know how some computations are more easily expressed as vectors or matrices. The NumPy library provides a multidimensional array object that lets you do that. You already installed numpy when the beginning of the course, so all you have to do is import it. The cell below imports the numpy module and binds it to the name np. This is the way every one imports it and saves you a view keystrokes every time you need to use it.
Arrays
numbers = [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
a = np.array(numbers)
aAll the linear algebra rules apply to arrays:
You may see this behavior referred to as broadcasting, but that is just the programming term for linear algebra way of doing computations.
b = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
print(a + b)
print(a * b)
print(a + 10)Arrays also come with a lot of useful methods like:
a.mean()np.float64 is Numpy’s own float type, which works the same as the regular float.
You can make a arrays with any number of dimensions. A two-dimensional array corresponds to a matrix. You already know how to represent a matrix as a list of lists. To make a matrix, just turn that into an array:
list_of_lists = [[1, 2, 3], [1, 2, 3], [1, 2, 3]]
matrix = np.array(list_of_lists)
matrixYou can access elements with square brackets just like you do with lists:
list_of_lists[1][1]matrix[1]matrix[1][1]For computational efficiency, numpy arrays also lets you specify indices for each dimension in a single set of square brackets:
matrix[1]matrix[1, 1]Many of the array methods can be used both on the entire array, but also separately for the rows or columns:
matrix.sum()Pass axis=0 to get the row sums:
matrix.mean(axis=0)Pass axis=1 to get the column sums:
matrix.mean(axis=1)The Pandas library
Run this cell to download an example data set to a penguins.csv file next to this notebook. Then open it to see what the contents look like and close it again (do not change it).
import seaborn as sns
sns.load_dataset('penguins').to_csv('penguins.csv', index=False)As you can see, penguins.csv is a table with a header line and lots of rows. The values in each row are separated by a comma, which is why the file name ends with .csv. CSV is short for comma-separated values. This kind of table format is one of the most common ways of storing data. Pandas is a rich and powerful library that lets you do all kinds of manipulation and advanced data analysis on such table data. This chapter shows introduces you to the basics, but if/when you need to, you can find all the information you need in the Pandas user guide.
You already installed pandas when the beginning of the course, so all you have to do is import it. The cell below imports the pandas module and binds it to the name pd. This is the way every one imports it and saves you a view keystrokes every time you need to use it.
import pandas as pdIf you are not convinced, you can use type to check which kind it is:
type(pd)DataFrames
You can read in the table data in penguins.csv using the read_csv function in the pandas module (available as the pd name). Lets do that and store it in a variable called penguins:
penguins = pd.read_csv('penguins.csv')Pandas read_csv creates DataFrame object. DataFrame is the main object of Pandas and its way of representing table data.
type(penguins)If you put a dataframe on the last (or only) line in a cell, it display it nicely. If it has many rows, it will only show the first and last five rows.
penguinsI see just the first or last rows, you can use the head and tail methods:
penguins.head()penguins.tail()Exercise 55-1
Look at the DataFrame documentation page and find the method you need to sort the rows of a dataframe. Then sort the penguins data by “flipper_length_mm”.
Extracting data
To access a column, you can use the familiar square brackets that we also use with strings, lists and dictionaries:
bill_depth = penguins['bill_depth_mm']
bill_depthYou may see other people access the columns as attributes (like penguins.bill_depth_mm), but we will stick to the bracket notation to keep it simple.
Columns is has their own type, Series.
type(bill_depth)Its data is just a numpy array, but on top of that it packs a lot of useful methods for computing statistics like the mean or for manipulating like sort_values:
bill_depth.mean()np.float64 is Numpy’s own float type, which works the same as the regular float.
You can see all the other methods on the Series documentation page.
Indexing
You may have noticed the numbers in the far left column that does not have a name. That is the row index that you need to access individual rows. Dataframes have a loc attribute that you can use for accessing data in all kinds of ways. To get a row, you just give it the row index in square brackets:
row = penguins.loc[4]
type(row)The row is returned as a Series indexed by the column names:
rowSo you can access the values by passing column names to loc:
row.loc['island']You an get the same single dataframe cell by passing both a row and a column index value to loc:
penguins.loc[4, 'island']You can also access data using slices or lists of row and column values index values:
penguins.loc[40:45, ['island', 'body_mass_g']]penguins.loc[[7,153,257], ['sex', 'species']]Rather than providing row index values, you can provide a series with a True or False for each row. Since Series support boadcasting the same way arrays do, statements like this produce such True/False series:
is_adelie = penguins['species'] == 'Adelie'
is_adeliePassing that to penguins.loc gives you all the rows where the “species” column as the value "Adelie":
penguins.loc[is_adelie]Exercise 55-2
Extract the rows where values in the “body_mass_g” column is larger than 4500.
Exercise 55-3
Extract the rows where values in the “body_mass_g” column is larger than 4500 and the the values in the “species” columns is "Adelie". Hint: You can use the loc twice to achieve this.
Exercise 55-4
Now that you know the basics of extracting data, try to ask the AI to give you a brief account of how the DataFrame.loc works. Then cross-check its explanation with the online documentation.
Build DataFrame from data
Sometimes the table data is not something your read in from a file, but data you are building inside your notebook. In that case, you can easily turn your data into a DataFrame. You create a DataFrame by either specifying its columns or its rows. To create it from columns you first make a dictionary where keys are column names and the values are lists with column values and then pass that dictionary as data to DataFrame():
columns_dict = {'seq_id': ['A', 'B', 'C'], 'gc_content': [0.452, 0.421, 0.421]}
gc_data = pd.DataFrame(columns_dict)
gc_dataYou can also make a DataFrame from rows, you first make a list of lists and then pass that as argument to the the from_records method of DataFrame:
row_records = [('A', 0.452), ('B', 0.421), ('C', 0.421)]
gc_data = pd.DataFrame().from_records(row_records, columns=['seq_id', 'gc_content'])
gc_dataIf you want to save your gc_data as a CSV file, you can save it to a file like this:
gc_data.to_csv('gc_data.csv')Grouping
penguins.groupby('island').mean(numeric_only=True)Exercise 55-5
Extract the rows where values in the “body_mass_g” column is larger than 4500.
Much more to learn as you need it
This is a small chapter about a very large module, and a lot of powerful techniques have been left out. You can do pretty much anything you might want to do with tables. To get a taste, you can browse 10 minutes to pandas.