**Python** is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric **Python** packages. **Pandas** is one of those packages and makes importing and analyzing data much easier.

Pandas aims to integrate the functionality of NumPy and matplotlib to give you a convenient tool for data analytics and visualization. Besides the integration, it also makes the usage far more better.

In this blog, I’ll give you a list of useful pandas snippets that can be reused over and over again. These will definitely save you some time that you may otherwise need to skim through the comprehensive Pandas docs.

The data structures in Pandas are capable of holding elements of *any* type: Series, DataFrame.

### Series

A one-dimensional object that can hold any data type such as integers, floats, and strings

A Series object can be created of different values. Series can be remembered similar to a Python list.

In the below example, **NaN **is NumPy’s nan symbol which tells us that the element is not a number but it can be used as one numerical type pointing out to be not a number. The type of series is an **object** because the series has mixed contents of strings and numbers.

CODE: https://gist.github.com/RonakMutha12/50186687052053d506cfd31e638bb6ad.js

Now if we use only numerical values, we get the basic NumPy dtype - **float** for our series.

CODE: https://gist.github.com/RonakMutha12/5f2f20c6b716982f3724cebe49719150.js

### DataFrame

A two-dimensional labeled data structure where columns can be of different types.

Each column in a Pandas DataFrame represents a Series object in memory.

In order to convert a certain Python object (dictionary, lists, etc) to a DataFrame, it is extremely easy. From the python dictionaries, the **keys** map to Column names while **values** correspond to a list of column values.

CODE: https://gist.github.com/RonakMutha12/bc27bf70d245468ea25f9c47828fb2a9.js

## Reading CSV files

Pandas can work with various file types while reading any file you need to remember.

pd.read_filetype()

Now you will have to only replace “filetype” with the actual type of the file, like csv or excel. You will have to give the path of the file inside the parenthesis as the first argument. You can also pass in different arguments that relate to opening the file. *(Reading a **csv** file? See **this**)*

CODE: https://gist.github.com/RonakMutha12/dc943d96b1faa23079bd56b9a3a1e466.js

## Accessing Columns and Rows

DataFrame comprises of three sub-components, the **index**, **columns**, and the **data ***(also known as **values**)*.

The **index **represents a sequence of values. In the DataFrame, it always on the left side. Values in an index are in **bold** font. Each individual value of the index is called a **label**. Index is like positions while the labels are values at that particular index. Sometimes the index is also referred to as **row labels**. In all the examples below, the **labels** and **indexes** are the same and are just integers beginning from 0 up to n-1, where n is the number of rows in the table.

Selecting rows is done using loc and iloc

- loc gets rows (or columns) with particular
from the index. Raises KeyError when the items are not found.*labels* - iloc gets rows (or columns) at particular
(so it only takes integers). Raises IndexError if a requested indexer is out-of-bounds.*positions*/*index*

CODE: https://gist.github.com/RonakMutha12/56dca611979ce2fe4289cb397703ee23.js

**Accessing the data using column names**

Pandas takes an extra step and allows us to access data through labels in DataFrames.

CODE: https://gist.github.com/RonakMutha12/14996bc9313e1118d94de91a972366d8.js

In Pandas, selecting data is very easy and similar to accessing an element from a dictionary or a list.

You can select a column (df[col_name]) and it will return column with label col_name as a Series, because rows and columns are stored as Series in a DataFrame, If you need to access more columns (df[[col_name_1, col_name_2]]) and it returns columns as a new DataFrame.

## Filtering DataFrames with Conditional Logic

Let’s say we want all the companies with the vertical as B2B, the logic would be:

CODE: https://gist.github.com/RonakMutha12/35479b1744101f5354d875258bf7606f.js

If we want the companies for the year 2009, we would use:

CODE: https://gist.github.com/RonakMutha12/bdd18f989a34049ff4e878e5f3fb01ed.js

Need to combine them both? Here’s how you would do it:

CODE: https://gist.github.com/RonakMutha12/8e4bc1ca7ed1aa6d092e6ade726765e9.js

[caption id="" align="alignnone" width="663.0"]

Get all companies with vertical as B2B for the year 2009 [/caption]

## Sort and Groupby

### Sorting

Sort values by a certain column in ascending order by using:

CODE: https://gist.github.com/RonakMutha12/8e703ce45b2a1c560657554f2262a285.js

CODE: https://gist.github.com/RonakMutha12/8c866cffc9210b18d3123ca6fa569edd.js

Furthermore, it’s also possible to sort values by multiple columns with different orders. colname_1 is being sorted in ascending order and colname_2 in descending order by using:

CODE: https://gist.github.com/RonakMutha12/f543f5af9037368fd5efc7924557e47f.js

### Grouping

This operation involves 3 steps; splitting of the data, applying a function on each of the group, and finally combining the results into a data structure. This can be used to group large amounts of data and compute operations on these groups.

df.groupby(colname) returns a groupby object for values from one column while df.groupby([col1,col2]) returns a groupby object for values from multiple columns.

### Data Cleansing

Data cleaning is a very important step in data analysis.

**Checking missing values in the data**

Check null values in the DataFrame by using:

CODE: https://gist.github.com/RonakMutha12/21acb6730b65b6e8128860ba159acad0.js

This returns a boolean array (an array of *true* for missing values and *false* for non-missing values).

CODE: https://gist.github.com/RonakMutha12/1a044982af9166e485022196121d39d0.js

*Check non null values in the DataFrame *using pd.notnull(). It returns a boolean array, exactly converse of df.notnull()

**Removing Empty Values**

Dropping empty values can be done easily by using:

CODE: https://gist.github.com/RonakMutha12/55ee3551f57e794ddba2f2e2aaacd2f2.js

This drops the rows having empty values or df.dropna(axis=1) to drop the columns.

Also, if you wish to fill the missing values with other values, use df.fillna(x). This fills all the missing values with the value x (here you can put any value that you want) or s.fillna(s.mean()) which replaces null values with the mean (** mean** can be replaced with any function from the arithmetic section).

**Operations on Complete Rows, Columns, or Even All Data**

CODE: https://gist.github.com/RonakMutha12/4630132a897284f021449397e4b809ca.js

The .map() operation applies a function to each element of a column.

.apply() applies a function to columns. Use .apply(axis=1) to do it on the rows.

**Iterating over rows**

Very similar to iterating any of the python primitive types such as *list, tuples, dictionaries.*

CODE: https://gist.github.com/RonakMutha12/d3d1fcde18434262c7db5faa3675ac3c.js

The .iterrows() loops 2 variables together i.e, the index of the row and the row itself, variable **i **is the index and variable **row **is the row in the code above.

### Tips & Tricks

Using **ufuncs**** ***(also known as Universal Functions). *Python has the .apply() which applies a function to columns/rows. Similarly, **Ufuncs** can be used while preprocessing. What is the difference between ** ufuncs** and

**?**

*.apply()*Ufuncs is a numpy library, implemented in C which is highly efficient *(ufuncs are around 10 times faster)*.

A list of common *Ufuncs**:*

isinf: Element-wise checks for positive or negative infinity.

isnan: Element-wise checks for NaN and returns result as a boolean array.

isnat: Element-wise checks for NaT (not time) and returns result as a boolean array.

trunc: Return the truncated value of the input, element-wise.

.dt commands: Element-wise processing for date objects.

## High-Performance Pandas

Pandas performs various vectorized/broadcasted operations and grouping-type operations. These operations are efficient and effective. They often rely on the creation of temporary intermediate objects, which can cause undue overhead in computational time and memory use.

As of *version 0.13*, Pandas included tools that allow us to directly access C-speed operations without costly allocation of intermediate arrays. There are two functions, eval() and query().

DataFrame.eval() for efficient operations:

CODE: https://gist.github.com/RonakMutha12/efbf974a4b518bee73e5743db0116d6e.js

To compute the sum of *df1, df2, df3, and df4* *DataFrames* using the typical Pandas approach, we can just write the sum:

CODE: https://gist.github.com/RonakMutha12/6bb6234b0415d0187ca2bc2331bbcb5f.js

A better and optimized approach for the same operation can be computed via pd.eval():

CODE: https://gist.github.com/RonakMutha12/33821adb73d327c43af77181e3c6c290.js

%timeit — Measure execution time of small code snippets.

The eval()

expression is about 50% faster *(it also consumes mush less memory).*

And it performs the same result:

CODE: https://gist.github.com/RonakMutha12/775d8804c9ce8564036ce0cfee2a6194.js

*np.allclose()** is a numpy function which returns True if two arrays are element-wise equal within a tolerance.*

### Column-Wise & Assignment Operations Using df.eval()

Normal expression to split the first character of a column and assigning it to the same column can be done by using:

CODE: https://gist.github.com/RonakMutha12/5a0dad063fd87a692476387dab4955e4.js

By using df.eval(), same expression can be performed much faster:

CODE: https://gist.github.com/RonakMutha12/8cd155f946c4641bc68bf5b2afbafbd2.js

DataFrame.query()

for efficient operations:

Similar to performing filtering operations with conditional logic, to filter rows with vertical as **B2B** and year as **2009, **we do it by using:

CODE: https://gist.github.com/RonakMutha12/b5279bf58b67a1a50529fb7f7c25519d.js

With .query() the same filtering can be performed about 50% faster.

CODE: https://gist.github.com/RonakMutha12/4b323d217bea1d723f6182d1156b7f61.js

*When to use eval() and query()? *

Two aspects: *computation time* and *memory us*age.

** Memory usage:** Every operation which involves NumPy/Pandas DataFrames results into implicit creation of temporary variables. In such cases, if the memory usage of these temporary variables is greater, using

*eval() and query()*is an appropriate choice to reduce the memory usage.

*Computation time:** *Traditional method of performing NumPy/Pandas operations is faster for smaller arrays! The real benefit of eval()/query() is achieved mainly because of the saved memory, and also because of the cleaner syntax they offer.

# Conclusion

Pandas is a really powerful and fun library for data manipulation/analysis. It comes with easy syntax and fast operations. The blog highlights the most used pandas implementation and optimizations. Best way to master your skills over pandas is to use real datasets, beginning with Kaggle kernels to learning how to use pandas for data analysis. Check out more on **real time text classification using Kafka and Scikit-learn** and **explanatory vs. predictive models in machine learning here****.**