%md 
# Apache Spark on Databricks for Data Scientists

** Welcome to Databricks! **

This notebook intended to be the second step in your process to learn more about how to best use Apache Spark and Databricks together. We'll be walking through the core concepts, the fundamental abstractions, and some machine learning tools from the perspective of as data scientist! As a data scientist you're likely querying data from a variety of sources, joining together various tables and leveraging that information to build out statistical models. Databricks and Apache Spark make this process simple by allowing you to connect to a variety of sources and work with the data in one unified environment, then deploy your work to production quickly. Additionally because Databricks and Apache Spark support multiple languages, it's straightforward to do a lot of the analysis using the languages you know and the tools that you're likely already familiar with.

First, it's worth defining Databricks. Databricks is a managed platform for running Apache Spark - that means that you do not have to learn complex cluster management concepts nor perform tedious maintenance tasks to take advantage of Apache Spark. Databricks also provides a host of features to help its users be more productive with Spark. It’s a point and click platform for those that prefer a user interface like data scientists or data analysts. This UI is accompanied by a sophisticated API for those that want to automate jobs and aspects of their data workloads. To meet the needs of enterprises, Databricks also includes features such as role-based access control and other intelligent optimizations that not only improve usability for users but also reduce costs and complexity for administrators.

** The Gentle Introduction Series **

This notebook is a part of a series of notebooks aimed to get you up to speed with the basics of Spark quickly. This notebook is best suited for those that have very little or no experience with Spark. The series also serves as a strong review for those that have some experience with Apache Spark but aren't as familiar with some of the more sophisticated tools like UDF creation and machine learning pipelines. The other notebooks in this series are:

- [A Gentle Introduction to Apache Spark on Databricks](https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/346304/2168141618055043/484361/latest.html)
- [Apache Spark on Databricks for Data Scientists](https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/346304/2168141618055194/484361/latest.html)
- [Apache Spark on Databricks for Data Engineers](https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/346304/2168141618055109/484361/latest.html)

## Tutorial Overview

This tutorial centers around a core idea that we hope to explore:

**The number of farmer's markets in a given zip code can be predicted from the income and taxes paid in a given area.**

It seems plausible that areas with higher income have more farmer's markets simply because there is more of a market for those goods. Of course there are many potential holes in this idea, but that's part of the desire to test it :). This tutorial will explore our process for discovering whether or not we can accurately predict the number of farmer's markets! *It is worth mentioning that this notebook is not intended to be a academically rigorous, but simply a good example of the work a data scientist might be performing using Apache Spark and Databricks.*

## The Data

![img](http://training.databricks.com/databricks_guide/USDA_logo.png)

The first of the two datasets that we will be working with is the **Farmers Markets Directory and Geographic Data**. This dataset contains information on the longitude and latitude, state, address, name, and zip code of Farmers Markets in the United States. The raw data is published by the Department of Agriculture. The version on the data that is found in Databricks (and is used in this tutorial) was updated by the Department of Agriculture on Dec 01, 2015.

![img](http://training.databricks.com/databricks_guide/irs-logo.jpg)

The second we will be working with is the **SOI Tax Stats - Individual Income Tax Statistics - ZIP Code Data (SOI)**. This study provides detailed tabulations of individual income tax return data at the state and ZIP code level and is provided by the IRS. This repository only has a sample of the data: 2013 and includes "AGI". The ZIP Code data show selected income and tax items classified by State, ZIP Code, and size of adjusted gross income. Data are based on individual income tax returns filed with the IRS and are available for Tax Years 1998, 2001, 2004 through 2013. The data include items, such as:

- Number of returns, which approximates the number of households
- Number of personal exemptions, which approximates the population
- Adjusted gross income
- Wages and salaries
- Dividends before exclusion
- Interest received

You can learn more about the two datasets on data.gov:

- [Farmer's Market Data](http://catalog.data.gov/dataset/farmers-markets-geographic-data/resource/cca1cc8a-9670-4a27-a8c7-0c0180459bef)
- [Zip Code Data](http://catalog.data.gov/dataset/zip-code-data)

### Getting the Data

As a data scientist, your data is likely going to be living in a place like S3 or Redshift. Apache Spark provides simple and easy connectors to these data sources and Databricks provides simple demonstrations of how to use them. Just search in the Databricks guide (use the `?` at the top left) to see if your data source is available. For the purposes of this tutorial, our files are already available on S3 via `dbfs` or the Databricks file system. [While you're free to upload the csvs made available on data.gov as a table](https://docs.cloud.databricks.com/docs/latest/databricks_guide/index.html#03%20Accessing%20Data/1%20Data%20Import%20UI.html) you can also (more easily) access this data via the `/databricks-datasets` directory which is a repository of public, Databricks-hosted datasets that is available on all Databricks accounts.

%md First things first! We've got to read in our data. This data is located in in csv files so we'll use the [spark-csv](https://github.com/databricks/spark-csv) package to do this. In Databricks, it's as simple as specifying the format of the csv file and loading it in as a DataFrame. In Apache Spark 2.0, you do not need to use the `spark-csv` package and can just read the data in directly.

As a data scientist, you've likely come across DataFrames either in R, or python and pandas. Apache Spark DataFrames do not stray too much from this abstraction except that they are distributed across a cluster of machines instead of existing on one machine (as is typically the case with R or pandas). If you're not quite familiar with this, it might be worth reading through [a Gentle Introduction to Apache Spark on Databricks](https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/346304/2168141618055043/484361/latest.html).

%md Now that we've loaded in the data - let's go ahead and register the DataFrames as SparkSQL tables. 

While this might seem unnecessary, what it's going to allow you as a data scientist to do is to leverage your SQL skills immediately to manipulate the data. Some people prefer working directly with DataFrames while others prefer working in SparkSQL directly. Whatever the case, they take advantage of the same tungsten optimizations under the hood.

%md You can see that we are using the `registerTempTable`/`createOrReplaceTempView` method call to create this table. The lifetime of this temporary table is tied to the Spark/SparkSQL Context that was used to create this DataFrame. This means when you shutdown the SQLContext that is associated with a cluster (like when you shutdown the cluster) then the temporary table will disappear as well. In Databricks, Spark and SparkSQL Contexts are associated 1 to 1 with clusters.

## Running SQL Commands

As we progress through the notebook, you'll notice that all SQL cells are prefaced with `%sql`. This tells the Databricks environment that you'd like to execute a SQL command. You can do the same with python and R as you will see in other tutorials and parts of the documentation.

In order to list the tables, we can show them very easily by simply executing `show tables`. You'll see that this also provides information about their lifetime (and whether or not they are temporary or not).

%md Now that we've loaded in the data, let's take a quick look at it. The `display` command makes it easy to quickly display a subset of the table that we have. This operates directly on our `DataFrame`.