%md # Deep Learning Pipelines for Apache Spark - Release 1.2.0

Deep Learning Pipelines is a new library published by Databricks to provide high-level APIs for scalable deep learning model application and transfer learning via integration of popular deep learning libraries with MLlib Pipelines and Spark SQL. For an overview and the philosophy behind the library, check out the Databricks [blog post](https://databricks.com/blog/2017/06/06/databricks-vision-simplify-large-scale-deep-learning.html). This notebook parallels the [Deep Learning Pipelines README](https://github.com/databricks/spark-deep-learning), detailing usage examples with additional tips for getting started with the library on Databricks.

%md ## Cluster set-up

Deep Learning Pipelines is available as a Spark Package. To use it on your cluster, create a new library with the Source option "Maven Coordinate", using "Search Spark Packages and Maven Central" to find "spark-deep-learning". Then [attach the library to a cluster](https://docs.databricks.com/user-guide/libraries.html). 

**Note**: 
* This notebook works with spark-deep-learning release `1.2.0-spark2.3-s_2.11`. Please refer to the project's [github page](https://github.com/databricks/spark-deep-learning) for the latest examples and docs.
* To run this notebook, also create and attach the following libraries via PyPI: `tensorflow==1.10.0`, `keras==2.2.2`, `h5py`.

Deep Learning Pipelines 1.2.0 is compatible with Spark versions 2.3 or higher and works with any instance type (CPU or GPU).

%md ## What's new in Deep Learning Pipelines 1.0.0

* Use Spark's native image reader (similar implementation in Deep Learning Pipelines has been removed).
* KerasImageFileEstimator can be used for model tuning with Spark MLlib (e.g. CrossValidator).
* Full Python & Scala support for transfer learning on images (DeepImageFeaturizer and DeepImagePredictor): InceptionV3, Xception, ResNet50, VGG16, and VGG19 models.

%md ## In this notebook

Deep Learning Pipelines provides a suite of tools around deep learning. The tools can be categorized as

Working with image data:
* **Loading images** natively in Spark DataFrames
* **Transfer learning**, a super quick way to leverage deep learning
* [New in 1.0.0] **Distributed hyperparameter tuning** via Spark MLlib Pipelines
* **Applying deep learning models at scale** to images, using your own or known popular models, to make predictions or transform them into features

Working with general tensors:
* **Applying deep learning models at scale** to tensors of up to 2 dimensions

Deploying Models in SQL:
* **Deploying models as SQL functions** to empower everyone by making deep learning available in SQL

We'll cover each one with examples below.

%md ## Working with image data

%md Let us first get some images to work with in this notebook. We'll use the flowers dataset from the [TensorFlow retraining tutorial](https://www.tensorflow.org/tutorials/image_retraining).

%md The 'file:/...' directory will be cleared out upon cluster termination. That doesn't matter for this example notebook, but in most cases we'd want to store the images in a more permanent place. 

Let us move the files to dbfs so we can see how to work with it in the use cases below.

%md Let us create a small sample set of images for quick demonstrations.

%md ### Loading images

The first step to apply deep learning on images is the ability to load the images. Spark and Deep Learning Pipelines include utility functions that can load millions of images into a Spark DataFrame and decode them automatically in a distributed fashion, allowing manipulation at scale.

%md Using Spark's ImageSchema

%md or we can use custom image library

%md The resulting DataFrame contains a string column named "image" containing an image struct with schema == ImageSchema.

%md ### Transfer learning
Deep Learning Pipelines provides utilities to perform [transfer learning](https://en.wikipedia.org/wiki/Transfer_learning) on images, which is one of the fastest (code and run-time -wise) ways to start using deep learning. Using Deep Learning Pipelines, it can be done in just several lines of code.

%md Firstly, we need to create training & test DataFrames for transfer learning - this piece of code is longer than transfer learning itself below!

%md **Note:** the training step may take a while on Community Edition - try making a smaller training set in that case.