Azure Blob Storage with Pyspark

Azure Blob Storage is Microsoft's highly scalable cloud storage solution designed for handling massive amounts of unstructured data. It's cost-effective and versatile, capable of storing all kinds of data—from text and binary to images and videos. This makes it a great choice for creating data lakes or warehouses, where you can store both raw and preprocessed data for future analysis.

In this post, we'll explore how to access Azure Blob Storage using PySpark, the Python API for Apache Spark. Apache Spark is a powerful, fast, and general-purpose cluster computing system that enables large-scale data processing. With its ability to manage data parallelism and fault tolerance across clusters, Spark is a critical tool in the data engineer's arsenal. Yeah, I've been talking about Spark for a while now.

Before diving into the nuts and bolts of accessing Azure Blob Storage with PySpark, let's highlight what sets Azure Blob Storage apart. It uses custom protocols called wasb/wasbs to facilitate data access. The wasb protocol, short for Windows Azure Storage Blob, is an extension built on top of the Hadoop Distributed File System (HDFS) APIs, offering an abstraction that separates storage concerns from processing. To interact with Azure Blob Storage via Spark, you must include the hadoop-azure.jar and azure-storage.jar files in your spark-submit command.

Here's how you can submit a Spark job with the necessary dependencies:

$ spark-submit --py-files src.zip \
            --master yarn \
            --deploy-mode=cluster \
            --jars hadoop-azure.jar,azure-storage.jar
            src/app.py

Alternatively, if you're using Docker or have your application installed on a cluster, place the JAR files where PySpark can locate them. Use the following commands to download the required JAR files and place them in the correct directory:

$ wget -nc -nv -O /usr/local/lib/python3.5/site-packages/pyspark/jars/azure-storage-2.2.0.jar http://central.maven.org/maven2/com/microsoft/azure/azure-storage/2.2.0/azure-storage-2.2.0.jar
$ wget -nc -nv -O /usr/local/lib/python3.5/site-packages/pyspark/jars/hadoop-azure-2.7.3.jar http://central.maven.org/maven2/org/apache/hadoop/hadoop-azure/2.7.3/hadoop-azure-2.7.3.jar

Now that we have the necessary libraries, let's create a Spark Session, which serves as the entry point for PySpark to interact with cluster resources:

from pyspark.sql import SparkSession

session = SparkSession.builder.getOrCreate()

To interact with Azure Blob Storage, you'll need to configure your Spark session with either an account access key or a Shared Access Signature (SAS) token.

Using an account key:

session.conf.set(
    "fs.azure.account.key.<storage-account-name>.blob.core.windows.net",
    "<your-storage-account-access-key>"
)

Or using a SAS token:

session.conf.set(
    "fs.azure.sas.<container-name>.blob.core.windows.net",
    "<sas-token>"
)

After setting up the Spark session and account key or SAS token, we can start reading and writing data from Azure Blob Storage using PySpark.

Once the Spark session is set up, you can start reading data from Azure Blob Storage. The read method of the Spark session object returns a DataFrame, and the path to your data should start with wasbs:// (for secure access) or wasb:// (for non-secure access).

Here's an example of reading a Parquet file from Azure Blob Storage:

sdf = session.read.parquet(
    "wasbs://<container-name>@<storage-account-name>.blob.core.windows.net/<prefix>"
)

In this path:

• <container-name> is the name of your container within Azure Blob Storage.
• <storage-account-name> is the name of your storage account.
• <prefix> is the optional path to the file or folder. If your file is at the root, you can omit the prefix.

You can also write data back to Azure Blob Storage using PySpark. The write method of a DataFrame object allows you to specify the output path:

df.write.csv(
    "wasbs://<container-name>@<storage-account-name>.blob.core.windows.net/<prefix>"
)

You can further customize the write operation by specifying options for format, compression, partitioning, and more.

Beyond basic reading and writing, PySpark offers a wealth of tools for working with data stored in Azure Blob Storage. You can use PySpark SQL for executing SQL queries or leverage PySpark MLlib for performing machine learning tasks.

Overall, Azure Blob Storage with PySpark is a powerful combination for building data pipelines and data analytics solutions in the cloud. With the ability to store and process large amounts of data in a scalable and cost-effective way, Azure Blob Storage and PySpark provide a powerful platform for building big data applications.

Combining Azure Blob Storage with PySpark offers a powerful platform for building scalable data pipelines and analytics solutions in the cloud. Whether you're processing large datasets, running complex queries, or training machine learning models, this combination provides the flexibility and performance needed for modern data engineering tasks.

Additional materials

• Processing Big Data with Azure HDInsight by Vinit Yadav