sparkmagic#

Sparkmagic is a set of tools for interactively working with remote Spark clusters in Jupyter notebooks. Sparkmagic communicates with remote Spark clusters through a REST server. Three server implementations are currently compatible with Sparkmagic:

• Livy — for running interactive sessions on YARN
• Lighter — for running interactive sessions on YARN or Kubernetes (PySpark sessions only)
• Ilum — for running interactive sessions on YARN or Kubernetes

Sparkmagic includes a set of magics for interactively running Spark code in multiple languages, as well as dedicated kernels that turn Jupyter into an integrated Spark environment — all without requiring any Spark components on the Jupyter server itself.

Features#

• Multi-language Spark execution — Run Spark code in Python, Scala, and R against any remote cluster through Livy 1
• Automatic context creation — SparkContext (sc) and HiveContext (sqlContext) are created automatically at session startup 2
• SparkSQL support — Run SQL queries easily with the %%sql magic 3
• Automatic visualization — Results from SQL queries are automatically visualized in PySpark, Spark, and SparkR kernels using an interactive, no-code visual interface 4
• Logs and app info — Inspect Spark application information and driver logs with the %%info and %%logs magics 5
• Pandas integration — Capture SQL query output as Pandas DataFrames to interact with any Python data science library 6
• Local-to-Spark data transfer — Send local variables, files, or DataFrames to the remote cluster with %%send_to_spark 7
• Flexible authentication — Supports no authentication, HTTP Basic Access, and Kerberos 8
• Configurable retries and heartbeats — Built-in session resilience via configurable retry policies and heartbeat management 9

Installation#

Prerequisites#

• Python 3.8 or later — CI tests run against Python 3.8, 3.9, 3.10, 3.11, and 3.12 10
• Jupyter Notebook or JupyterLab (see version-specific steps below)
• libkrb5-dev (Linux) — required only if you plan to use Kerberos authentication 10

Jupyter Notebook 7.x / JupyterLab 3.x#

1. Install the library

   ⧉

   pip install sparkmagic
   

2. Ensure ipywidgets is installed

   ⧉

   pip install ipywidgets
   

3. (Optional) Install the wrapper kernels

   Run pip show sparkmagic to find the installation path, then cd to that location and run:

   ⧉

   jupyter-kernelspec install sparkmagic/kernels/sparkkernel
   jupyter-kernelspec install sparkmagic/kernels/pysparkkernel
   jupyter-kernelspec install sparkmagic/kernels/sparkrkernel
   

4. (Optional) Configure Sparkmagic

   Edit (or create) ~/.sparkmagic/config.json. See sparkmagic/example_config.json for a full list of options. 9

5. (Optional) Enable the server extension

   Enables programmatic cluster switching via the /reconnectsparkmagic API:

   ⧉

   jupyter server extension enable --py sparkmagic
   

Jupyter Notebook 5.2 or Earlier / JupyterLab 1 or 2#

1. Install the library

   ⧉

   pip install sparkmagic
   

2. Enable the notebook widget extension

   ⧉

   jupyter nbextension enable --py --sys-prefix widgetsnbextension
   

3. JupyterLab 1 or 2 only — install the JupyterLab widget manager:

   ⧉

   jupyter labextension install "@jupyter-widgets/jupyterlab-manager"
   

4. (Optional) Install the wrapper kernels (same as above)

5. (Optional) Configure ~/.sparkmagic/config.json

6. (Optional) Enable the server extension (legacy command):

   ⧉

   jupyter serverextension enable --py sparkmagic
   

11

Installing from Source#

If you want to contribute or test the latest unreleased changes, install the three constituent packages in editable mode:

git clone https://github.com/jupyter-incubator/sparkmagic
cd sparkmagic

pip install -e hdijupyterutils
pip install -e autovizwidget
pip install -e sparkmagic

Alternatively, use Poetry for an isolated virtual environment:

poetry install
# If numpy or pandas installation fails, run:
# poetry run pip install numpy pandas
# then re-run:
poetry install

12 13

Configuration#

Sparkmagic reads its configuration from ~/.sparkmagic/config.json. If the file does not exist, sensible defaults are used. A fully annotated reference is available at sparkmagic/example_config.json. 9

Kernel Credentials#

Set the Livy endpoint, credentials, and auth type for each kernel language:

{
    "kernel_python_credentials": {
        "username": "",
        "password": "",
        "url": "http://your-livy-server:8998",
        "auth": "None"
    },
    "kernel_scala_credentials": {
        "username": "",
        "password": "",
        "url": "http://your-livy-server:8998",
        "auth": "None"
    },
    "kernel_r_credentials": {
        "username": "",
        "password": "",
        "url": "http://your-livy-server:8998",
        "auth": "None"
    }
}

Valid auth values are "None", "Basic_Access", and "Kerberos".

Spark Session Defaults#

Two keys control default Spark session configuration 14:

• session_configs — Default settings that are replaced whenever a user calls %%configure.
• session_configs_defaults — Settings that are preserved even after a user calls %%configure; a user must explicitly override them.

{
    "session_configs": {
        "driverMemory": "1000M",
        "executorCores": 2
    },
    "session_configs_defaults": {
        "spark.sql.catalog.type": "hive"
    }
}

Timeouts and Retries#

{
    "wait_for_idle_timeout_seconds": 15,
    "livy_session_startup_timeout_seconds": 60,
    "heartbeat_refresh_seconds": 30,
    "livy_server_heartbeat_timeout_seconds": 0,
    "heartbeat_retry_seconds": 10,
    "retry_policy": "configurable",
    "retry_seconds_to_sleep_list": [0.2, 0.5, 1, 3, 5],
    "configurable_retry_policy_max_retries": 8
}

HTTP Session Adapters#

For custom HTTP behavior (mTLS, custom retry logic, etc.), configure a transport adapter 15:

{
    "http_session_config": {
        "adapters": [
            {
                "prefix": "http://",
                "adapter": "myadapter.module.MyAdapter"
            }
        ]
    }
}

Papermill Integration#

To make Sparkmagic stop on Spark errors during Papermill rendering 16:

{
    "shutdown_session_on_spark_statement_errors": true,
    "all_errors_are_fatal": true
}

To clean up all registered Livy sessions on exit regardless of process termination type:

{
    "cleanup_all_sessions_on_exit": true,
    "all_errors_are_fatal": true
}

Programmatic Overrides#

Configuration options can also be overridden at runtime in a notebook 17:

import sparkmagic.utils.configuration as conf
conf.override('cleanup_all_sessions_on_exit', True)

# Or reference the config key by name:
conf.override(conf.cleanup_all_sessions_on_exit.__name__, True)

Note: Any override that affects session initialization (such as cleanup_all_sessions_on_exit) must be set before loading the extension with %load_ext sparkmagic.magics.

Authentication#

Sparkmagic supports three built-in authentication methods for connecting to Livy:

18

Kerberos#

Kerberos authentication is implemented via the requests-kerberos package. Sparkmagic expects a valid Kerberos ticket to already be present in the system credential cache — run kinit first. 19

The default constructor used is:

HTTPKerberosAuth(mutual_authentication=REQUIRED)

This can be customized in ~/.sparkmagic/config.json 20:

{
    "kerberos_auth_configuration": {
        "mutual_authentication": 1,
        "service": "HTTP",
        "delegate": false,
        "force_preemptive": false,
        "principal": "principal",
        "hostname_override": "hostname_override",
        "sanitize_mutual_error_response": true,
        "send_cbt": true
    }
}

Custom Authenticators#

You can implement custom authentication by subclassing the base Authenticator class and overriding __call__(request), which attaches HTTP authentication to the outgoing request. 21

Subclasses that add new credential attributes (like the built-in Basic authenticator which adds username and password) should also override __hash__, __eq__, update_with_widget_values, and get_widgets so the widget-based credential UI works correctly. 22

Registering your custom authenticator:

After installing your custom package (e.g. pip install git+https://git_repo_url/#egg=customauthenticator), register it in ~/.sparkmagic/config.json 23:

{
    "authenticators": {
        "Kerberos": "sparkmagic.auth.kerberos.Kerberos",
        "None": "sparkmagic.auth.customauth.Authenticator",
        "Basic_Access": "sparkmagic.auth.basic.Basic",
        "Custom_Auth": "customauthenticator.customauthenticator.CustomAuthenticator"
    }
}

The key ("Custom_Auth") becomes the display name in the %manage_spark widget dropdown and the value passed to the -t flag in %%spark add.

Usage#

There are two ways to use Sparkmagic. Full working examples are available in the examples/ directory.

1. Via the IPython Kernel#

Use Sparkmagic as a magic extension inside a standard IPython/Jupyter notebook.

Load the extension:

%load_ext sparkmagic.magics

Manage connections with the widget:

%manage_spark

This opens an interactive widget for adding/removing Spark endpoints and sessions. 24

Run Spark code with %%spark:

%%spark -s mysession -l python
sc.version

Run a SQL query:

%%spark -s mysession -c sql
SELECT * FROM my_table LIMIT 10

Available %%spark subcommands include add, config, run, logs, delete, cleanup, and info. See the Magics in IPython Kernel notebook for a full walkthrough. 25

2. Via the Spark Kernels#

Sparkmagic provides three dedicated Jupyter kernels that automatically connect to a remote Spark cluster. Select one when creating or switching a notebook:

26 27 28 29

Once inside a kernel, the following magics are available:

30

Example — run Spark code and capture results locally:

%%spark -o df
spark.sql("SELECT year, count(*) as cnt FROM events GROUP BY year ORDER BY year")

%%local
import matplotlib.pyplot as plt
df.plot(x="year", y="cnt", kind="bar")
plt.show()

Example — send a local DataFrame to Spark:

%%send_to_spark -i local_df -t df -n remote_df

3. Sending Local Data to Spark#

See the Send Local Data to Spark notebook for a step-by-step guide on moving DataFrames and other objects between your local environment and the remote Spark cluster.

Docker (Local Development)#

The included docker-compose.yml spins up a complete Sparkmagic stack for local testing and development: a Jupyter notebook server with all extensions installed, and a Livy server backed by a local-mode Spark instance. 31

Note: This setup is intended for developing and testing Sparkmagic itself. In production, Sparkmagic is most useful when your Spark cluster runs on a separate, remote machine.

Requirements: Docker and Docker Compose

Start the stack:

docker compose build
docker compose up

Open Jupyter in your browser at http://localhost:8888. Inside a notebook, configure a Sparkmagic endpoint at http://spark:8998. This endpoint supports both Scala and Python sessions. You can also start a wrapper kernel for Scala, Python, or R directly from the kernel list. 32

Stop the stack:

# Interrupt docker compose with Ctrl-C, then optionally:
docker compose down

Developing with live edits:

Set dev_mode: true in the docker-compose.yml build arguments for the jupyter service, then rebuild. This installs your local versions of autovizwidget, hdijupyterutils, and sparkmagic in editable mode, so changes to the source files are reflected immediately inside the running container. 33

# After changing dev_mode to true:
docker compose build
docker compose up

Server Extension API#

Sparkmagic ships an optional Jupyter server extension that exposes a REST endpoint for programmatic cluster management. Enable it with:

jupyter server extension enable --py sparkmagic # Jupyter Notebook 7.x / JupyterLab 3.x
# or
jupyter serverextension enable --py sparkmagic # older Jupyter

34

POST /reconnectsparkmagic#

Connects (or reconnects) a specific notebook to a Spark cluster. The kernel is started or restarted and pointed at the provided endpoint. 35

Request body:

{
    "path": "path/to/notebook.ipynb",
    "username": "username",
    "password": "password",
    "endpoint": "http://your-livy-server:8998",
    "auth": "Kerberos",
    "kernelname": "pysparkkernel"
}

auth must be one of None, Basic_Access, or Kerberos. kernelname is optional and defaults to the value set in config.json (pysparkkernel if not configured).

Responses:

Success response body:

{
    "success": true,
    "error": null
}

36

Architecture#

Sparkmagic uses Livy, a REST server for Spark, to remotely execute all user code. The library collects output from your code — as plain text or a JSON document — and displays results as formatted text or Pandas DataFrames, as appropriate. 37

Advantages#

1. No local Spark installation required — All Spark code runs remotely; the Jupyter server only needs Python and Sparkmagic installed. 38

2. Uniform multi-language support — Python, Scala, and R kernels share the same feature set. Adding new language support requires only a new kernel wrapper. 39

3. Multiple simultaneous endpoints — A single notebook can open Spark jobs in different languages against different remote clusters concurrently. 40

4. Ecosystem integration — SQL output is materialized as Pandas DataFrames, making it trivially composable with any Python data science or visualization library (e.g., matplotlib, Plotly). 41

Limitations#

1. Livy overhead — All code and results travel through the Livy REST API, which adds latency compared to running code directly on the cluster.

2. JSON serialization boundary — Structured data (DataFrames) must be serialized to JSON by Livy and deserialized by Sparkmagic on the client side. Client-side data manipulation should use %%local mode with Python. 42

Repository Structure#

This repository is a monorepo containing three interdependent Python packages: 43

Contributing#

Contributions of all kinds are welcome — bug reports, feature requests, documentation improvements, and code changes!

Reporting Issues#

• Bugs and feature requests: Open an issue on GitHub.
• Security vulnerabilities: Report privately via https://jupyter.org/security (security@ipython.org) or through Tidelift. Do not open a public issue for security reports. 44

Development Setup#

1. Clone the repository

   ⧉

   git clone https://github.com/jupyter-incubator/sparkmagic
   cd sparkmagic
   

2. Install packages in editable mode 45

   ⧉

   pip install -e hdijupyterutils
   pip install -e autovizwidget
   pip install -e sparkmagic
   

   Or use Poetry for an isolated virtual environment: 13

   ⧉

   poetry install
   # If numpy/pandas installation fails:
   # poetry run pip install numpy pandas && poetry install
   

Running Tests#

Tests use pytest and run against Python 3.8–3.12 in CI. 10

# Standard install
pytest

# Poetry install
poetry run pytest

Tests are organized per package. The CI workflow runs all three packages (hdijupyterutils, autovizwidget, sparkmagic) on every push and pull request, and on a daily schedule. 10

To run tests for a specific package only:

cd sparkmagic && pytest

Kerberos note: The CI environment installs libkrb5-dev before running tests. If you are testing Kerberos-related code locally, ensure this system library is installed.

Code Style#

All code is formatted with Black. Style compliance is enforced by CI on every push and pull request. 46

# Format your changes before committing:
black .

Submitting a Pull Request#

1. Fork the repository and create a feature branch.
2. Make your changes, ensuring tests pass and code is formatted with Black.
3. Open a pull request against the master branch with a clear description of the change.

Releases#

Releases are fully automated via GitHub Actions. Maintainers trigger a release by running the Release workflow, selecting the release type (patch, minor, or major). The CHANGELOG.md should be updated on master before triggering a release. 47

License#

Sparkmagic is licensed under the Modified BSD License (3-Clause BSD). 48

Copyright © 2001–2015 IPython Development Team; 2015– Jupyter Development Team.