deps.md#

Overview#

Dependency management in sparkmagic is distributed across multiple file types and locations. The repository uses a monorepo structure with three sub-packages—sparkmagic (core magic commands and kernels), hdijupyterutils (Jupyter utilities shared library), and autovizwidget (automatic visualization widget)—each with its own requirements.txt 1 2 3 and setup.py file. Development is unified via Poetry using a single pyproject.toml at the repository root.

Dependencies are declared or referenced in:

• 3 requirements.txt files — one per sub-package defining production dependencies 1 2 3
• 3 setup.py files — packaging metadata for each sub-package; install_requires dynamically reads from the corresponding requirements.txt
• pyproject.toml — Poetry development environment that installs all three sub-packages in editable mode with pinned versions for numpy, pandas, pytest, and mock
• poetry.lock — fully resolved lockfile for the dev environment
• 2 Dockerfiles — Dockerfile.spark installs Livy 0.7.1-incubating on Spark 2.4.7; Dockerfile.jupyter installs sparkmagic with optional editable mode
• docker-compose.yml — orchestrates the spark and jupyter services with ports 8998 and 8888
• 5 GitHub Actions workflow files — tests.yml, lint.yaml, publish.yml, release.yml, and docker_build.yml in .github/workflows/

This document serves as a centralized reference to eliminate setup and release ambiguity. It is the single place to look for what dependencies exist, why they are included, and what version constraints matter.

Core Dependencies#

sparkmagic's dependencies organize into several categories based on their function: Spark integration, Jupyter infrastructure, data manipulation, internal sub-packages, testing, authentication, and development tools.

Spark / Livy Integration#

Apache Livy (server-side, not a Python pip dependency) is the REST server that bridges Jupyter to Spark clusters. sparkmagic's livyclientlib communicates with it over HTTP. The Docker image pins version 0.7.1-incubating 4.

requests (no version pin) serves as the HTTP client used by livyclientlib to submit Spark statements and poll their status via the Livy REST API 5.

requests_kerberos >=0.8.0 extends requests with Kerberos/SPNEGO authentication for accessing Kerberos-secured Livy endpoints common in enterprise Hadoop and CDH clusters 6.

tornado >=4 provides the async networking framework required by Jupyter kernel infrastructure and sparkmagic's wrapperkernel for I/O 7.

Jupyter / IPython Kernel Dependencies#

ipython >=4.1.2 is the core interactive Python shell. It provides the magic command infrastructure (the % and %% syntax) and the IPython.display API sparkmagic uses to render output. Version 0.23.0 added support for ipython>=9.0.0 8 9.

ipykernel >=4.2.2 implements the Jupyter kernel messaging protocol over ZeroMQ. sparkmagic's pysparkkernel, sparkkernel, and sparkrkernel are custom kernels built on top of this 10.

ipywidgets >5.0.0 provides interactive widgets for the notebook UI. Used in autovizwidget and sparkmagic's controllerwidget for components like the Endpoint widget and interactive charts 11.

notebook >=4.2 is the Jupyter Notebook server. sparkmagic ships a server extension that registers the Livy proxy endpoint; notebook provides the extension API 12.

jupyter >=1 (hdijupyterutils only) is a meta-package ensuring the full Jupyter stack is present 13.

nest_asyncio >1.5.5 patches asyncio to allow nested event loops. Required because the Jupyter notebook already runs an event loop, and submitting Spark jobs involves another async layer. Without this, calls would deadlock 14.

Data and Visualization#

pandas <3.0.0 is used across all three sub-packages for DataFrame manipulation and display. Results from Livy are deserialized into pandas DataFrames. Upper-capped because pandas 3.x introduced breaking API changes not yet verified as compatible 15 16 17.

numpy (no pin in sparkmagic; >=1.16.5 in hdijupyterutils) is the numerical computing library underlying pandas and array operations in autovizwidget 18 19.

plotly >=3 (autovizwidget only) is the interactive chart library powering autovizwidget's visualization capabilities. Version 0.12.9 switched from plotly 2.x to 3.x 20 21.

Internal Sub-package Dependencies#

hdijupyterutils >=0.6 is required by both sparkmagic and autovizwidget. It provides shared Jupyter utility functions 22 23.

autovizwidget >=0.6 is required by sparkmagic. It provides the auto-visualization widget for DataFrames 24.

Testing#

pytest (CI: no pin; pyproject.toml: ^8.3.3) is the test runner that replaced the old nose-based test suite. The migration to pytest happened in version 0.20.2 25 26 27.

mock ^5.1.0 is the mocking library for unit tests 28 25.

Authentication Transitive Dependencies#

These Kerberos-related packages are pulled in transitively by requests_kerberos:

gssapi 1.8.2 provides low-level Kerberos bindings 29.

krb5 0.4.1 provides Kerberos API bindings for Python 29.

pyspnego 0.7.0 provides Windows Negotiate (NTLM/Kerberos) authentication support 29.

libkrb5-dev (system dependency, CI and Docker) provides C library headers needed to compile Kerberos Python bindings on Linux. The Docker image uses conda to install requests-kerberos as a workaround for missing Linux headers 30 31.

Build / Dev Tools#

Poetry is the unified dev environment manager. The pyproject.toml is named "development" and installs all three sub-packages as editable path dependencies 32.

black is the code formatter enforced in CI via lint.yaml 33.

Version Constraints#

Python#

The minimum supported Python version is 3.8 34. CI tests run against Python 3.8, 3.9, 3.10, 3.11, and 3.12 35.

Python 3.7 support was dropped in version 0.22.0 36. Note that the setup.py classifier list still incorrectly references Python 3.7; the actual minimum is 3.8 per pyproject.toml.

pandas#

Production environments cap pandas at <3.0.0 across all three requirements.txt files 15 16 17. The development environment pins it more tightly to ^2.0.3 37.

The upper cap exists because pandas 3.x introduces breaking API changes that have not yet been verified for compatibility. Before version 0.22.0, pandas was capped at <2.0.0 38; it was updated to 2.x in that release 39.

ipython#

The production constraint is >=4.1.2 8. Version 0.23.0 added support for ipython>=9.0.0 40.

Historically, ipython was pinned at <8.0.0 in version 0.19.2 due to breaking changes 41. That upper limit was removed in version 0.20.0.

ipykernel#

The current constraint is >=4.2.2 10. Support for ipykernel 6.x was added in version 0.20.0 42.

Previously, ipykernel was pinned at <6.0.0 in version 0.19.1 due to breaking changes 43.

ipywidgets#

The production constraint is >5.0.0 11. Version 8.0.0 introduced a breakage that was fixed in sparkmagic 0.20.2 44.

notebook#

The production constraint is >=4.2 12. Support for notebook 7.x was added in version 0.21.0 45, though the Docker Jupyter image pins notebook==6.5.5 46.

Apache Livy and Spark (Docker)#

The Docker environment pins Livy 0.7.1-incubating 4 on a Spark 2.4.7 base with Scala 2.11 and Java 8 47.

These versions are pinned due to the LIVY-423 bug, which requires Scala 2.11 compatibility. Upgrading Livy requires updating the Spark base image accordingly.

tornado#

The constraint is >=4 7. This is a very loose constraint and remains compatible with current tornado 6.x releases.

nest_asyncio#

The constraint is >1.5.5 14. This strict lower bound avoids known event-loop bugs in earlier versions.

numpy#

hdijupyterutils specifies >=1.16.5 18, while sparkmagic has no version pin 19. The development environment pins numpy to ^1.24.4 48.

Dependency Files Reference#

Dependencies are defined across multiple files serving different purposes: production requirements for each sub-package, unified development environment configuration, Docker container setups, and continuous integration workflows.

requirements.txt Files#

Three requirements.txt files define the production dependencies for each sub-package:

sparkmagic/requirements.txt contains 12 entries specifying core sparkmagic dependencies 1. This includes internal dependencies on hdijupyterutils and autovizwidget, Jupyter kernel infrastructure (ipython, ipykernel, ipywidgets, notebook), networking libraries (requests, tornado), Kerberos authentication (requests_kerberos), data manipulation (pandas, numpy), and async event loop support (nest_asyncio).

hdijupyterutils/requirements.txt contains 7 entries for the utility library 2. This covers Jupyter components (ipython, ipywidgets, ipykernel, jupyter, notebook) and data libraries (pandas, numpy).

autovizwidget/requirements.txt contains 5 entries for the visualization widget 3. This includes plotly for charting, ipywidgets for interactive widgets, hdijupyterutils as an internal dependency, notebook for Jupyter integration, and pandas for data handling.

setup.py Files#

Each sub-package has a setup.py that uses an identical pattern to read dependencies from its local requirements.txt. The requirements() helper function reads the file, strips comments and empty lines, and returns the result directly as install_requires 49. This means setup.py and requirements.txt are always in sync—the requirements.txt file is the single source of truth for production dependencies.

sparkmagic/setup.py packages the core sparkmagic library including controllerwidget, four kernel implementations (pysparkkernel, sparkkernel, sparkrkernel, wrapperkernel), livyclientlib, auth, magics, utils, and serverextension modules 50.

hdijupyterutils/setup.py packages the single hdijupyterutils module 51.

autovizwidget/setup.py packages autovizwidget along with plotlygraphs, widget, and utils submodules 52.

pyproject.toml#

The pyproject.toml defines a Poetry-managed development environment named "development" 53—not intended for distribution, but for unified local development across all three sub-packages. It declares Python ^3.8 as the minimum version 34 and installs all three sub-packages as editable path dependencies 54. The file pins numpy ^1.24.4, pandas ^2.0.3, pytest ^8.3.3, and mock ^5.1.0 55 for a consistent development environment.

poetry.lock#

The poetry.lock file is a fully resolved lockfile spanning 2338 lines 29. It pins all transitive dependencies for the development environment, including low-level libraries like gssapi, krb5, cryptography, and jupyter-client. Notable pinned versions include ipykernel 6.19.4, ipython 8.10.0, numpy 1.24.4, pandas 2.0.3, plotly 5.24.1, pytest 8.3.3, requests 2.32.3, and tornado 6.4.2.

Dockerfile.spark#

Dockerfile.spark builds the Apache Livy server container. It starts from datamechanics/spark:2.4.7-hadoop-3.1.0-java-8-scala-2.11-python-3.7-latest 56. The Spark 2.4.7 and Scala 2.11 versions are pinned due to the LIVY-423 compatibility issue 57. System dependencies installed via apt-get include curl, git, Python 3 tooling (python3, python3-setuptools, python3-venv, python3-pip), and R (r-base, r-base-core) 58. The file downloads and installs Apache Livy 0.7.1-incubating from the Apache archives and exposes port 8998 59.

Dockerfile.jupyter#

Dockerfile.jupyter builds the Jupyter notebook container with sparkmagic installed. It starts from jupyter/base-notebook 60 and accepts a dev_mode build argument 61. The file installs requests-kerberos via conda rather than pip to handle Linux header dependencies for Kerberos libraries automatically. It upgrades pip and setuptools before copying the three sub-package directories into the container 62. When dev_mode=true, all three packages install as editable local packages; when dev_mode=false (the default), it installs from PyPI with pip install sparkmagic 63. After sparkmagic installation, ipywidgets is installed separately 64 and the three kernel specs are registered before enabling the sparkmagic server extension 65.

docker-compose.yml#

The docker-compose.yml orchestrates the two-container setup. It defines a spark service using Dockerfile.spark that exposes port 8998 66 and a jupyter service using Dockerfile.jupyter on port 8888 67. The jupyter service links to the spark service 68 so the sparkmagic config.json can reference the Livy server as the hostname spark. The dev_mode build argument defaults to "false" 69, causing a production install from PyPI.

GitHub Actions Workflows#

Five GitHub Actions workflow files manage testing, linting, publishing, release automation, and Docker builds.

.github/workflows/tests.yml is the main CI pipeline. It triggers on every push, pull request, daily via cron, and as a reusable workflow 70. The workflow tests against a matrix of Python 3.8 through 3.12 on ubuntu-latest 71. Each job installs libkrb5-dev for Kerberos support, installs pytest and mock 25, then installs each sub-package via editable install after its requirements.txt dependencies. It runs pytest for each of the three packages separately 72.

.github/workflows/lint.yaml validates code formatting using the Black formatter 33.

.github/workflows/publish.yml handles PyPI publishing when a GitHub release is created 73. It first runs the test workflow, then uses Python 3.8 74 to build and publish each sub-package to PyPI with separate authentication tokens.

.github/workflows/release.yml provides workflow_dispatch automation for version bumping 75. It accepts a bump type parameter (major, minor, or patch, defaulting to patch) and creates a GitHub release with the CHANGELOG.md contents. Added in version 0.22.0 76.

.github/workflows/docker_build.yml builds Docker images for both services 77. When triggered on a push, it sets dev_mode="true" to use local editable installs; when triggered on the daily schedule, it builds with production packages.

Installation Scenarios#

Dependency requirements and installation procedures differ across development, production, testing, and containerized deployment environments.

Development Setup#

For local development, use Poetry at the repository root:

poetry install

This installs all three sub-packages as editable from their local paths 54. Poetry pins key development dependencies—numpy ^1.24.4, pandas ^2.0.3, pytest ^8.3.3, and mock ^5.1.0 55—ensuring a consistent environment across contributors. Python ^3.8 is required 34.

If Poetry is unavailable, each package can be installed manually in editable mode:

pip install -r hdijupyterutils/requirements.txt -e hdijupyterutils
pip install -r autovizwidget/requirements.txt -e autovizwidget
pip install -r sparkmagic/requirements.txt -e sparkmagic
pip install pytest mock

This mirrors the CI workflow 78 but lacks Poetry's version pinning.

Production Deployment#

For production environments, install sparkmagic from PyPI:

pip install sparkmagic

This installs sparkmagic with loose version constraints from requirements.txt and automatically pulls in hdijupyterutils and autovizwidget as transitive dependencies 79. No separate installation of the sub-packages is needed.

For Jupyter Notebook 7.x / JupyterLab 3.x, ensure ipywidgets is installed and optionally register the wrapper kernels and server extension 80:

pip install ipywidgets
jupyter-kernelspec install <sparkmagic_path>/sparkmagic/kernels/sparkkernel
jupyter-kernelspec install <sparkmagic_path>/sparkmagic/kernels/pysparkkernel
jupyter-kernelspec install <sparkmagic_path>/sparkmagic/kernels/sparkrkernel
jupyter server extension enable --py sparkmagic

For Jupyter Notebook ≤5.2 / JupyterLab 1 or 2, enable the ipywidgets notebook extension 81:

jupyter nbextension enable --py --sys-prefix widgetsnbextension
# For JupyterLab 1 or 2:
jupyter labextension install "@jupyter-widgets/jupyterlab-manager"

Testing Environment (CI)#

The GitHub Actions workflow tests across Python 3.8–3.12 on ubuntu-latest 71. The full setup sequence:

# 1. Install system dependency for Kerberos
sudo apt-get install -y libkrb5-dev

# 2. Install test framework and packages (editable installs)
pip install pytest mock
pip install -r hdijupyterutils/requirements.txt -e hdijupyterutils
pip install -r autovizwidget/requirements.txt -e autovizwidget
pip install -r sparkmagic/requirements.txt -e sparkmagic

# 3. Run tests
pytest hdijupyterutils
pytest autovizwidget
mkdir ~/.sparkmagic # required before running sparkmagic tests
pytest sparkmagic

82

Note that the ~/.sparkmagic directory must exist before running the sparkmagic tests 83.

Docker Containerized Deployment#

The docker-compose.yml orchestrates two services: a Spark/Livy backend and a Jupyter frontend.

Spark/Livy backend — exposes the Livy REST API on port 8998 66:

docker compose up spark

Jupyter frontend — accessible on port 8888 67:

docker compose up jupyter

By default, dev_mode=false installs sparkmagic from PyPI 69. For development, set dev_mode=true in docker-compose.yml to install all three packages as editable from the copied source tree 63:

args:
  dev_mode: "true"

Full environment — launch both services together:

docker compose up

The Jupyter service links to the Spark service, resolving the hostname spark to the Livy endpoint at http://spark:8998 68. Note that requests-kerberos is installed via conda (not pip) in the Jupyter container to avoid Linux header issues on Debian-based images 31.

Maintenance Notes#

Updating Dependencies Safely#

The requirements.txt files are the single source of truth for production dependencies. Each sub-package's setup.py dynamically reads from requirements.txt at install time—never edit install_requires directly in setup.py.

After changing any requirements.txt file, run poetry update to regenerate poetry.lock and confirm the dev environment resolves correctly. DependaBot is enabled for pip packages and GitHub Actions 84 and will open pull requests for outdated dependencies automatically.

Always validate changes by running the full test suite:

pytest hdijupyterutils
pytest autovizwidget
pytest sparkmagic

The CI pipeline tests against Python 3.8–3.12 35.

pandas: The Key Constraint#

pandas is capped at <3.0.0 across all three requirements.txt files 15 16 17 because pandas 3.x API changes have not been verified for compatibility. This follows a historical pattern: pandas was previously capped at <2.0.0 and upgraded to 2.x in version 0.22.0 39 38.

Before bumping the upper cap, verify DataFrame operations in all three sub-packages work correctly with the new pandas version.

Resolving Dependency Conflicts#

Kerberos dependencies: If Kerberos-related dependencies fail to install on Linux via pip, use conda as a workaround. Dockerfile.jupyter installs requests-kerberos via conda to avoid missing Linux header issues on Debian-based images 31. The CI workflow installs libkrb5-dev as a system package before Python dependencies 85.

ipython and ipykernel: Both have been pinned at major version boundaries in the past due to breaking changes. Historical context from CHANGELOG.md:

• ipython was pinned <8.0.0 in v0.19.2 due to breaking changes 41; removed in v0.20.0
• ipykernel was pinned <6.0.0 in v0.19.1 43; removed in v0.20.0
• Support for ipython>=9.0.0 added in v0.23.0 9

ipywidgets: Version 8.0.0 was a known breakage point, fixed in sparkmagic 0.20.2 44. When encountering widget issues, check the ipywidgets version first.

Adding New Dependencies#

1. Add to the relevant sub-package's requirements.txt (sparkmagic, hdijupyterutils, or autovizwidget)
2. If the dependency is only needed for tests or dev tooling, add it to pyproject.toml [tool.poetry.dependencies] instead
3. If the dependency has C extensions requiring system libraries, add installation steps to both Dockerfile.jupyter (e.g., via conda) and tests.yml (e.g., via apt-get)
4. Run poetry update and commit the updated poetry.lock

Known Stale Metadata#

The sparkmagic setup.py classifiers still list Python 3.7, but the actual minimum supported version is Python 3.8 per pyproject.toml and the v0.22.0 CHANGELOG 36. This classifier should be updated.

Livy and Docker Image Pinning#

Dockerfile.spark is pinned to Spark 2.4.7 / Scala 2.11 / Livy 0.7.1-incubating due to LIVY-423 47 4. Upgrading Livy requires:

1. Finding a Spark base image compatible with the newer Scala version
2. Updating the datamechanics/spark base image reference in Dockerfile.spark
3. Verifying compatibility between the new Spark and Livy versions

The Docker setup is intended for local development and testing, not production Spark deployments.

PyPI Publishing#

The publish workflow uses Python 3.8 specifically for building and publishing packages. The release workflow handles automated version bumping via bumpversion. See RELEASING.md for full release instructions.