AGENTS.md#

A concise reference guide for AI coding agents (and human contributors) working in the sparkmagic repository.

Project Overview#

Sparkmagic is a set of tools for interactively working with remote Spark clusters inside Jupyter notebooks. It communicates with Spark through a REST server — primarily Apache Livy, but also compatible with Lighter and Ilum — and supports Python (PySpark), Scala, and R execution. 1

Three-Package Architecture#

The repository is a monorepo containing three independently distributed Python packages with a layered dependency graph 2:

hdijupyterutils ← base layer (no internal deps)
       ↑
autovizwidget ← depends on hdijupyterutils
       ↑
sparkmagic ← depends on both

Two Usage Modes#

1. IPython magic: Load sparkmagic in a standard Python kernel and use %%spark, %%sql, %%scala, %%sparkr cell magics.
2. Dedicated kernels: Use the PySpark, Spark (Scala), or SparkR kernels directly — every cell is executed on the remote cluster by default.

Current version: 0.23.0 — requires Python 3.8+ 6 2

Development Setup#

Prerequisites#

• Python 3.8+

• Kerberos headers (required to build requests-kerberos):

  ⧉

  # Debian / Ubuntu
  sudo apt-get install -y libkrb5-dev
  
  # macOS (via Homebrew)
  brew install krb5
  
  # Conda (avoids the header requirement entirely)
  conda install requests-kerberos -y
  

  7 8

Option A — Editable pip installs (recommended for most development)#

Install the packages in dependency order — bottom-up. Order matters because autovizwidget and sparkmagic list the other packages as install requirements. 7

git clone https://github.com/jupyter-incubator/sparkmagic
cd sparkmagic

pip install -r hdijupyterutils/requirements.txt -e hdijupyterutils
pip install -r autovizwidget/requirements.txt -e autovizwidget
pip install -r sparkmagic/requirements.txt -e sparkmagic

Each pip install -e <pkg> installs the package in editable mode so changes to source files take effect immediately without reinstalling. 1

Option B — Poetry (all packages in one virtual environment)#

The root pyproject.toml defines a single Poetry project named development that pulls in all three packages as editable path dependencies. 2

poetry install # creates venv and installs all packages + dev deps
poetry run pytest # run full test suite
poetry run python # drop into the venv's Python

Note: The root pyproject.toml is for development only — it is not the source of truth for PyPI distribution. Packages are distributed via the individual setup.py files inside each sub-directory.

Configuration file for kernels#

The Spark kernels look for ~/.sparkmagic/config.json at startup. Copy the example and adjust the Livy URL:

mkdir -p ~/.sparkmagic
cp sparkmagic/example_config.json ~/.sparkmagic/config.json
# Edit ~/.sparkmagic/config.json — set "url" to your Livy endpoint

The default URL in the example config is http://localhost:8998. 9

Gotcha: The root config.json file in the repository is empty — it is a placeholder. Always use sparkmagic/example_config.json as your reference. 9

Docker development environment#

The repo ships two Dockerfiles and a docker-compose.yml that stands up a complete local Spark + Jupyter stack. 10

# Build and start both containers
docker-compose build
docker-compose up
# Open http://localhost:8888 in your browser

Using local source in Docker (dev mode)#

By default the Jupyter container installs sparkmagic from PyPI. To use your local source code instead, enable dev mode: 8

# docker-compose.yml — change the jupyter build args:
    build:
      context: .
      dockerfile: Dockerfile.jupyter
      args:
        dev_mode: "true" # <-- change from "false"

When dev_mode=true, the Dockerfile runs pip install -e . for all three packages instead of pip install sparkmagic. 8

Gotcha: docker-compose.yml ships with dev_mode: "false". You must explicitly change this argument — it is easy to forget and run tests against the published PyPI version instead of your local changes.

Key Commands#

Running Tests#

Tests are run per-package. The sparkmagic package requires ~/.sparkmagic/ to exist before tests run. 7

# hdijupyterutils (6 test files)
pytest hdijupyterutils

# autovizwidget (7 test files)
pytest autovizwidget

# sparkmagic (26 test files) — must create the config dir first
mkdir -p ~/.sparkmagic
pytest sparkmagic

With Poetry:

poetry run pytest hdijupyterutils
poetry run pytest autovizwidget
mkdir -p ~/.sparkmagic && poetry run pytest sparkmagic

Run a specific test file or test:

pytest sparkmagic/sparkmagic/tests/test_livysession.py
pytest sparkmagic/sparkmagic/tests/test_livysession.py::test_start_success

Installing / Building Packages#

# Editable installs (local development)
pip install -r hdijupyterutils/requirements.txt -e hdijupyterutils
pip install -r autovizwidget/requirements.txt -e autovizwidget
pip install -r sparkmagic/requirements.txt -e sparkmagic

# Production install from PyPI
pip install sparkmagic

# Build distribution archives (run inside each package directory)
cd hdijupyterutils && python -m build && cd ..
cd autovizwidget && python -m build && cd ..
cd sparkmagic && python -m build && cd ..

12

Registering Jupyter Kernels#

After installing sparkmagic, kernel specs must be registered separately: 8

SPARKMAGIC_LOC=$(pip show sparkmagic | grep Location | cut -d" " -f2)

jupyter-kernelspec install --user $SPARKMAGIC_LOC/sparkmagic/kernels/pysparkkernel
jupyter-kernelspec install --user $SPARKMAGIC_LOC/sparkmagic/kernels/sparkkernel
jupyter-kernelspec install --user $SPARKMAGIC_LOC/sparkmagic/kernels/sparkrkernel

# Enable the server extension (for the widget/config UI)
jupyter server extension enable --py sparkmagic

Linting & Formatting#

# Format all Python files with Black (required before every commit)
black .

# Check formatting without modifying files (what CI does)
black --check .

Black is the only code style tool enforced by CI. 13

Docker Commands#

# Build images
docker-compose build

# Start full stack (Spark + Jupyter)
docker-compose up

# Start in background
docker-compose up -d

# View logs
docker-compose logs -f jupyter

# Stop and remove containers
docker-compose down

10

Architecture & Structure#

Repository Layout#

sparkmagic/ ← repo root
├── hdijupyterutils/ ← package 1 (base utilities)
│ ├── hdijupyterutils/ ← Python package
│ │ ├── tests/
│ │ ├── configuration.py
│ │ ├── events.py
│ │ ├── filehandler.py
│ │ ├── filesystemreaderwriter.py
│ │ ├── ipythondisplay.py
│ │ ├── ipywidgetfactory.py
│ │ ├── log.py
│ │ └── utils.py
│ ├── requirements.txt
│ └── setup.py
├── autovizwidget/ ← package 2 (visualization)
│ ├── autovizwidget/
│ │ ├── tests/
│ │ ├── plotlygraphs/ ← area, bar, line, pie, scatter graphs
│ │ ├── widget/ ← AutoVizWidget, EncodingWidget
│ │ └── utils/
│ ├── requirements.txt
│ └── setup.py
├── sparkmagic/ ← package 3 (main package)
│ ├── sparkmagic/
│ │ ├── tests/
│ │ ├── auth/ ← Basic, Kerberos, Custom auth
│ │ ├── controllerwidget/ ← cluster management UI
│ │ ├── kernels/ ← pysparkkernel, sparkkernel, sparkrkernel
│ │ │ └── wrapperkernel/ ← SparkKernelBase
│ │ ├── livyclientlib/ ← core Livy client layer
│ │ ├── magics/ ← IPython magic commands
│ │ ├── serverextension/ ← Jupyter server extension handlers
│ │ └── utils/ ← configuration, constants, logging
│ ├── example_config.json
│ ├── requirements.txt
│ └── setup.py
├── examples/ ← example Jupyter notebooks
├── helm/ ← Kubernetes Helm chart
├── .github/workflows/ ← CI/CD pipelines
├── Dockerfile.jupyter
├── Dockerfile.spark
├── docker-compose.yml
├── pyproject.toml ← Poetry dev environment (not for distribution)
└── .bumpversion.cfg ← synchronized version management

Key Modules#

sparkmagic/livyclientlib/ — Core Livy Client 14#

sparkmagic/magics/ 15#

• remotesparkmagics.py — Defines the %%spark, %%sql, %%scala, %%sparkr cell magics and %spark line magic.
• sparkmagicsbase.py — Base class shared by magic and kernel implementations.

sparkmagic/kernels/ 16#

• pysparkkernel/pysparkkernel.py — PySpark Jupyter kernel
• sparkkernel/sparkkernel.py — Scala Spark Jupyter kernel
• sparkrkernel/sparkrkernel.py — SparkR Jupyter kernel
• wrapperkernel/sparkkernelbase.py — Shared base class for all three kernels

Each kernel directory also contains kernel.json (Jupyter kernel spec) and kernel.js. 17 18 19

sparkmagic/auth/ 20#

• basic.py — HTTP Basic authentication
• kerberos.py — Kerberos/SPNEGO authentication
• customauth.py — Base class for custom auth implementations

Custom auth can be registered via authenticators in config.json. 21

sparkmagic/utils/configuration.py#

Global configuration singleton loaded from ~/.sparkmagic/config.json. Overriding is done in tests via conf.override_all({}). 22

Data Flow#

Jupyter cell input
      ↓
Magic / Kernel (sparkmagicsbase.py)
      ↓
SparkController (sparkcontroller.py)
      ↓
LivyReliableHTTPClient (HTTP + retry)
      ↓
Apache Livy REST API (port 8998)
      ↓
Remote Spark Cluster
      ↓
Response → DataFrameParser → AutoVizWidget → Cell output

Conventions#

Code Style#

• Black is the sole enforced formatter. All code must be formatted before opening a PR — the lint.yaml CI workflow runs black --check on every push and PR and will fail if any file is unformatted. 13

  ⧉

  black . # format in-place
  black --check . # dry-run (what CI runs)
  

• No other linters (flake8, pylint, mypy) are currently configured in CI.

Testing Patterns#

The test suite uses pytest with the mock library (not unittest.mock directly). Tests are organized per-package under <package>/<package>/tests/. 2

Key patterns to follow when writing new tests:

1. Module-level setup/teardown — use setup_function() and teardown_function(), not pytest class fixtures or @pytest.fixture. 23

   ⧉

   def setup_function():
       conf.override_all({}) # always reset config
       global mock_session
       mock_session = MagicMock()
   

2. Global variables for fixtures — test files declare globals at the top and populate them in setup_function(). 24

3. MagicMock for all external dependencies — replace Livy clients, HTTP sessions, IPython displays, etc. with MagicMock(). 25

4. Reset configuration in every setup_function — conf.override_all({}) clears all overrides; use conf.override(key, value) for specific keys. 26

5. Plain assert statements — the codebase uses bare asserts, not pytest assertion helpers. 27

6. No conftest.py — there are no shared fixtures or plugins. Each test file is self-contained. 28

PR and Review Process#

Every pull request should satisfy the checklist in .github/pull_request_template.md 29:

• Write a clear description of what changed and why
• Run black . and commit the result
• Add a bullet point to the top of CHANGELOG.md
• Add or update unit tests
• Manually verify the change in a Jupyter notebook
• For new features: add an example notebook and/or update README.md

Versioning#

• All three packages share a single synchronized version number — currently 0.23.0. 6
• Versions live in each package's __init__.py (__version__ = "0.23.0") and are managed exclusively by .bumpversion.cfg. Never edit version strings by hand.
• Releases are triggered via GitHub Actions: Actions → Release workflow → Run workflow, choosing patch, minor, or major. 30
• The release workflow bumps versions, creates a git tag, generates a GitHub Release from CHANGELOG.md, and then the publish.yml workflow publishes all three packages to PyPI. 31 12

Gotchas & Non-Obvious Workflows#

1. Install order is mandatory#

autovizwidget depends on hdijupyterutils and sparkmagic depends on both. Always install bottom-up. Installing sparkmagic first will fail because its declared dependencies (hdijupyterutils>=0.6, autovizwidget>=0.6) will be pulled from PyPI, not your local editable versions. 32 33

# CORRECT order
pip install -e hdijupyterutils
pip install -e autovizwidget
pip install -e sparkmagic

# WRONG — do not do this
pip install -e sparkmagic # will pull hdijupyterutils from PyPI

2. ~/.sparkmagic/ must exist before running sparkmagic tests#

The sparkmagic package writes logs to ~/.sparkmagic/ and the test suite will error out if the directory is absent. CI creates it explicitly before running tests; you must do the same locally. 7

mkdir -p ~/.sparkmagic
pytest sparkmagic

3. Kerberos C headers are a hard build dependency#

requests-kerberos is in sparkmagic/requirements.txt and has a C extension that requires Kerberos development headers. On a fresh Linux machine, pip install -e sparkmagic will fail without them. 7 8

# Linux
sudo apt-get install -y libkrb5-dev

# macOS
brew install krb5
export LDFLAGS="-L/usr/local/opt/krb5/lib"
export CPPFLAGS="-I/usr/local/opt/krb5/include"

# Conda (avoids the issue entirely)
conda install requests-kerberos -y

4. Docker dev mode defaults to false#

The docker-compose.yml ships with args: dev_mode: "false", which installs sparkmagic from PyPI rather than your local code. You must manually change this to "true" when doing Docker-based development. 10 8

# docker-compose.yml — edit this section:
    build:
      args:
        dev_mode: "true" # ← change from "false"

After changing this, rebuild the image: docker-compose build jupyter.

5. Dockerfile.spark pins an old Spark + Python version#

Dockerfile.spark is based on datamechanics/spark:2.4.7-hadoop-3.1.0-java-8-scala-2.11-python-3.7-latest, which bundles Python 3.7 inside the Spark container and Scala 2.11. The reason is a Livy 0.7.x compatibility constraint with Scala 2.12. This means the Spark-side Python version is different from the 3.8+ requirement for the Python packages. 11

6. Root config.json is intentionally empty#

The config.json at the repository root is an empty placeholder ({}). The actual annotated example configuration is at sparkmagic/example_config.json. 9

7. Root pyproject.toml is for development only, not distribution#

The root Poetry project is named development and is only used to manage the dev virtual environment. The packages are published to PyPI from their individual setup.py files inside each sub-directory. Never run python -m build from the repo root expecting publishable artifacts — do it from within each package directory. 2 12

8. Kernel specs are not auto-registered on install#

pip install sparkmagic does not register the Jupyter kernel specs. You must run jupyter-kernelspec install for each kernel after installation. Without this step, the kernels will not appear in Jupyter's kernel picker. 8

9. No shared test fixtures — follow the module-level pattern#

There is no conftest.py anywhere in the repository. Pytest fixtures are not used. All new tests must follow the existing pattern: declare module-level globals, use setup_function() to initialize them, and use MagicMock for dependencies. Deviating from this pattern creates inconsistency. 28 34

10. All package versions must stay in sync#

.bumpversion.cfg keeps the three packages at the same version. Never manually edit __version__ in any __init__.py — always go through the GitHub Actions release workflow, which uses bumpversion to atomically update all three files and create the git tag. 6 30