AGENTS.md#

A guide for AI coding agents working in the jupyter-incubator/sparkmagic repository. Covers project structure, development setup, key commands, architecture, conventions, and common pitfalls.

Project Overview#

Sparkmagic is a set of tools for interactively working with remote Spark clusters in Jupyter notebooks 1. Rather than requiring Spark to be installed on the Jupyter server, sparkmagic communicates with Spark entirely through a REST server. Three server implementations are currently supported: Livy (for Yarn), Lighter (for Yarn or Kubernetes, PySpark only), and Ilum (for Yarn or Kubernetes) 2.

There are two ways to use sparkmagic 3:

1. Via the IPython kernel — a %%spark magic runs code against a remote cluster from a normal IPython notebook.
2. Via wrapper kernels — dedicated PySpark, Scala (sparkkernel), and SparkR kernels that automatically connect to a remote cluster, execute code and SQL queries, manage Livy sessions, and generate automatic visualizations.

Three-Package Monorepo#

The repository is a monorepo containing three independently-installable Python packages, each with its own setup.py, setup.cfg, and requirements.txt. All three are currently at version 0.23.0. Their dependency chain flows bottom-up:

sparkmagic
  ├── depends on → hdijupyterutils
  └── depends on → autovizwidget
                      └── depends on → hdijupyterutils

The root-level pyproject.toml is a Poetry workspace that pulls all three packages together for local development convenience — it is not the authoritative package definition for any of the three 7.

Development Setup#

Prerequisites#

• Python ≥ 3.8 (CI tests against 3.8, 3.9, 3.10, 3.11, and 3.12) 8

• libkrb5-dev (Linux) — required to build requests-kerberos. On Ubuntu/Debian:

  ⧉

  sudo apt-get install -y libkrb5-dev
  

  On macOS, install via Homebrew: brew install krb5 9

Option A: Direct Editable Installs (Recommended)#

This mirrors what CI does. Install packages in dependency order — order matters:

pip install --upgrade pip
pip install pytest mock

# Install in order: foundation first, then dependents
pip install -r hdijupyterutils/requirements.txt -e hdijupyterutils
pip install -r autovizwidget/requirements.txt -e autovizwidget
pip install -r sparkmagic/requirements.txt -e sparkmagic

10

The -e flag installs each package in editable mode, meaning changes to source files take effect immediately without reinstalling.

Option B: Poetry Virtual Environment#

The root pyproject.toml configures a Poetry workspace that installs all three packages in develop mode in a single step:

poetry install

# If you hit numpy/pandas resolution errors:
poetry run pip install numpy pandas
poetry install

11 7

Post-Install: Create the Config Directory#

The sparkmagic package reads from ~/.sparkmagic/. This directory must exist before running tests:

mkdir -p ~/.sparkmagic

# Optional but helpful: copy the example config
cp sparkmagic/example_config.json ~/.sparkmagic/config.json

12 13

Optional: Register Jupyter Kernels and Server Extension#

To use sparkmagic interactively in Jupyter after installing:

# Find the sparkmagic install location
pip show sparkmagic

# Install wrapper kernels (run from the sparkmagic install location)
jupyter-kernelspec install sparkmagic/kernels/sparkkernel
jupyter-kernelspec install sparkmagic/kernels/pysparkkernel
jupyter-kernelspec install sparkmagic/kernels/sparkrkernel

# Enable the server extension (for programmatic cluster management)
jupyter server extension enable --py sparkmagic

14 15

Docker Development Setup#

For a fully integrated local environment with both Jupyter and a Livy-backed Spark instance:

# Build and start the full stack
docker compose build
docker compose up

This brings up two containers 16:

• spark — Livy 0.7.1 REST server on http://localhost:8998, backed by a local-mode Spark instance
• jupyter — Jupyter notebook on http://localhost:8888, with sparkmagic pre-installed

Inside the notebook, configure your sparkmagic endpoint to http://spark:8998.

Development mode — to test local code changes in Docker without publishing to PyPI, edit docker-compose.yml and set dev_mode to "true", then rebuild:

# docker-compose.yml
services:
  jupyter:
    build:
      args:
        dev_mode: "true" # <-- change this

Then:

docker compose build # rebuild with local editable packages
docker compose up

17 18

In dev mode, the Jupyter container installs local hdijupyterutils, autovizwidget, and sparkmagic with the editable flag, so you can make further edits inside the container for real-time debugging.

Key Commands#

Running Tests#

Tests are run per-package by pointing pytest at the package's top-level directory (not the tests/ subdirectory directly):

# Run all three test suites
pytest hdijupyterutils
pytest autovizwidget
pytest sparkmagic # requires ~/.sparkmagic to exist

# Run a single test file
pytest sparkmagic/sparkmagic/tests/test_livysession.py

# Run tests with verbose output
pytest -v sparkmagic

# Run all tests (via Poetry)
poetry run pytest

19

Note: pytest sparkmagic must be preceded by mkdir -p ~/.sparkmagic if the directory doesn't yet exist, otherwise config-related tests will fail.

Linting and Formatting#

The project uses Black for code formatting. This is enforced as a CI check on every push and PR 20:

# Check formatting (dry run)
black --check .

# Apply formatting
black .

There is no separate lint step beyond Black — no flake8, mypy, or isort configuration is present in the repo.

Build Distributions#

To build PyPI distribution artifacts for publishing (mirrors what the publish CI workflow does):

python -m build hdijupyterutils
python -m build autovizwidget
python -m build sparkmagic

21

Each command produces dist/ inside the respective package directory.

Docker Commands#

# Build Docker images
docker compose build

# Start full stack (Jupyter on :8888, Livy on :8998)
docker compose up

# Start in background
docker compose up -d

# Stop and remove containers
docker compose down

# Rebuild after local code changes (with dev_mode: "true")
docker compose build && docker compose up

22 16

Editable Install (Refresh)#

If you add new files or entry points to a package, you may need to re-run the editable install to pick up changes:

pip install -e hdijupyterutils
pip install -e autovizwidget
pip install -e sparkmagic

Architecture & Structure#

How It Works#

Sparkmagic uses Livy (or a compatible REST server) to remotely execute all user code 23. No Spark components need to be installed on the Jupyter server itself. The data flow is:

Jupyter Notebook Cell
       ↓
sparkmagic kernel / %%spark magic
       ↓ (HTTP REST)
Livy Server ←→ Remote Spark Cluster
       ↓
JSON / plain-text response
       ↓
sparkmagic (deserializes, auto-visualizes via autovizwidget)
       ↓
Rendered output in notebook

Key architectural properties 24:

• No local Spark — code runs entirely on the remote cluster driver via Livy
• Multi-language — Python, Scala, and R kernels are equally featured; sparkmagic serializes all I/O
• Multiple endpoints — a single notebook can open sessions against different clusters in different languages
• Structured data via JSON — SQL query results are serialized to JSON, parsed by sparkmagic, then passed to autovizwidget for visualization or converted to Pandas DataFrames

Repository Layout#

sparkmagic/ # repo root
│
├── hdijupyterutils/ # Package 1
│ ├── hdijupyterutils/ # source
│ │ ├── configuration.py # JSON config reader/writer
│ │ ├── events.py # event publishing
│ │ ├── filehandler.py # MagicsFileHandler for logging
│ │ ├── ipythondisplay.py # IPython display helpers
│ │ ├── logger.py # logging setup
│ │ └── tests/ # unit tests
│ ├── setup.py / setup.cfg
│ └── requirements.txt
│
├── autovizwidget/ # Package 2
│ ├── autovizwidget/ # source
│ │ ├── plotlygraphs/ # Plotly chart renderers
│ │ ├── widget/ # encoding widget (interactive UI)
│ │ ├── utils/ # dataframe utilities
│ │ └── tests/ # unit tests
│ ├── setup.py / setup.cfg
│ └── requirements.txt
│
├── sparkmagic/ # Package 3
│ ├── sparkmagic/ # source
│ │ ├── livyclientlib/ # Livy HTTP client, session, retry policy
│ │ ├── kernels/ # wrapper kernels
│ │ │ ├── pysparkkernel/ # PySpark kernel definition
│ │ │ ├── sparkkernel/ # Scala kernel definition
│ │ │ ├── sparkrkernel/ # SparkR kernel definition
│ │ │ └── wrapperkernel/ # base kernel logic shared by all three
│ │ ├── magics/ # sparkmagics.py — %%spark, %%sql, etc.
│ │ ├── auth/ # kerberos.py, basic.py, customauth.py
│ │ ├── controllerwidget/ # UI widget for cluster/session management
│ │ ├── serverextension/ # /reconnectsparkmagic REST API handler
│ │ ├── utils/ # constants, dataframe utils
│ │ └── tests/ # ~25 unit tests
│ ├── example_config.json # reference configuration
│ ├── setup.py / setup.cfg
│ └── requirements.txt
│
├── examples/ # sample notebooks
├── helm/ # Kubernetes Helm charts
├── screenshots/ # docs screenshots
├── docker-compose.yml
├── Dockerfile.jupyter
├── Dockerfile.spark # Spark + Livy server
├── pyproject.toml # Poetry dev workspace
├── CHANGELOG.md
├── RELEASING.md
└── README.md

25 26 27 28

Configuration#

Sparkmagic reads ~/.sparkmagic/config.json at startup. Key configuration areas 13:

The example config lives at sparkmagic/example_config.json 13. In Docker dev mode, the Jupyter container copies it to ~/.sparkmagic/config.json and replaces localhost with spark (the Docker hostname) 29.

Server Extension#

Sparkmagic registers a Jupyter server extension that exposes a /reconnectsparkmagic POST endpoint. This allows programmatic cluster switching — useful for multi-tenant environments and orchestration tools 30. It is enabled via:

jupyter server extension enable --py sparkmagic

31

Development Conventions#

Code Style#

• Formatter: Black — all Python code must be formatted with Black. This is enforced automatically on every push and pull request via GitHub Actions 20.
• Run black . from the repo root before committing. The CI lint check will fail on any unformatted code.
• No separate flake8, mypy, or isort configurations exist in this repository.

Testing Practices#

Tests live alongside source code inside each package:

<pkg>/<pkg>/tests/test_<module>.py

For example: sparkmagic/sparkmagic/tests/test_livysession.py 32.

Test patterns to follow:

• Use pytest as the test framework; functions are named test_<behavior>() 33
• Use setup_function() and teardown_function() for module-level test setup/teardown 34
• Use MagicMock from the mock library for all dependency injection and object stubbing 35
• Use pytest.raises() for expected exception testing 36
• For parallel-safe test classes, add _multiprocess_can_split_ = True class attribute 37
• New tests go in the tests/ subdirectory of the relevant package — not a top-level tests/ directory

When to add tests: Every code change should include corresponding unit tests. The PR checklist explicitly requires it 38.

Pull Request Workflow#

The .github/pull_request_template.md defines the required checklist for every PR 39:

1. Write a description of what changed and why
2. Format with Black — run black . and commit the result
3. Update CHANGELOG.md — add a bullet point at the top describing the change
4. Add or modify unit tests — all behavior changes need test coverage
5. Manually test with a notebook — integration-test your change end-to-end in Jupyter
6. For new features: add an example notebook in examples/ and/or update README.md

CI runs tests on Python 3.8 through 3.12 8. All test matrix versions must pass before merge.

Naming Conventions#

• Test files: test_<module_name>.py (mirrors the source file being tested)
• Test functions: test_<behavior_being_tested>()
• Auth plugins: implement in sparkmagic/auth/ following the pattern of existing modules (basic.py, kerberos.py, customauth.py)
• New kernels: follow the <language>kernel/ subdirectory structure under sparkmagic/kernels/

Versioning#

All three packages are versioned together (currently 0.23.0). Version numbers live in:

• hdijupyterutils/hdijupyterutils/__init__.py 40
• autovizwidget/autovizwidget/__init__.py 41
• sparkmagic/sparkmagic/__init__.py 42

Releases are triggered manually via the GitHub Actions Release workflow 43, which bumps all versions using bumpversion and tags the commit. Do not manually edit version numbers — use the release workflow.

Gotchas & Common Issues#

1. Install Order Is Mandatory#

The three packages have hard inter-package dependencies. Installing them in the wrong order will fail:

# ❌ Wrong — autovizwidget requires hdijupyterutils
pip install -e autovizwidget

# ✓ Correct order
pip install -r hdijupyterutils/requirements.txt -e hdijupyterutils
pip install -r autovizwidget/requirements.txt -e autovizwidget
pip install -r sparkmagic/requirements.txt -e sparkmagic

44 5

2. ~/.sparkmagic Directory Must Exist Before Running Tests#

The sparkmagic test suite will fail with file-not-found errors if ~/.sparkmagic/ does not exist. Always create it first:

mkdir -p ~/.sparkmagic

This is why CI explicitly runs mkdir ~/.sparkmagic before pytest sparkmagic 12. This is not needed for hdijupyterutils or autovizwidget tests.

3. libkrb5-dev Is Required on Linux#

The sparkmagic package depends on requests-kerberos, which requires Kerberos development headers. Without them, pip install -r sparkmagic/requirements.txt will fail to build on Linux:

sudo apt-get install -y libkrb5-dev # Ubuntu/Debian

9

On macOS: brew install krb5 and set the relevant env vars. In Docker, Kerberos is installed via conda 45.

4. pytest sparkmagic — Not pytest sparkmagic/sparkmagic/tests#

pytest should be pointed at the package root directory, not the tests subdirectory. The CI commands are:

pytest hdijupyterutils # ✓
pytest autovizwidget # ✓
pytest sparkmagic # ✓

# also works, but not the canonical form:
# pytest sparkmagic/sparkmagic/tests

19

5. Docker dev_mode Must Be a String "true", Not a Boolean#

The docker-compose.yml passes dev_mode as a build arg to Dockerfile.jupyter. The Dockerfile checks if [ "$dev_mode" = "true" ] — it expects the literal string "true", not YAML boolean true:

# docker-compose.yml
args:
  dev_mode: "true" # ✓ string
  # dev_mode: true # ✗ YAML boolean — will NOT enable dev mode

46 47

After changing dev_mode, you must run docker compose build — docker compose up alone will not pick up the change.

6. pyproject.toml Is a Dev Convenience, Not the Package Source of Truth#

The root pyproject.toml is a Poetry workspace for local development that wraps all three packages. It is not the authoritative source for any package's metadata, dependencies, or entry points. Those live in each package's setup.py/setup.cfg:

• hdijupyterutils/setup.py and hdijupyterutils/setup.cfg 48
• autovizwidget/setup.py and autovizwidget/setup.cfg 49
• sparkmagic/setup.py and sparkmagic/setup.cfg 50

When modifying dependencies or entry points, edit the per-package files — not pyproject.toml.

7. Editable Installs Don't Auto-Pick-Up New Package Data#

If you add new package_data entries (e.g., new kernel JSON files), you need to re-run pip install -e <package> to re-register them. Simply editing a source file is sufficient for code changes, but not for resource/data file additions.

8. The Docker Spark Image Uses an Old Spark Version (Intentionally)#

Dockerfile.spark is pinned to Spark 2.4.7 and Livy 0.7.1 51. This is intentional — Livy 0.7.1 requires Scala 2.11, which constrains the Spark version. Do not upgrade these without verifying Livy compatibility.

9. Livy Endpoint Is spark:8998 Inside Docker, localhost:8998 Outside#

The example config at sparkmagic/example_config.json defaults Livy to http://localhost:8998 52. The Docker container's Jupyter image patches this automatically with sed -i 's/localhost/spark/g' 53. If you're configuring sparkmagic manually inside Docker, use http://spark:8998 as the endpoint 54.

10. Three Separate PyPI Packages, But One Repo Version Tag#

Despite being a monorepo, each package is published to PyPI separately with independent tokens. The release workflow bumps all three versions simultaneously using bumpversion, so they stay in sync 43 55. If you need to test a local version of just one package against the published versions of the others, install the local one with -e and the others normally — but be careful about version constraint mismatches in requirements.txt.