AGENTS.md#

A concise guide for AI coding agents working in this repository.

Project Overview#

Sparkmagic is a set of tools for interactively working with remote Spark clusters in Jupyter notebooks. Rather than running Spark locally, sparkmagic communicates with a remote Spark cluster through a REST server — currently supporting Livy, Lighter, and Ilum 1.

This is a three-package Python monorepo. All packages share the same version number (currently 0.23.0) and are released together 2.

The Three Packages#

Key Features#

• Execute Spark code in Python, Scala, and R from a Jupyter notebook 3
• Automatic SparkContext (sc) and HiveContext (sqlContext) creation
• %%sql magic with automatic query result visualization
• %%info for Spark application info and logs
• Capture SQL results as Pandas DataFrames for local analysis
• Send local files or DataFrames to a remote cluster
• Basic Access and Kerberos authentication to Livy

Repository Structure#

sparkmagic/ ← repo root
├── hdijupyterutils/ ← Package 1: shared utilities
│ ├── hdijupyterutils/ ← Python source
│ │ ├── tests/
│ │ ├── configuration.py
│ │ ├── events.py / eventshandler.py
│ │ ├── filesystemreaderwriter.py
│ │ ├── ipythondisplay.py / ipywidgetfactory.py
│ │ ├── log.py / utils.py / guid.py / constants.py
│ │ └── ...
│ ├── requirements.txt
│ └── setup.py / setup.cfg
├── autovizwidget/ ← Package 2: auto-visualization
│ ├── autovizwidget/
│ │ ├── tests/
│ │ ├── plotlygraphs/ ← graphbase, linegraph, bargraph, areagraph, piegraph, scattergraph, graphrenderer
│ │ ├── widget/ ← autovizwidget.py, encodingwidget.py, encoding.py
│ │ └── utils/
│ ├── requirements.txt
│ └── setup.py / setup.cfg
├── sparkmagic/ ← Package 3: main package
│ ├── sparkmagic/
│ │ ├── tests/ ← 26 test files
│ │ ├── auth/ ← customauth.py, basic.py, kerberos.py
│ │ ├── controllerwidget/ ← Jupyter UI widgets for cluster/session management
│ │ ├── kernels/ ← pysparkkernel/, sparkkernel/, sparkrkernel/, wrapperkernel/
│ │ ├── livyclientlib/ ← HTTP client, session mgmt, retry policies, command execution
│ │ ├── magics/ ← remotesparkmagics.py, sparkmagicsbase.py
│ │ ├── serverextension/ ← /reconnectsparkmagic endpoint
│ │ └── utils/ ← configuration.py, constants.py
│ ├── example_config.json
│ ├── requirements.txt
│ └── setup.py / setup.cfg
├── examples/ ← sample notebooks
├── helm/ ← Kubernetes Helm charts
├── pyproject.toml ← Poetry dev wrapper (NOT a publishable package)
├── docker-compose.yml
├── Dockerfile.jupyter
├── Dockerfile.spark
└── .github/workflows/tests.yml

Package Dependency Chain#

sparkmagic → autovizwidget → hdijupyterutils
             ↗
sparkmagic → hdijupyterutils

hdijupyterutils has no internal dependencies. autovizwidget depends on hdijupyterutils. sparkmagic depends on both 4 5. Install in this order.

Root-Level pyproject.toml#

The pyproject.toml at the repo root is a Poetry convenience wrapper for development only — it declares all three local packages as path dependencies with develop = true 6. It is not the packaging configuration for any of the three distributed packages. Each package's own setup.py/setup.cfg governs what gets published to PyPI.

Development Setup#

Prerequisites#

• Python 3.8–3.12 7

• libkrb5-dev on Ubuntu/Debian (required for Kerberos support — requests-kerberos will fail to build without it)

  ⧉

  sudo apt-get install -y libkrb5-dev
  

Option A: Direct Editable Install (Recommended for most contributors)#

Install the three packages in dependency order using pip's editable (-e) flag 8:

git clone https://github.com/jupyter-incubator/sparkmagic
cd sparkmagic

pip install -e hdijupyterutils
pip install -e autovizwidget
pip install -e sparkmagic

Order matters. Each package depends on the ones before it. Installing sparkmagic before hdijupyterutils will fail.

To also install test tools:

pip install pytest mock

Or install with requirements files (mirrors CI exactly):

pip install -r hdijupyterutils/requirements.txt -e hdijupyterutils
pip install -r autovizwidget/requirements.txt -e autovizwidget
pip install -r sparkmagic/requirements.txt -e sparkmagic

Option B: Poetry#

Use Poetry to create an isolated virtual environment with all three packages 9:

poetry install

If you encounter issues with numpy or pandas during poetry install:

poetry run pip install numpy pandas
poetry install

Run commands in the Poetry-managed environment:

poetry run pytest
poetry run jupyter notebook

Docker Development Environment#

Docker provides a self-contained stack: JupyterLab on port 8888 + a Livy server on port 8998 backed by a local-mode Spark instance 10.

docker compose build
docker compose up
# Access Jupyter at http://localhost:8888
# Use http://spark:8998 as the Livy endpoint inside notebooks

Dev mode — to test local code changes without pushing to PyPI, edit docker-compose.yml and set 11:

args:
  dev_mode: "true"

Then rebuild:

docker compose build
docker compose up

With dev_mode: "true", the container installs all three packages in editable mode (-e) from the copied source, enabling real-time debugging inside JupyterLab 12.

To stop:

# Ctrl-C to interrupt, then optionally:
docker compose down

One-Time Runtime Config#

Sparkmagic looks for its config at ~/.sparkmagic/config.json. This directory is not created automatically by the install. Create it manually before running tests or starting Jupyter:

mkdir -p ~/.sparkmagic
# Optionally copy the example config:
cp sparkmagic/example_config.json ~/.sparkmagic/config.json

Key Commands#

Install#

# Editable install of all three packages (in dependency order)
pip install -e hdijupyterutils
pip install -e autovizwidget
pip install -e sparkmagic

# With explicit requirements (mirrors CI)
pip install -r hdijupyterutils/requirements.txt -e hdijupyterutils
pip install -r autovizwidget/requirements.txt -e autovizwidget
pip install -r sparkmagic/requirements.txt -e sparkmagic

# Install test dependencies
pip install pytest mock

# Poetry (all-in-one)
poetry install

Testing#

# Run each package's tests separately (preferred — mirrors CI)
pytest hdijupyterutils
pytest autovizwidget

mkdir -p ~/.sparkmagic # required before sparkmagic tests
pytest sparkmagic

# With Poetry
poetry run pytest hdijupyterutils
poetry run pytest autovizwidget
poetry run pytest sparkmagic

Note: Pass the package directory name to pytest, not the inner source directory. pytest sparkmagic is correct; pytest sparkmagic/sparkmagic/tests/ also works but is less idiomatic.

Docker#

# Build images
docker compose build

# Start stack (Jupyter :8888, Livy :8998)
docker compose up

# Stop and remove containers
docker compose down

Kernel & Extension Registration#

# Find the sparkmagic install path
pip show sparkmagic

# Install kernels (run from the sparkmagic install location)
jupyter-kernelspec install sparkmagic/kernels/sparkkernel
jupyter-kernelspec install sparkmagic/kernels/pysparkkernel
jupyter-kernelspec install sparkmagic/kernels/sparkrkernel

# Enable server extension
jupyter server extension enable --py sparkmagic

Release (maintainers only)#

Releases are fully automated via GitHub Actions 13. Go to Actions → Release → Run workflow, select the bump type (patch / minor / major), and confirm. The workflow bumps all three package versions together using .bumpversion.cfg.

Testing#

Overview#

Each of the three packages has its own test suite. Tests must be run per package by passing the package directory name to pytest. There is no single root-level pytest configuration — test discovery is fully automatic 14.

The Key Commands section above covers the exact pytest invocations. This section explains the test infrastructure in more detail.

Running Tests#

# 1. Prepare the sparkmagic config dir (required for sparkmagic tests only)
mkdir -p ~/.sparkmagic

# 2. Run each package's tests
pytest hdijupyterutils # 6 test files
pytest autovizwidget # 7 test files
pytest sparkmagic # 26 test files

Test Layout#

Tests live inside the package's source directory, not at the package root:

hdijupyterutils/hdijupyterutils/tests/ ← 6 test files
autovizwidget/autovizwidget/tests/ ← 7 test files
sparkmagic/sparkmagic/tests/ ← 26 test files (covers magics, livyclientlib, kernels, auth, etc.)

Each tests/ directory contains an empty __init__.py 15 16 17.

Test Infrastructure#

• Framework: pytest + mock library only. No conftest.py, pytest.ini, tox.ini, or custom plugins 18.
• Fixtures: Tests use standard setup_function() / teardown_function() pytest conventions for per-test setup.
• Mocking: The mock library is used extensively for isolating Livy HTTP calls and IPython display interactions.

CI Matrix#

GitHub Actions runs tests against Python 3.8, 3.9, 3.10, 3.11, and 3.12 on every push and pull request 7. CI installs libkrb5-dev before running tests to support Kerberos-related packages.

System Dependency#

On Ubuntu/Debian, Kerberos headers are required:

sudo apt-get install -y libkrb5-dev

Without this, pip install -r sparkmagic/requirements.txt will fail because requests-kerberos requires native Kerberos headers to build.

Architecture & Conventions#

High-Level Architecture#

Sparkmagic executes all Spark code remotely via a Livy REST API — no Spark components are required on the Jupyter server 19. The flow is:

Jupyter Notebook / Magic
        ↓ (submit code/SQL as JSON)
  livyclientlib (REST client)
        ↓ (HTTP to :8998)
  Livy / Lighter / Ilum (REST server)
        ↓
  Remote Spark Cluster

Results are returned as plain text or JSON, deserialized by sparkmagic, and rendered as formatted text or Pandas DataFrames.

Module Organization#

Each package follows a consistent internal layout: <package_name>/<package_name>/ contains the source, with tests/ as a subdirectory:

<package>/
  <package>/ ← importable module
    tests/ ← test files
    *.py ← implementation
  setup.py
  requirements.txt
  setup.cfg

Key Modules in sparkmagic#

Authentication Pattern#

Authentication is pluggable 20. The Authenticator base class in auth/customauth.py is a requests-compatible auth object. Subclass it and override __call__(request) to attach credentials. Built-in implementations: basic.py and kerberos.py. Custom authenticators are registered in ~/.sparkmagic/config.json under the "authenticators" key.

Kernel Pattern#

Three language-specific kernels (pysparkkernel, sparkkernel, sparkrkernel) are thin wrappers around a common wrapperkernel base 21. Each kernel directory includes a kernel.json for Jupyter kernel registration. The kernels communicate with Livy using the livyclientlib session management layer.

Configuration#

Runtime configuration is read from ~/.sparkmagic/config.json. The sparkmagic/utils/configuration.py module provides typed accessors for all config keys. Configuration can also be overridden programmatically in a notebook:

import sparkmagic.utils.configuration as conf
conf.override('cleanup_all_sessions_on_exit', True)

Versioning & Code Style#

• All three packages share one semantic version, bumped together via .bumpversion.cfg 2.
• Code style: Black (enforced, see README badge).
• pandas < 3.0.0 is pinned across all packages 4 5 22.

Gotchas & Important Notes#

1. Install Order Is Non-Negotiable#

Always install the three packages in dependency order: hdijupyterutils → autovizwidget → sparkmagic. Installing sparkmagic first will fail because it imports from the other two at install time 4 5.

# ✅ Correct
pip install -e hdijupyterutils
pip install -e autovizwidget
pip install -e sparkmagic

# ❌ Will fail
pip install -e sparkmagic # hdijupyterutils and autovizwidget not installed yet

2. ~/.sparkmagic Must Exist Before Running sparkmagic Tests#

The sparkmagic test suite reads from ~/.sparkmagic/ at import time. If the directory does not exist, tests will error out immediately. This is a required manual step — nothing creates this directory automatically 23:

mkdir -p ~/.sparkmagic
pytest sparkmagic

3. Pass Package Directory Names to pytest, Not Inner Test Paths#

The official pattern is pytest <package_dir>, not pytest <package_dir>/<package_dir>/tests/. Both work, but the directory-level invocation correctly mirrors CI and picks up all tests via pytest's automatic discovery 24:

pytest hdijupyterutils # ✅ matches CI
pytest autovizwidget # ✅ matches CI
pytest sparkmagic # ✅ matches CI

4. Docker Dev Mode Still Requires a Rebuild for New Files#

Setting dev_mode: "true" installs packages with -e inside the container, so edits to already-copied files are reflected live. However, the COPY instruction in Dockerfile.jupyter snapshots the local source at build time 25. If you add new files or make structural changes, you must re-run docker compose build before docker compose up.

5. requests-kerberos Needs Native Kerberos Headers on Linux#

pip install requests-kerberos fails on Debian/Ubuntu without libkrb5-dev. Docker works around this by using conda install requests-kerberos -y instead of pip 26. For local development on Linux, install the system package first:

sudo apt-get install -y libkrb5-dev

6. Root pyproject.toml Is Not a Publishable Package#

The pyproject.toml at the repo root is a Poetry development environment wrapper named "development" 27. It is not the build configuration for any of the three packages. Each package is independently published to PyPI from its own setup.py. Do not modify the root pyproject.toml expecting it to affect package metadata on PyPI.

7. All Packages Are Versioned Together#

Version bumps must update all three packages simultaneously. Use .bumpversion.cfg or the GitHub Actions release workflow. Never manually edit __version__ in just one package — they must stay in sync 2.

8. Docker Livy Endpoint Is http://spark:8998, Not localhost#

Inside the Docker stack, the Jupyter service resolves the Livy server by its Docker Compose service hostname spark. When configuring a sparkmagic endpoint from inside the JupyterLab container, use http://spark:8998, not http://localhost:8998 28 29.

9. conf.override for Session Cleanup Must Be Called Before %load_ext#

If you set cleanup_all_sessions_on_exit via conf.override(...), this must happen before loading the sparkmagic extension in a notebook. Calling it after %load_ext sparkmagic.magics has no effect 30.