AGENTS.md#

A concise guide for AI coding agents working in the sparkmagic repository.

Project Overview#

Sparkmagic is a set of tools for interactively working with remote Spark clusters in Jupyter notebooks. Rather than requiring a local Spark installation, sparkmagic communicates with a remote Spark cluster entirely through a REST server, meaning no Spark components need to be installed on the Jupyter server 1.

Three REST server backends are currently supported 2:

• Livy – for running interactive sessions on YARN
• Lighter – for YARN or Kubernetes (PySpark only)
• Ilum – for YARN or Kubernetes

Two Usage Modes#

1. IPython Kernel magics – Load sparkmagic.magics in a regular IPython notebook and use the %spark / %%spark cell magic to execute code against a remote cluster 3.
2. Dedicated Spark kernels – Select a PySpark, Spark (Scala), or SparkR kernel when creating a notebook. The kernel auto-connects to a preconfigured Livy endpoint and exposes magics like %%sql, %%local, %%configure, %%info, and %%logs 4.

Key Features#

• Run Spark code in Python, Scala, and R against any remote Spark cluster 5
• Automatic SparkContext (sc) and HiveContext (sqlContext) creation
• %%sql magic for SparkSQL with automatic Plotly-based visualization
• Capture SQL output as Pandas DataFrames for local Python processing
• Send local files or DataFrames to the remote cluster
• Authentication via Basic Access, Kerberos, or custom authenticators
• Configurable retry policies and heartbeat monitoring for reliable remote sessions

Repository Structure#

The repository is a monorepo containing three independently installable Python packages 6:

Additional top-level resources include examples/ (sample notebooks), helm/ (Kubernetes Helm charts), and Docker files for local development 7.

Key Commands#

Installing Dependencies#

The three sub-packages must be installed in dependency order. Use editable installs for development so code changes are picked up immediately 8:

pip install -e hdijupyterutils
pip install -e autovizwidget
pip install -e sparkmagic

Alternatively, use Poetry to set up a fully isolated virtual environment from the root 9:

poetry install
# If numpy/pandas fail to install, run these first then retry:
# poetry run pip install numpy pandas
# poetry install

For Kerberos authentication support (required by CI and on Linux systems) 10:

sudo apt-get install -y libkrb5-dev

Running Tests#

Tests are organized per sub-package. The ~/.sparkmagic directory must exist before running the sparkmagic test suite 11:

pip install pytest mock

pytest hdijupyterutils
pytest autovizwidget

mkdir -p ~/.sparkmagic # Required for sparkmagic tests
pytest sparkmagic

With Poetry 12:

poetry run pytest

Linting#

The project enforces the Black code formatter. All PRs are checked automatically in CI 13:

# Check formatting (CI mode):
black --check .

# Auto-format:
black .

Installing Jupyter Kernels#

After installing sparkmagic, register the dedicated Spark kernels with Jupyter 14:

# Find install path first:
pip show sparkmagic

# Then cd to that path and run:
jupyter-kernelspec install sparkmagic/kernels/sparkkernel
jupyter-kernelspec install sparkmagic/kernels/pysparkkernel
jupyter-kernelspec install sparkmagic/kernels/sparkrkernel

Enabling the Server Extension#

The server extension enables programmatic cluster switching via the /reconnectsparkmagic API 15:

# Jupyter 7.x / JupyterLab 3.x+:
jupyter server extension enable --py sparkmagic

# Older Jupyter (5.2 or earlier) / JupyterLab 1 or 2:
jupyter serverextension enable --py sparkmagic

Docker Development Environment#

The Docker Compose setup spins up a full Jupyter + Livy + Spark stack for local development and testing 16:

docker compose build
docker compose up # Access Jupyter at http://localhost:8888
                         # Livy endpoint: http://spark:8998
docker compose down # Tear down containers

To test local code changes inside Docker, set dev_mode: true in docker-compose.yml before building. This installs all three sub-packages in editable mode inside the container.

Releasing#

Releases use the GitHub Actions Release workflow with semantic versioning. The steps are 17:

1. Update CHANGELOG.md on the master branch
2. In GitHub Actions → Release workflow → Run workflow → select patch | minor | major

Architecture#

Sparkmagic acts as a bridge between the local Jupyter environment and a remote Spark cluster, delegating all code execution to the cluster via a REST API 18.

┌─────────────────────────────┐ REST API ┌─────────────────────┐
│ Jupyter Notebook │ ──────────────────────▶│ Livy / Lighter / │
│ (magics or Spark kernel) │◀────────────────────── │ Ilum REST Server │
└─────────────────────────────┘ JSON results/status └──────────┬──────────┘
                                                                   │
                                                        ┌──────────▼──────────┐
                                                        │ Remote Spark │
                                                        │ Cluster (YARN / │
                                                        │ Kubernetes) │
                                                        └─────────────────────┘

Package Layout#

sparkmagic/ – Core Package 19#

sparkmagic/sparkmagic/
├── magics/ # IPython magic implementation
│ ├── remotesparkmagics.py # %spark magic (RemoteSparkMagics)
│ └── sparkmagicsbase.py # Shared base class (SparkMagicBase)
├── kernels/ # Dedicated Spark kernel wrappers
│ ├── kernelmagics.py # Kernel-specific magics (%%spark, %%sql, %%local, etc.)
│ ├── pysparkkernel/ # PySpark kernel
│ ├── sparkkernel/ # Scala kernel
│ ├── sparkrkernel/ # SparkR kernel
│ └── wrapperkernel/ # Generic wrapper kernel
├── livyclientlib/ # REST client layer
│ ├── sparkcontroller.py # Session orchestration & command routing
│ ├── livysession.py # Remote session state, heartbeat thread
│ ├── livyreliablehttpclient.py # HTTP client with retry policies
│ ├── command.py # Wraps and executes user code statements
│ ├── sqlquery.py # SQL query translation and execution
│ ├── sessionmanager.py # Local session registry
│ └── endpoint.py # URL + auth credential holder
├── auth/ # Authentication providers
│ ├── customauth.py # Base Authenticator class (no-auth default)
│ ├── basic.py # HTTP Basic Auth
│ └── kerberos.py # Kerberos Auth (via requests-kerberos)
├── serverextension/ # Jupyter server extension
│ └── handlers.py # Tornado ReconnectHandler (/reconnectsparkmagic)
├── controllerwidget/ # IPython widget UI for session management
└── utils/ # Shared utilities
    ├── configuration.py # Config loading, defaults, and overrides
    ├── constants.py # UPPER_SNAKE_CASE constants (session kinds, etc.)
    ├── sparkevents.py # Pluggable event emission system
    ├── utils.py # Argument parsing, DataFrame conversion helpers
    ├── dataframe_parser.py # Livy JSON → Pandas DataFrame conversion
    └── sparklogger.py # Logging setup

autovizwidget/ – Automatic Visualization 20#

Provides Plotly-based chart widgets that auto-render SQL query results. Contains plotlygraphs/ and widget/ sub-directories.

hdijupyterutils/ – Shared Utilities 21#

Provides configuration.py, events.py, ipythondisplay.py, ipywidgetfactory.py, log.py, and filesystem helpers shared by all three packages.

Key Execution Flow#

1. User executes a %%spark cell or Spark kernel cell in Jupyter
2. The magic (or kernel magic) parses arguments and retrieves the Endpoint (URL + auth) 22
3. SparkController looks up or creates a LivySession for that endpoint 23
4. User code is wrapped in a Command (or SQLQuery for %%sql) object 24
5. LivyReliableHttpClient POSTs the statement to Livy's /sessions/{id}/statements endpoint 25
6. The client polls the statement status until completion, then fetches results
7. Results (plain text, HTML tables, or image data) are rendered in the notebook output cell

Authentication Architecture#

All authenticators inherit from the Authenticator base class and implement __call__(request) to attach credentials to outgoing HTTP requests 26. The three built-in providers — None, Basic_Access, and Kerberos — are registered by dotted-path string in the config file 27. Custom authenticators can be added by registering their dotted path in ~/.sparkmagic/config.json 28.

Configuration System#

Runtime configuration is managed by sparkmagic/utils/configuration.py and loaded from ~/.sparkmagic/config.json. See sparkmagic/example_config.json for a fully annotated reference 29. Key configuration areas:

Conventions#

Code Style#

• Black is the enforced formatter. All code must pass black --check . before merging. CI will fail on unformatted code 13.
• There is no separate flake8/pylint configuration; Black is the sole style gate.

Naming Conventions#

• Constants use UPPER_SNAKE_CASE and live in sparkmagic/utils/constants.py 30.
• Classes use PascalCase (e.g., LivySession, SparkController, RemoteSparkMagics).
• Functions and variables use snake_case.
• Test files are named test_*.py and live in the tests/ subdirectory of each package 31.

File Organization#

• Each of the three sub-packages (sparkmagic, autovizwidget, hdijupyterutils) is self-contained with its own setup.py, setup.cfg, requirements.txt, and MANIFEST.in 19.
• Shared logic between packages flows in one direction: hdijupyterutils → autovizwidget → sparkmagic. Never add reverse dependencies.
• All configuration keys and their default values are defined in sparkmagic/utils/configuration.py 32. Add new config options here with sensible defaults.
• Constants referenced across modules must be defined in sparkmagic/utils/constants.py rather than inlined as string literals 30.

Testing Conventions#

Tests use pytest with the mock library 33. The standard pattern is:

from mock import MagicMock
import pytest
import sparkmagic.utils.configuration as conf

def setup_function():
    # Reset config state before every test
    conf.override_all({})
    # Initialize shared mock objects here

def test_something():
    # Arrange
    mock_client = MagicMock()
    # Act
    result = some_function(mock_client)
    # Assert
    assert result == expected
    mock_client.some_method.assert_called_once_with(expected_arg)

Key testing conventions 34:

• Use module-level setup_function() (not class-based setUp) to initialize state before each test.
• Always call conf.override_all({}) in setup_function() to prevent config state from leaking between tests.
• Use MagicMock for test doubles; verify interactions with assert_called_once_with() and similar methods.
• Use standard Python assert statements (not assertEqual-style).

Pull Request Conventions#

PRs should follow this checklist 35 36:

• Write a clear description of the changes
• Format code with black
• Add an entry to CHANGELOG.md under the "Next Release" section
• Add or modify unit tests to cover the change
• Manually test with a notebook if the change affects user-facing behavior
• Update documentation if the change adds a new feature

Changelog and Versioning#

• The CHANGELOG.md has a permanent "Next Release" section at the top for unreleased changes 37.
• Each release section is organized under ### Updates and ### Bug Fixes headings.
• The project follows semantic versioning (MAJOR.MINOR.PATCH). Version is single-sourced from sparkmagic/sparkmagic/__init__.py and managed with .bumpversion.cfg.
• Releases are cut via the GitHub Actions Release workflow — do not bump the version manually 17.

Configuration Override Pattern#

To programmatically override config in notebooks or tests 38:

import sparkmagic.utils.configuration as conf

# Override a single key:
conf.override('max_results_sql', 5000)

# Override multiple keys at once (useful in tests):
conf.override_all({'max_results_sql': 5000, 'use_auto_viz': False})

Gotchas#

1. ~/.sparkmagic directory must exist#

Sparkmagic expects the directory ~/.sparkmagic to exist at runtime and will fail if it does not. The CI workflow explicitly creates it before running the test suite 39. Always run:

mkdir -p ~/.sparkmagic

Place your config.json there. Use sparkmagic/example_config.json as a starting template 29.

2. All code executes remotely — %%local for client-side work#

No Spark runs locally. Every cell sent through %%spark or a Spark kernel is serialized and executed on the remote driver via Livy 40. This means:

• You cannot directly inspect Python objects from a Spark cell in local Python scope.
• Use the %%local magic to execute code in the local IPython kernel.
• Structured data returned from Spark is serialized to JSON by sparkmagic before arriving in the notebook; large result sets can be slow.

3. pandas is pinned to < 3.0.0#

The sparkmagic/requirements.txt pins pandas<3.0.0 41. Do not upgrade past this limit without verifying DataFrame parsing compatibility throughout livyclientlib/ and autovizwidget/.

4. Kerberos requires libkrb5-dev and an active ticket#

Installing requests_kerberos is not enough. The system package libkrb5-dev must be installed (sudo apt-get install -y libkrb5-dev) and a valid Kerberos ticket must be present (via kinit) before sparkmagic can authenticate 42. The CI workflow installs libkrb5-dev as an explicit step 43.

5. ignore_ssl_errors defaults to false — keep it that way#

The config key ignore_ssl_errors is false by default 44. Setting it to true disables certificate verification for all Livy requests. Never enable this in production environments.

6. conf.override() must be called before %load_ext sparkmagic.magics#

Some configuration values (e.g., cleanup_all_sessions_on_exit) are read once during sparkmagic initialization. Overriding them after the extension is loaded has no effect 45:

# ✅ Correct order:
import sparkmagic.utils.configuration as conf
conf.override('cleanup_all_sessions_on_exit', True)
%load_ext sparkmagic.magics

# ❌ Too late — override is ignored:
%load_ext sparkmagic.magics
conf.override('cleanup_all_sessions_on_exit', True)

7. VSCode-Jupyter bootstrapping code must not be wrapped in %%spark#

When using VS Code's Jupyter extension, the runtime may inject bootstrapping code containing _VSCODE_ markers into a cell. If sparkmagic wraps this code and sends it to Livy, the notebook hangs indefinitely 35. Any code-injection logic that pre-processes cells must check for and exclude _VSCODE_-marked content.

8. Leading whitespace before magic commands causes misclassification#

A cell that starts with blank lines before a magic command (e.g., %%configure) can be misclassified by sparkmagic's parser, causing incorrect execution behavior 36. When writing code that programmatically constructs or submits cells, normalize leading whitespace before parsing.

9. session_configs vs. session_configs_defaults#

These two config keys behave differently when a user runs %%configure 46:

• session_configs: The entire dictionary is replaced when the user changes any session setting.
• session_configs_defaults: Individual keys are only overridden when explicitly specified. Use this for settings (like Spark catalog type) that should persist unless the user takes deliberate action to change them.

10. Sub-package install order matters#

The three packages have a hard dependency chain: hdijupyterutils → autovizwidget → sparkmagic. Always install them in this order 47. Installing sparkmagic first will fail because hdijupyterutils and autovizwidget are not yet available.

11. Livy session startup timeout may need tuning#

The default livy_session_startup_timeout_seconds is 60 seconds 48. Slow or heavily loaded clusters may need a larger value. If you see spurious session-creation timeouts, increase this in ~/.sparkmagic/config.json.

12. Docker Livy endpoint is http://spark:8998, not localhost#

Inside the Docker Compose development environment, the Livy service is reachable at http://spark:8998 (the Docker Compose service name), not at http://localhost:8998 49. Using localhost inside a notebook running in Docker will fail to connect.

13. Poetry numpy/pandas install may fail on first run#

On some platforms, poetry install can fail when resolving numpy or pandas. The documented workaround is 50:

poetry run pip install numpy pandas
poetry install

14. Python version support is 3.8–3.12#

The CI matrix tests Python 3.8 through 3.12 51. Do not use language features or library APIs that require Python 3.13+. The pyproject.toml specifies python = "^3.8" 52.