Building Extension Modules#

{{% warning %}}
While this feature has been around since almost the beginning of the Poetry project and has needed minimal changes,
it is still considered unstable. You can participate in the discussions about stabilizing this feature
here.

And as always, your contributions towards the goal of improving this feature are also welcome.
{{% /warning %}}

Poetry allows a project developer to introduce support for, build and distribute native extensions within their project.
In order to achieve this, at the highest level, the following steps are required.

{{< steps >}}
{{< step >}}
Add Build Dependencies

The build dependencies, in this context, refer to those Python packages that are required in order to successfully execute
your build script. Common examples include cython, meson, maturin, setuptools etc., depending on how your
extension is built.

{{% note %}}
You must assume that only Python built-ins are available by default in a build environment. This means, if you need
even packages like setuptools, it must be explicitly declared.
{{% /note %}}

The necessary build dependencies must be added to the build-system.requires section of your pyproject.toml file.

[build-system]
requires = ["poetry-core", "setuptools", "cython"]
build-backend = "poetry.core.masonry.api"

{{% note %}}
It is recommended that you consider specifying version constraints to all entries in build-system.requires in order to
avoid surprises if one of the packages introduce a breaking change. For example, you can set cython to
cython>=3.0.11,<4.0.0 to ensure no major version upgrades are used.
{{% /note %}}

{{% note %}}
If you wish to develop the build script within your project's virtual environment, then you must also add the
dependencies to your project explicitly to a dependency group - the name of which is not important.

poetry add --group=build setuptools cython

{{% /note %}}

{{< /step >}}

{{< step >}}
Add Build Script

The build script can be a free-form Python script that uses any dependency specified in the previous step. This can be
named as needed, but must be located within the project root directory (or a subdirectory) and also must
be included in your source distribution. You can see the [example snippets section]({{< relref "#example-snippets" >}})
for inspiration.

{{% note %}}
The build script is always executed from the project root. And it is expected to move files around to their destinations
as expected by Poetry as per your pyproject.toml file.
{{% /note %}}

[tool.poetry.build]
script = "relative/path/to/build-extension.py"

{{% note %}}
The name of the build script is arbitrary. Common practice has been to name it build.py, however, this is not
mandatory. You should consider [using a subdirectory]({{< relref "#can-i-store-the-build-script-in-a-subdirectory" >}})
if feasible.
{{% /note %}}

{{< /step >}}

{{< step >}}
Specify Distribution Files

{{% warning %}}
The following is an example, and should not be considered as complete.
{{% /warning %}}

[tool.poetry]
...
include = [
    { path = "package/**/*.so", format = "wheel" },
    # sources must be present in sdist, can be ignored if you only have *.pyx sources
    { path = "package/**/*.c", format = "sdist" },
]

The key takeaway here should be the following. You can refer to the [pyproject.toml]({{< relref "pyproject#exclude-and-include" >}})
documentation for information on each of the relevant sections.

1. Include your build outputs in your wheel.
2. Exclude your build inputs from your wheel.
3. Include your build inputs to your source distribution.

{{< /step >}}

{{< /steps >}}

Example Snippets#

Cython#

{{< tabs tabTotal="3" tabID1="cython-pyproject" tabName1="pyproject.toml" tabID2="cython-build-script" tabName2="build-extension.py" tabID3="cython-src-tree" tabName3="Source Tree">}}

{{< tab tabID="cython-pyproject" >}}

[build-system]
requires = ["poetry-core", "cython", "setuptools"]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
...
packages = [
    { include = "package", from = "src"},
]
include = [
    { path = "src/package/**/*.so", format = "wheel" },
]
# if not already excluded via .gitignore
exclude = [
    "**/*.c"
]

[tool.poetry.build]
script = "scripts/build-extension.py"

{{< /tab >}}

{{< tab tabID="cython-build-script" >}}

from __future__ import annotations

import os
import shutil

from pathlib import Path

from Cython.Build import cythonize
from setuptools import Distribution
from setuptools import Extension
from setuptools.command.build_ext import build_ext

COMPILE_ARGS = ["-march=native", "-O3", "-msse", "-msse2", "-mfma", "-mfpmath=sse"]
LINK_ARGS = []
INCLUDE_DIRS = []
LIBRARIES = ["m"]

def build() -> None:
    extensions = [
        Extension(
            "*",
            ["src/package/*.pyx"],
            extra_compile_args=COMPILE_ARGS,
            extra_link_args=LINK_ARGS,
            include_dirs=INCLUDE_DIRS,
            libraries=LIBRARIES,
        )
    ]
    ext_modules = cythonize(
        extensions,
        include_path=INCLUDE_DIRS,
        compiler_directives={"binding": True, "language_level": 3},
    )

    distribution = Distribution({
        "name": "package",
        "ext_modules": ext_modules
    })

    cmd = build_ext(distribution)
    cmd.ensure_finalized()
    cmd.run()

    # Copy built extensions back to the project
    for output in cmd.get_outputs():
        output = Path(output)
        relative_extension = Path("src") / output.relative_to(cmd.build_lib)

        shutil.copyfile(output, relative_extension)
        mode = os.stat(relative_extension).st_mode
        mode |= (mode & 0o444) >> 2
        os.chmod(relative_extension, mode)

if __name__ == "__main__":
    build()

{{< /tab >}}

{{< tab tabID="cython-src-tree" >}}

scripts/
└── build-extension.py
src/
└── package
    ├── example.pyx
    └── __init__.py

{{< /tab >}}

{{< /tabs >}}

Meson#

{{< tabs tabTotal="2" tabID1="meson-pyproject" tabName1="pyproject.toml" tabID2="meson-build-script" tabName2="build-extension.py">}}

{{< tab tabID="meson-pyproject" >}}

[tool.poetry.build]
script = "build-extension.py"

[build-system]
requires = ["poetry-core", "meson"]
build-backend = "poetry.core.masonry.api"

{{< /tab >}}

{{< tab tabID="meson-build-script" >}}

from __future__ import annotations

import subprocess

from pathlib import Path

def meson(*args):
    subprocess.call(["meson", *args])

def build():
    build_dir = Path(__file__).parent.joinpath("build")
    build_dir.mkdir(parents=True, exist_ok=True)

    meson("setup", build_dir.as_posix())
    meson("compile", "-C", build_dir.as_posix())
    meson("install", "-C", build_dir.as_posix())

if __name__ == "__main__":
    build()

{{< /tab >}}

{{< /tabs >}}

Maturin#

{{< tabs tabTotal="2" tabID1="maturin-pyproject" tabName1="pyproject.toml" tabID2="maturin-build-script" tabName2="build-extension.py">}}

{{< tab tabID="maturin-pyproject" >}}

[tool.poetry.build]
script = "build-extension.py"

[build-system]
requires = ["poetry-core", "maturin"]
build-backend = "poetry.core.masonry.api"

{{< /tab >}}

{{< tab tabID="maturin-build-script" >}}

import os
import shlex
import shutil
import subprocess
import zipfile

from pathlib import Path

def maturin(*args):
    subprocess.call(["maturin", *list(args)])

def build():
    build_dir = Path(__file__).parent.joinpath("build")
    build_dir.mkdir(parents=True, exist_ok=True)

    wheels_dir = Path(__file__).parent.joinpath("target/wheels")
    if wheels_dir.exists():
        shutil.rmtree(wheels_dir)

    cargo_args = []
    if os.getenv("MATURIN_BUILD_ARGS"):
        cargo_args = shlex.split(os.getenv("MATURIN_BUILD_ARGS", ""))

    maturin("build", "-r", *cargo_args)

    # We won't use the wheel built by maturin directly since
    # we want Poetry to build it, but we need to retrieve the
    # compiled extensions from the maturin wheel.
    wheel = next(iter(wheels_dir.glob("*.whl")))
    with zipfile.ZipFile(wheel.as_posix()) as whl:
        whl.extractall(wheels_dir.as_posix())

        for extension in wheels_dir.rglob("**/*.so"):
            shutil.copyfile(extension, Path(__file__).parent.joinpath(extension.name))

    shutil.rmtree(wheels_dir)

if __name__ == "__main__":
    build()

{{< /tab >}}

{{< /tabs >}}

FAQ#

When is my build script executed?#

If your project uses a build script, it is run implicitly in the following scenarios.

1. When poetry install is run, it is executed prior to installing the project's root package.
2. When poetry build is run, it is executed prior to building distributions.
3. When a PEP 517 build is triggered from source or sdist by another build frontend.

How does Poetry ensure my build script's dependencies are met?#

Prior to executing the build script, Poetry creates a temporary virtual environment with your project's active Python
version and then installs all dependencies specified under build-system.requires into this environment. It should be
noted that no packages will be present in this environment at the time of creation.

Can I store the build script in a subdirectory?#

Yes you can. If storing the script in a subdirectory, your pyproject.toml might look something like this.

[tool.poetry]
...
packages = [
    { include = "package", from = "src"}
]
include = [
    { path = "src/package/**/*.so", format = "wheel" },
]
# exclude any intermediate source files
exclude = [
    "**/*.c"
]

[tool.poetry.build]
script = "scripts/build-extension.py"