Pybind11 - SciPy 2021

Pybind11 - SciPy 2021

• 1.
  SciPy 2021 -Virtual Maintainers Track July 16, 2021 Eric Cousineau Toyota Research Institute Ralf Grosse-Kunstleve Google Axel Huebl Lawrence Berkeley National Laboratory Yannick Jadoul Vrije Universiteit Brussel
 Max Planck Institute for Psycholinguistics Wenzel Jakob Ecole Polytechnique Fédérale de Lausanne Henry Schreiner Princeton University Boris Staletic University of Novi Sad
• 2.
  Outline 2 What is pybind11? Advancementsin 2020/2021 Plans for 2021/2022 Related projects we ♥
• 3.
  What is pybind11? 3 Header-onlypure C++11 CPython/PyPy interface Trivial to add to a project No special build requirements No dependencies No precompile phase Not a new language (Boost.Python + BJam) (Cython, SWIG) ← Counter examples Think of it like the missing C++ API for CPython Designed for one purpose!
• 4.
  Example of usage(simple) 4 #include <pybind11/pybind11.h> int add(int i, int j) { return i + j; } PYBIND11_MODULE(example, m) { m.def("add", &add); } Standard include Normal C++ to bind Create a Python module Signature statically inferred Docs and parameter names optional g++ -shared -fPIC example.cpp $(pipx run pybind11 --includes) -o example$(python3-config --extension-suffix) Complete, working, no-install example (linux version)!
• 5.
  Example of usage(class) 5 #include <pybind11/pybind11.h> #include <pybind11/operators.h> namespace py = pybind11; using namespace pybind11::literals; PYBIND11_MODULE(example, m) { py::class_<Pet>(m, "Pet") .def(py::init<const std::string &>()) .def("setName", &Pet::setName, "value"_a = "", "docs") .def("getName", [](const Pet& self){return self.getName();}) .def_property("name", &Pet::getName, &Pet::setName) .def(py::self *= float()); } Easy operators Properties Names, defaults, docs Any number of constructors Lambdas
• 6.
  Unique design 6 Simple enoughfor tiny projects 491K code mentions of pybind11 on GitHub Powerful enough for huge projects SciPy, PyTorch, dozens of Google projects Small binaries prioritized Small overload pointer check per call Powerful object lifetime controls py::keep_alive, std::shared_ptr, and more NumPy support without NumPy headers No need to lock minimum NumPy at build Supports interop in both directions Can even be used to embed Python in C++ Most STL containers and features supported Including C++17 additions, like std::variant (or boost) Vectorize methods or functions py::vectorize, even on a lambda function Trampoline classes and multiple inheritance Complex C++ is supported Complete control of the Python representation Special methods, inheritance, pickling, and more Bu ff er protocol and array classes Includes Eigen support too Cross-extension ABI One extension can return a type wrapped in another one
• 7.
  New maintainer team 7 WenzelJakob is the original author Original team: Wenzel Jakob Dean Moldovan Jason Rhinelander Ivan Smirnov Sylvain Corley New team (starting mid 2020): Wenzel Jakob (BDFL) Eric Cousineau Ralf Grosse-Kunstleve Axel Huebl
 Yannick Jadoul
 Henry Schreiner Boris Staletic (uno ffi cial) Changes this year: Aaron Gokaslan: new member 👋 Boris Staletic: (mostly) stepped away for now 😢 Wenzel Jakob: moved from BDFL to co-maintainer
• 8.
  New in 2.6:CI 8 Massive rewrite of the CI in GHA, with 60+ jobs covering far more than ever before Special thanks to Google for funding extra parallel jobs! Jobs: Windows, macOS and Linux, Python 2.7-3.9 (3.10 now) & PyPI GCC (4.8 and several newer), Clang (8 versions), MSVC 2015-2019 ICC, NVCC (CUDA 11), and NVHPC (PGI) 20.9 CentOS 7 and 8 C++: 11, 14, 17, and 20 Debug build Valgrind build Clang-tidy build CMake con fi gure check, 3.4, 3.7, 3.8, and 3.18 Packaging tests verifying every fi le in the wheel/SDist Newly supported compilers! Discovered and fi xed bug in CPython 3.9.0
• 9.
  setup.py New in 2.6:build systems 9 CMake Setuptools Integration with standard CMake features CMake 3.4+ required Support for newer features (3.18.2+ best) FindPython optionally supported CUDA as a language supported and tested Python discovery can be deactivated Portable con fi g fi le (now included in PyPI package) Helper functions added Check importable libraries easily New modular target design Follows best practices given in https://cliutils.gitlab.io/modern-cmake New helpers can be used for easy setuptools support Proper C++ fl ags & pybind11 headers from pybind11.setup_helpers import Pybind11Extension, build_ext module = Pybind11Extension( "python_example", ["src/main.cpp"], cxx_std=11, ) setup( ..., ext_modules=ext_modules, cmdclass={"build_ext": build_ext}, # Optional! ) Optional parallel compiler utility included
• 10.
  New in 2.6:other 10 py::kw_only and py::pos_only Support Python 3 keyword only arguments Support Python 3.8 position only arguments All from any version of Python (C API) More correctness checking Throw errors sooner if a mistake is made py::final support Makes a class uninheritable (CPython only) py::prepend Add to the beginning of the overload chain py::type Access and manipulate the type More compilers supported Thanks to broadly expand checking Many fi xes And docs updates! py::module → py::module_ Avoid soft keyword in C++20 (old name should always work) https://iscinumpy.gitlab.io/post/pybind11-2-6-0/
• 11.
  New in 2.7 11 Largeclang-tidy cleanups Readability, better performance, modernization Optional C++17 fi lesystem / pathlib caster 3.6+ only feature In-tree extensions Not ideal for all users, but mimics Cython’s tool py::bytearray Similar to py::bytes py::str change Always unicode Extra GIL safety when C++ calls back into Python 3.6+ only feature Checks can be run from nox Easier for new developers to contribute Fixes Many rare and edge cases fi xed Development focus: stability, reliability, support for the latest C++/Python changes
• 12.
  New in 2.7 11 Largeclang-tidy cleanups Readability, better performance, modernization Optional C++17 fi lesystem / pathlib caster 3.6+ only feature In-tree extensions Not ideal for all users, but mimics Cython’s tool py::bytearray Similar to py::bytes py::str change Always unicode Extra GIL safety when C++ calls back into Python 3.6+ only feature Checks can be run from nox Easier for new developers to contribute Fixes Many rare and edge cases fi xed Development focus: stability, reliability, support for the latest C++/Python changes Released today!
• 13.
  Plans 12 Refactors Decoupling unit tests Breakinginto smaller, IWUY headers https://github.com/pybind/pybind11/wiki/Roadmap Optional Precompilation Would dramatically speed up compile Huge change, so being held back by other work Drop Python 2.7 & 3.5 Either 3.0 or Jan 1, 2022, whichever is fi rst Merge the smart holder branch Full interoperability with std::unique_ptr and std::shared_ptr Opt-in with <pybind11/smart_holder.h> Safe passing to C++, including trampolines Further ideas (not promised yet) Better MyPy integration Better Scikit-build integration Overhauls for inheritance and type casting
• 14.
  Our other projects 13 pybind/python_example: Asetuptools project using setuptools helpers and binary wheel builds, conda recipe, and more. pybind/scikit_build_example: A CMake project using scikit-build (new for 2.6). pybind/cmake_example: A CMake project using manual setuptools integration. Major cleanup for 2.6, now includes Apple Silicon support and more. pybind/pybind11_bazel: Support for Google’s bazel build system. pybind/pybind11_mkdoc: A documentation parser using LLVM. pybind/pybind11_json: A nlohmann::json adaptor for pybind11. pybind/pybind11_abseil: Adaptor for Google’s abseil project. pybind/pybind11_protobuf: Adaptor for Google’s protobuf project. Support for conda, pip, and cibuildwheel work fl ows on GitHub Actions Support for Apple Silicon cross-compile Support for PyPy wheels Dependabot GHA updates Template repositories now pybind11 also was added as an option to https://github.com/scikit-hep/cookie, make a new package in <60 seconds based on https://scikit-hep.org/developer guidelines
• 15.
  Projects we ♥:🎡 cibuildwheel .github/work fl ows/wheels.yml pyproject.toml 14 on: [push, pull_request] jobs: build_wheels: runs-on: ${{ matrix.os }} strategy: matrix: os: - ubuntu-20.04 - windows-2019 - macos-10.15 steps: - uses: actions/checkout@v2 - name: Build wheels uses: pypa/cibuildwheel@v2.0.0 - uses: actions/upload-artifact@v2 with: path: ./wheelhouse/*.whl [tool.cibuildwheel.macos] archs: ["x86_64", "universal2"] test-skip: ["*_universal2:arm64"] Binary distribution is really important for pybind11 projects. Maybe then it’s no surprise we share two maintainers with cibuildwheel! Supports all major CI providers GitHub Action provided too! Can run locally on Linux or Windows A ffi liated (shared maintainer) with manylinux Close collaboration with PyPy devs Recently accepted into the PyPA! Used by matplotlib, mypy, scikit-learn, and more Supports: Targeting macOS 10.9+ Apple Silicon cross-compiling All variants of manylinux Repairing and testing wheels Sensible, pinned defaults (can unpin) New in 2.0, release today! Python 2 & 3.5 removed Python 3.10 as pre-release PyPy 7.3.5 (manylinux uni fi cation!) Auditwheel 4 pyproject.toml support Optional pypa/build support All pybind examples include cibuildwheel!
• 16.
  Projects we ♥:scikit-build 15 Scikit-build is a CMake-setuptools adaptor from KitWare, the makers of CMake First introduced as PyCMake at SciPy 2014 and renamed in 2016 Includes CMake for Python and ninja for Python We now have a scikit-build example! Our pybind/cmake_example is one of the most popular examples of combining CMake and Python on GitHub Updated now with lots of fi xes! But this duplicates the code for everyone Adding Apple Silicon support will now have to be done on every project that copied the example, etc. Two new maintainers have now joined the project One from pybind11, one from cibuildwheel/manylinux! What’s new in scikit-build 0.12 (out soon!) CI refactor, using GHA, nox, pre-commit Several fi xes (often for Cython) Apple Silicon support MSVC 2019 support What’s planned: Better integration with pybind11 Better support for PEP 517 Support testing against development versions of Python Removal of Python 2.7 and 3.5 before or on Jan 1, 2022 See https://github.com/scikit-build/scikit-build/wiki Same day release of CMake for Python for CMake 3.21.0! All manylinux archs • Apple Silicon • cibuildwheel • nox Revamped ninja for Python too All manylinux archs • cibuildwheel • nox
• 17.
  Acknowledgements 16 Special thank youto Google for sponsoring our expanded CI, and for Ralf Grosse-Kunstleve’s time. Henry Schreiner supported by IRIS-HEP and the NSF under Cooperative Agreement OAC-1836650 Axel Huebl is with LBNL and supported by the U.S. DOE/NNSA Exascale Computing Project (17-SC-20-SC)