Extensible Python: Robustness through Addition - PyCon 2024

Copyright 2024 Kudzera,
LLC
Software is both
archeology and time-
travel

LLC
Your job is to deliver
value in a timely
manner

LLC
How do you enable the
future to be just as
efficient?

LLC
Extensibility

LLC
0) Think about the humans

LLC
Extensibility Themes

LLC
1) Reduce Commit Complexity

LLC
2) Be Mindful of OCP

LLC
Open-Closed Principle

LLC
Code should be open to
extension and closed
for modification

LLC
When To Split
● Are easy things hard to do?
● Do you encounter pushback against similar features?
● Do you have consistently high estimates?
● Do commits contain large changesets?
● Are you mixing policy changes with mechanisms changes

LLC
Policies vs. Mechanisms

LLC
3) Separate Policies and
Mechanisms

LLC
From backoff documentation

LLC
From Flask documentation

LLC
4) Data-Driven Designs

LLC
Data options
● Can you put your data in a collection?
● Configuration Files?
● Persistence Layers (i.e. databases, bucket storage, etc.)?

LLC
5) Develop Libraries First

LLC
Good Library Design
● README-driven
● Tested
● Easily Consumable
● Clearly-stated opinions
● Releasable (w/ Documentation)
● Extensible
● Composable

LLC
Composability

LLC
6) Create Building Blocks

LLC
Architectural Extensibility

LLC
Dependencies

LLC
Physical Dependencies

LLC
from typing import Callable
@dataclass
class Metadata:
does_match: Callable[[User], bool]
if __name__ == "__main__":
run_self_test()

LLC
pip install requests

LLC
Physical Dependencies
● Hard-coded into source code
● Easy to follow from A to B
● Easily understandable
● Easy to find, hard to change
● Hard to substitute or mock out
● Understandable by static analysis tools

LLC
pipdeptree

LLC
pydeps

LLC
pyan3

LLC
What Happens If You Need
Something To Change?

LLC
A
B
C

LLC
A
B
D

LLC
A
B
Whatever

LLC
Logical Dependencies

LLC
requests.post("table-management/pizza-made", {
"id": order,
"pizza": pizza.to_json()
})
# meal is an abstract type
meal: Meal = meal_factory.create("pizza")

LLC
Logical Dependencies
● Typically determined at run-time
● Readability suffers
● Debuggability suffers
● Hard to find, easy to change
● Easy to substitute or mock out
● Not understandable by static analysis tools
● Crucial for Abstraction

LLC
7) Trade-off Dependencies
Judiciously

LLC
Event-Driven Architectures

LLC
def complete_order(order: Order):
package_order(order)
notify_customer_that_order_is_done(order)
notify_restaurant_that_order_is_done(order)

LLC
Producer
Transport
Consumer

LLC
pypubsub

LLC
from pubsub import pub
def notify_customer_that_meal_is_done(order: Order):
# ... snip ...
pub.subscribe(notify_customer_that_meal_is_done, "meal-done")

LLC
from pubsub import pub
def complete_order(order: Order):
package_order(order)
pub.publish("meal-done", order)

LLC
8) Use Event-Driven
Architectures to Decouple
Producers and Consumers

LLC
Pluggable Architectures

LLC
Plug-in examples
● pytest
● poetry
● hypothesis
● pylint

LLC
Plugging Into Algorithms

LLC
@dataclass
class PizzaCreationFunctions:
prepare_ingredients: Callable
add_pre_bake_toppings: Callable
add_post_bake_toppings: Callable

LLC
def create_pizza(
pizza_creation_functions: PizzaCreationFunctions
):
pizza_creation_functions.prepare_ingredients()
roll_out_pizza_base()
pizza_creation_functions.add_pre_bake_toppings()
bake_pizza()
pizza_creation_functions.add_post_bake_toppings()

LLC
pizza_creation_functions = PizzaCreationFunctions(
prepare_ingredients=mix_zaatar,
add_pre_bake_toppings=add_meat_and_halloumi,
add_post_bake_toppings=drizzle_olive_oil
)
create_pizza(pizza_creation_functions)

LLC
pizza_creation_functions = PizzaCreationFunctions(
prepare_ingredients=cook_bulgogi,
add_pre_bake_toppings=add_bulgogi_toppings,
add_post_bake_toppings=garnish_with_scallions_and_sesame
)
create_pizza(pizza_creation_functions)

LLC
Plugging Into Packages

LLC
Stevedore

LLC
from abc import abstractmethod
from typing import runtime_checkable, Protocol
@runtime_checkable
class UltimateKitchenAssistantModule(Protocol):
ingredients: list[Ingredient]
@abstractmethod
def get_recipes() -> list[Recipe]:
raise NotImplementedError
@abstractmethod
def prepare_dish(inventory: dict[Ingredient, Amount],
recipe: Recipe) -> Dish:
raise NotImplementedError

LLC
class PastaModule(UltimateKitchenAssistantModule):
def __init__(self):
self.ingredients = ["Linguine",
# ... snip ...]
def get_recipes(self) -> list[Recipe]:
# ... snip returning all possible recipes ...
def prepare_dish(self, inventory: dict[Ingredient, Amount],
recipe: Recipe) -> Dish:
# interact with Ultimate Kitchen Assistant

LLC
from stevedore import extension
def get_all_recipes() -> list[Recipe]:
mgr = extension.ExtensionManager(
namespace='ultimate_kitchen_assistant.recipe_maker',
invoke_on_load=True,
)
def get_recipes(extension):
return extension.obj.get_recipes()
return list(itertools.chain(mgr.map(get_recipes)))

LLC
from setuptools import setup
setup(
name='ultimate_kitchen_assistant',
version='1.0',
#.... snip ....
entry_points={
'ultimate_kitchen_assistant.recipe_maker':
['pasta_maker =
ultimate_kitchen_assistant.pasta_maker:PastaModule'],
},
)

LLC
9) Use Pluggable Python for
Extension Points

LLC
0) Think about the humans
1) Reduce Commit Complexity
2) Be Mindful of OCP
3) Separate Policies and Mechanisms
4) Consider Data-Driven Design
5) Develop Libraries First
6) Create Building Blocks
7) Trade-off Dependencies Judiciously
8) Use Event-Driven Architectures to Decouple Producers/Consumers
9) Use Pluggable Python for Extension Points

LLC
10) Don't Overextend Yourself

LLC
Who Am I?
Principal Software Engineer
Cloud Software Group
Owner of Kudzera, LLC
Author of Robust Python
Organizer of HSV.py

Extensible Python:
Robustness Through
Addition
Patrick Viafore

Extensible Python: Robustness through Addition - PyCon 2024

More Related Content

Similar to Extensible Python: Robustness through Addition - PyCon 2024

More from Patrick Viafore

Recently uploaded

Extensible Python: Robustness through Addition - PyCon 2024

Editor's Notes