Python postgre sql a wonderful wedding

Python & Postgresql
A Wonderful Wedding
Stéphane Wirtel
@PyConFR 2014 - Lyon
http://goo.gl/UsPLwh

Stéphane Wirtel
#fellow @ThePSF
#member EPS
#member AFPy
@PythonFOSDEM
@matrixise
stephane@wirtel.be
http://wirtel.be

Agenda Qu’allons-nous apprendre ?

Python, c’est quoi ?
Langage de Programmation
Syntaxe simple & lisible
Interprété
Multi-plateformes
Typage Dynamique Fort
Multi-paradigmes
Garbage Collector

PostgreSQL - Story
Développée à Berkeley
Réécriture de Ingres
1985 -> Postgres (vient de Post-Ingres)
1995 -> Postgres95 (0.01) (Ajout d’un interpréter SQL)
1997 -> PostgreSQL 6.0
2013 -> PostgreSQL 9.3
2014 -> PostgreSQL 9.4 (b4)

PostgreSQL - TL;DR
ORDBMS, ACID
SQL:2011
DataTypes
Transactional DDL
Concurrent Index
Extensions
Common Table Expression
MultiVersion Concurrency Control
Cross-Platform

PostgreSQL - TL;DR
Replication
Foreign Data Wrappers
Procedural Languages
Triggers
Full text Search
Views (Materialized)
Table Inheritance
Listen/Notify

PostgreSQL, FDW ?
Utiliser une source externe à PostgreSQL
Twitter, RSS, CSV, XML
FileSystem
Processes

PostgreSQL, PL ?
Étend la base de données
Fonctions stockées
Safe (sandbox, SQL, PL/pgSQL) / Unsafe (C)
PL/Python, PL/V8, PL/PERL

DB-API 2.0 Commençons par le commencement…
http://python.org/peps/pep-0249

DB-API 2.0 aka PEP-249
API pour les connecteurs de Base de Données
Facile à utiliser, et à comprendre
Utilise deux concepts:
• Connection
• Curseur
http://legacy.python.org/dev/peps/pep-0249/

Connection
Emballe une connection vers la base de données
Gère les transactions et leur état (Commit/
Rollback)
N’exécute pas de requêtes SQL.

Connection - API
• connect(parameters=None)
• close()
• commit()
• rollback()
• cursor([name=None])

Connection - Exemple
import driver
conn = driver.connect(database='database',
host='localhost',
port=5432)
try:
# cree le curseur
# utilise le curseur
except Exception:
conn.rollback()
else:
conn.commit()
conn.close()

Curseur
Créé via une instance d’une connection
Utilisé pour la manipulation et interrogation de la
Base de Données
cursor = conn.cursor()
cursor.execute("""
SELECT column1, column2
FROM tableA
""")
for column1, column2 in cursor.fetchall():
print(column1, column2)

Curseur - API
• callproc(procname[, parameters])
• execute(query[, parameters])
• fetchone(),
fetchmany([size=cursor.arraysize]),
fetchall()
• close()

Curseur - execute
execute(query[, parameters])
Performance et Sécurité
JAMAIS utiliser l’interpolation (%)
et la concaténation (+)
=> SQL Injection

Curseur - execute (2)
Accepte un formatage de la requête.
qmark Question mark WHERE field = ?
numeric Numeric positional WHERE field = :1
named Named WHERE field = :code
format ANSI C print format WHERE field = %s
pyformat Python format WHERE field = %(name)s

Exemple plus complet ;-)
import driver
conn = driver.connect(database='database',
host='localhost',
port=5432)
try:
cursor.execute("SELECT column1, column2 FROM tableA")
except Exception:
conn.rollback()
else:
conn.commit()
finally:
cursor.close()
conn.close()

psycopg2 La Base…
pip install psycopg2

Introduction
PostgreSQL Adaptor, basé sur libpq
DB-API 2.0 Compliant
Multi-thread
Pool de Connections
Full Asynchronous, Coroutines
Supporte TOUS les types de données de PostgreSQL (Json,
Hstore, …)
Python 2.5+, 3.1+, PostgreSQL 7.4+

Curseurs
cursor = conn.cursor(name=“pycon_cursor”)
• Curseur coté PostgreSQL
• Interrogation, manipulation des données de la
base.
• Nommé => Utilisé pour les gros ensembles de
données

Requêtes SQL
cursor.execute("SELECT * FROM table")
cursor.execute("INSERT INTO table (field1, field2) VALUES (%s, %s)",
(field1, field2))
cursor.execute("DELETE FROM table")
cursor.execute("UPDATE table SET field1=%s", ('value',))
cursor.execute("CREATE DATABASE database")
cursor.execute("DROP DATABASE database")

Exemple
import psycopg2
conn = psycopg2.connect(host='localhost', port=5432,
user='username', password='password',
database='database')
try:
except Exception:
conn.rollback()
else:
conn.commit()
finally:
cursor.close()
conn.close()

Exemple - Context Manager
import psycopg2
DSN = dict(host='localhost', port=5432,
user='username', password='password',
database='database')
with psycopg2.connect(**DSN) as conn:
with conn.cursor() as cursor:
conn.commit()

Introduction
“Technique de Programmation créant l’illusion
d’une base de données orientée objet à partir
d’une base de données relationnelle en définissant
des correspondances entre cette base de
données et les objets du langage utilisé.”
Wikipedia

En bref
• Mapper une classe Business sur une table !
• Méthodes pour ajouter, modifier, supprimer et
rechercher

Peewee-ORM Object Relational Mapping
pip install peewee

Peewee
ORM
Simple, facile à comprendre et extensible
Python 2.6+ et 3.2+
Extensions
3200 lignes de code (MySQL, Sqlite3 et PostgreSQL)
Très bonne documentation

Connection
import peewee
database = peewee.PostgresqlDatabase('demo')

Modèles
class BaseModel(peewee.Model):
id = peewee.PrimaryKeyField()
class Meta:
database = database
class Address(BaseModel):
street = peewee.CharField()
zipcode = peewee.CharField()
country = peewee.CharField()
class Contact(BaseModel):
firstname = peewee.CharField(null=False)
lastname = peewee.CharField(null=False)
email = peewee.CharField(unique=True, null=False, index=True)
address = peewee.ForeignKeyField(Address, null=False,
related_name=‘contacts')

Création des tables
database.create_tables([Address, Contact])

Peewee - Transactions
tx = database.transaction()
tx.commit()
tx.rollback()
with database.transaction():
pass

Create
address = Address.create(
street='Rue de Lyon',
zipcode='90001',
country='France')
contact = Contact.create(
firstname='Stephane', lastname='Wirtel',
email='stephane@wirtel.be',
address=address
)
print(contact.id)

Read
contact = Contact.get(Contact.email == 'stephane@wirtel.be')
print(contact.firstname)
contacts = Contact.select()
for contact in contacts:
for contact in contacts.where(Contact.email == 'stephane@wirtel.be'):

Update
contact.firstname = 'Speaker'
contact.save()

Delete
# Suppression pour une collection
contacts = Contact.delete().where(Contact.email == 'stephane@wirtel.be')
contacts.execute()
# Suppression d'un seul enregistrement
contact.delete_instance()

SQLAlchemy Object Relational Mapping
pip install sqlalchemy

SQLAlchemy
• Gestionnaire de connections
• Abstraction des connecteurs SQL
• Langage d’Expression SQL
• ORM

Connection
from sqlalchemy import create_engine
engine = create_engine('postgresql:///demo', echo=True)

Modèles from sqlalchemy import Column, Integer, String, ForeignKey
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship
Base = declarative_base()
class Address(Base):
__tablename__ = 'address'
id = Column(Integer, primary_key=True)
street = Column(String)
zipcode = Column(String)
country = Column(String, nullable=False)
class Contact(Base):
__tablename__ = 'contact'
id = Column(Integer, primary_key=True)
firstname = Column(String, nullable=False)
lastname = Column(String, nullable=False)
email = Column(String, nullable=False)
address_id = Column(Integer, ForeignKey(Address.id), nullable=False)
address = relationship('Address')

Création des tables
Base.metadata.create_all(engine)

Sessions
Wraps les transactions
session = Session(engine)
session.commit()

Create
session = Session(engine)
address = Address(street='SQLAlchemy Street', zipcode='5432',
country='Belgique')
contact = Contact(
firstname='Stephane', lastname='Wirtel',
email='stephane@wirtel.be',
address=address
)
session.add(contact)
session.commit()

Read
contact =
session.query(Contact).filter_by(email='stephane@wirtel.be').first()
contacts = session.query(Contact).all()
contacts =
session.query(Contact).filter_by(email='stephane@wirtel.be').all()

Update
contact = session.query(Contact)
.filter_by(email='stephane@wirtel.be').first()
contact.email = 'sw@mgx.io'
session.add(contact)
session.commit()

Delete
contact = session.query(Contact)
.filter_by(email='stephane@wirtel.be').first()
session.delete(contact)
session.commit()

Alembic La Migration Facile
pip install alembic

Alembic
Basé sur SQLAlchemy
Utilise un environnement de Migration
Fichier de configuration INI
Scripts de migration
http://alembic.readthedocs.org/

Alembic
cd votreproject
alembic init migrations
> tree
migrations/
!"" README
!"" alembic.ini
!"" env.py
!"" script.py.mako
#"" versions

Alembic
alembic revision -m “bootstrap"
revision = '22630db6f519'
down_revision = None
from alembic import op
import sqlalchemy as sa
def upgrade():
op.create_table('user',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('created_at', sa.DateTime(), nullable=False),
sa.Column('name', sa.String(length=255), nullable=False),
sa.Column('email', sa.String(length=255), nullable=True),
sa.Column('password', sa.String(length=255), nullable=True),
sa.Column('active', sa.Boolean(), nullable=True),
sa.Column('confirmed_at', sa.DateTime(), nullable=True),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('email')
)
def downgrade():
op.drop_table('user')
alembic upgrade head

Alembic
alembic revision -m “add_slug”
revision = '7e4b6a43e6c'
down_revision = '22630db6f519'
import sqlalchemy as sa
def upgrade():
op.add_column('user', sa.Column('slug', sa.Unicode(length=255),
nullable=False, server_default=''))
def downgrade():
op.drop_column('user', 'slug')

Alembic
> tree
migrations/
!"" README
!"" alembic.ini
!"" env.py
!"" script.py.mako
#"" versions
!"" 22630db6f519_bootstrap.py
#"" 7e4b6a43e6c_add_slug.py

Alembic
from sqlalchemy.orm import Session
from youproject import models
def upgrade():
engine = op.get_bind()
session = Session(bind=engine)
for contact in session.query(models.Contact).all():
session.add(
models.WifiUser(
login=contact.name.lower(),
password=random_string()
)
)
session.commit()

Alembic
alembic revision
alembic upgrade +2
alembic downgrade -1
alembic downgrade base
alembic current
alembic history

Multicorn Où comment se connecter au monde !
http://multicorn.org

Rappel: Foreign Data Wrappers
Utiliser une source externe à PostgreSQL
Twitter, RSS, CSV, XML
FileSystem
Processes
PostgreSQL, Oracle, MongoDB

declarations for dynamic loading */
PG_MODULE_MAGIC;
PG_FUNCTION_INFO_V1(mongo_fdw_handler);
PG_FUNCTION_INFO_V1(mongo_fdw_validator);
/*
* mongo_fdw_handler creates and returns a struct with pointers to foreign table
* callback functions.
*/
Datum
mongo_fdw_handler(PG_FUNCTION_ARGS)
{
FdwRoutine *fdwRoutine = makeNode(FdwRoutine);
fdwRoutine->GetForeignRelSize = MongoGetForeignRelSize;
fdwRoutine->GetForeignPaths = MongoGetForeignPaths;
fdwRoutine->GetForeignPlan = MongoGetForeignPlan;
fdwRoutine->ExplainForeignScan = MongoExplainForeignScan;
fdwRoutine->BeginForeignScan = MongoBeginForeignScan;
fdwRoutine->IterateForeignScan = MongoIterateForeignScan;
fdwRoutine->ReScanForeignScan = MongoReScanForeignScan;
fdwRoutine->EndForeignScan = MongoEndForeignScan;
fdwRoutine->AnalyzeForeignTable = MongoAnalyzeForeignTable;
PG_RETURN_POINTER(fdwRoutine);
}

Multicorn
• Extension PostgreSQL
• Permet d’écrire un Foreign Data Wrapper
• Wrapper Python de l’API C de PostgreSQL
• Tous les outils utilisant SQL peuvent utiliser un
FDW
• Support FULL SQL ;-)

FDW de base
• RSS, CSV, XML
• LDAP
• Gmail, IMAP
• Google Search
• SQLAlchemy (mysql, sqlite, oracle, …)

Simple exemple
CREATE EXTENSION multicorn;

Simple exemple (2)
CREATE SERVER wirtel_be_srv
FOREIGN DATA WRAPPER multicorn
OPTIONS (
wrapper 'multicorn.rssfdw.RssFdw'
);
CREATE FOREIGN TABLE wirtel_be_rss (
"pubDate" TIMESTAMP,
description CHARACTER VARYING,
title CHARACTER VARYING,
link CHARACTER VARYING
) SERVER wirtel_be_srv OPTIONS(
url 'http://wirtel.be/feeds/python.rss.xml'
);

Simple exemple (3)
SELECT "pubDate", title, link FROM wirtel_be_rss LIMIT 1;
-[ RECORD 1 ]------------------------------------------------------------------------
pubDate | 2014-10-19 00:00:00
title | Python @ FOSDEM 2015 - Call For Proposals
link | http://wirtel.be/posts/en/2014/10/19/python-fosdem-2015-call-for-proposals/

avec OpenERP
from multicorn import ForeignDataWrapper
import erppeek
class OpenERPForeignDataWrapper(ForeignDataWrapper):
def __init__(self, options, columns):
super(OpenERPForeignDataWrapper, self).__init__(options, columns)
url = 'http://{hostname}:{password}'.format(**options)
self.client = erppeek.Client(url,
options['database'],
options['username'],
options['password'])
self.object_name = options['object']
def execute(self, quals, columns):
proxy = self.client.model(self.object_name)
item = {}
for record in proxy.browse([]):
for column in columns:
item[column] = record[column]
yield item

avec OpenERP (2) CREATE EXTENSION multicorn;
CREATE SERVER multicorn_openerp
FOREIGN DATA WRAPPER multicorn
OPTIONS (wrapper 'multicorn.openerpfdw.OpenERPForeignDataWrapper');
CREATE FOREIGN TABLE oe_users (
login character varying,
name character varying)
SERVER multicorn_openerp OPTIONS (
hostname 'localhost',
port '8069',
database 'openerp',
username 'admin',
password 'admin',
object 'res.users'
);
SELECT id, login, name, active
FROM oe_users
WHERE login = 'admin';

PL/Python
Python 2 ou Python 3
Utilise toutes les librairies Python (PyPI)
Apprentissage plus rapide que PL/pgSQL

PL/Python
apt-get install postgresql-plpython-9.3
CREATE EXTENSION IF NOT EXISTS plpythonu;

PL/Python
CREATE OR REPLACE FUNCTION str_title(s VARCHAR) RETURNS VARCHAR AS $$
return s.title()
$$ LANGUAGE plpythonu;
demo=# select str_title('hello world');
-[ RECORD 1 ]----------
str_title | Hello World

PL/Python - DataTypes
PostgreSQL Python
integer, bigint int, long
boolean bool
text types str
SQL Array list
Custom Types dict

PL/Python - Debug
Fonctions de base pour afficher les messages
• plpy.notice(“<msg>”)
• plpy.debug(“<msg>”)
• plpy.error(“<msg>”)
• plpy.fatal(“<msg>”)
Oublier le ”print”

PL/Python
CREATE OR REPLACE FUNCTION get_pid_cpu_mem(pid INT)
RETURNS TABLE(pid INT, cpu_perc FLOAT, mem_perc FLOAT) AS $$
import psutil
process = psutil.Process(pid)
return [
{
'pid': pid,
'cpu_perc': process.get_cpu_percent(interval=0),
'mem_perc': process.get_memory_percent()
}
]
$$ LANGUAGE plpythonu;

PL/Python
SELECT * FROM get_pid_cpu_mem(14680);
pid | cpu_perc | mem_perc
-------+----------+--------------
14680 | 0 | 1.4454081372
(1 row)

PL/Python
WITH stats AS (
SELECT psa.*, get_pid_cpu_mem(psa.pid) as attrs
from pg_stat_activity psa
)
SELECT (stats.attrs).pid,
(stats.attrs).cpu_perc,
(stats.attrs).mem_perc,
stats.application_name
FROM stats;
pid | cpu_perc | mem_perc | application_name
-------+----------+---------------+------------------
14680 | 0 | 1.50435626678 | psql
(1 row)

PL/Python - Attention
Unsafe -> pas de sandbox
Difficile à maintenir et à debugger
Comment rendre fou votre DBA ;-)
Pas de virtualenv et demande les privilèges
superuser.

PL/Python - Pistes ;-)
Traitement sur grosses masses de données
Ajouter des contraintes fortes et logique dans le système.
Ajouter de nouvelles fonctionnalités dans PostgreSQL
“str_title”
Triggers ?
Utilisation des libs de PyPI (requests, redis, zmq, smtp…)
Accès complet à la base de données !!!

Questions ?
@matrixise
https://speakerdeck.com/matrixise

Python postgre sql a wonderful wedding

More Related Content

What's hot

Similar to Python postgre sql a wonderful wedding

Recently uploaded

Python postgre sql a wonderful wedding