21

I need to generate a list of insert statements (for postgresql) from html files, is there a library available for python to help me properly escape and quote the names/values? in PHP i use PDO to do the escaping and quoting, is there any equivalent library for python?

Edit: I need to generate a file with sql statements for execution later

3
  • stackoverflow.com/questions/309945/… Commented Oct 14, 2009 at 2:50
  • 7
    Generating "plain SQL statements" with data in-line is inherently prone to security failures for reasons as abstract as varying interpretations of Unicode characters between the database engine and the library doing the generation. It is NOT something you should try to do. The output, therefore, should be phrased -- and passed to your database API -- as a query string / data list combo. SQLAlchemy will automate this. Commented Oct 14, 2009 at 4:24
  • I understand, but the use case was generating an SQL file to be executed later Commented Mar 4, 2014 at 7:45

5 Answers 5

30

I know this is an old question, but I've often wanted what it seems the OP wants: A VERY simple library for generating basic SQL.

The below functions do just that. You give them a table name and a dictionary containing the data you want to use and they return the SQL query for the operation you need.

The key/value pairs represent field names and values in the database rows.

def read(table, **kwargs):
    """ Generates SQL for a SELECT statement matching the kwargs passed. """
    sql = list()
    sql.append("SELECT * FROM %s " % table)
    if kwargs:
        sql.append("WHERE " + " AND ".join("%s = '%s'" % (k, v) for k, v in kwargs.items()))
    sql.append(";")
    return "".join(sql)


def upsert(table, **kwargs):
    """ update/insert rows into objects table (update if the row already exists)
        given the key-value pairs in kwargs """
    keys = ["%s" % k for k in kwargs]
    values = ["'%s'" % v for v in kwargs.values()]
    sql = list()
    sql.append("INSERT INTO %s (" % table)
    sql.append(", ".join(keys))
    sql.append(") VALUES (")
    sql.append(", ".join(values))
    sql.append(") ON DUPLICATE KEY UPDATE ")
    sql.append(", ".join("%s = '%s'" % (k, v) for k, v in kwargs.iteritems()))
    sql.append(";")
    return "".join(sql)


def delete(table, **kwargs):
    """ deletes rows from table where **kwargs match """
    sql = list()
    sql.append("DELETE FROM %s " % table)
    sql.append("WHERE " + " AND ".join("%s = '%s'" % (k, v) for k, v in kwargs.items()))
    sql.append(";")
    return "".join(sql)

You use it like so. Just give it a table name and a dictionary (or use the **kwargs feature of python):

>>> upsert("tbl", LogID=500, LoggedValue=5)
"INSERT INTO tbl (LogID, LoggedValue) VALUES ('500', '5') ON DUPLICATE KEY UPDATE LogID = '500', LoggedValue = '5';"

>>> read("tbl", **{"username": "morten"})
"SELECT * FROM tbl WHERE username = 'morten';"

>>> read("tbl", **{"user_type": 1, "user_group": "admin"})
"SELECT * FROM tbl WHERE user_type = '1' AND user_group = 'admin';"

But BEWARE OF SQL INJECTION ATTACKS

Look what happens when a malicious user of your code does this:

>>> read("tbl", **{"user_group": "admin'; DROP TABLE tbl; --"})
"SELECT * FROM tbl WHERE user_group = 'admin'; DROP TABLE tbl; --';"

It's easy to make your own makeshift ORM but you only get what you see -- you have to escape the input yourself :)

EDIT:

I'm still using my old library. I've updated it a bit lately: https://github.com/kokke/nano-ORM-py

Sign up to request clarification or add additional context in comments.

4 Comments

Here for the insert statements it always generates values inside single quotes. Therefore this code has to be modified as below, ``` keys = ["%s" % k for k in kwargs] # values = ["'%s'" % v for v in kwargs.values()] values = [] for value in kwargs.values(): if type(value) is str: values.append("'%s'" % (value)) elif value is None: values.append("%s" % ("''")) else: values.append("%s" % value) ```
in Python 3+, replace kwargs.iteritems() with kwargs.items()
10 years later ... it would be better to use %s = ? instead of %s = '%s'. And you can pass a tuple of params to the SQL execute() function. The SQL database will automatically sanitize your data (to avoid SQL injection attacks) and it will put quotes around the string as appropriate.
@JohnHenckel I agree with your point and I think this is addressed in the code in the GitHub-repo linked in the bottom of the post, e,g, here github.com/kokke/nano-ORM-py/blob/main/nanoorm.py#L28
15

SQLAlchemy provides a robust expression language for generating SQL from Python.

Like every other well-designed abstraction layer, however, the queries it generates insert data through bind variables rather than through attempting to mix the query language and the data being inserted into a single string. This approach avoids massive security vulnerabilities and is otherwise The Right Thing.

4 Comments

Link is dead. I think this is the relavant document: docs.sqlalchemy.org/en/latest/core/expression_api.html
@phasetwenty, thanks, updated (five years later, heh).
I think it is dead yet again
Updated to link directly to the root of the SQLAlchemy Core docs.
2

For robustness, I recommend using prepared statements to send user-entered values, no matter what language you use. :-)

1 Comment

is there already a library to do so? i want plain sql statements as output though :)
2

The python db api 2.0 has a ".execute" method for connection objects. You can specify parameters (use a comma NOT a % sign to separate params from the query string) with this function.

Comments

2

Quoting parameters manually in general is a bad idea. What if there is a mistake in escaping rules? What if escape doesn't match used version of DB? What if you just forget to escape some parameter or erroneously assumed it can't contain data requiring escaping? That all may cause SQL injection vulnerability. Also, DB can have some restrictions on SQL statement length while you need to pass large data chunk for LOB column. That's why Python DB API and most databases (Python DB API module will transparently escape parameters, if database doesn't support this, as early MySQLdb did) allow passing parameters separated from statement:

.execute(operation[,parameters])

1 Comment

I understand, but i just want a list of sql statements (not executing them)

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.