Re: Stuff for 2.4.1 - Mailing list psycopg

On Sun, Mar 27, 2011 at 6:23 PM, David Blewett <david@dawninglight.net> wrote:
> On Sun, Mar 27, 2011 at 9:46 AM, Daniele Varrazzo
> <daniele.varrazzo@gmail.com> wrote:
>> On Sun, Mar 27, 2011 at 12:51 PM, Harald Armin Massa
>> <harald@2ndquadrant.com> wrote:
>>> Are you really sure that psycopg2 should go the road of having own
>>> parsers in addition to libpq-s routines? As much as I am happy about
>>> the robustness when having other libpqs, and about the performance
>>> benefit, as much I fear to have some new areas for possible bugs -
>>> especially security-relevant things like SQL-injections.
>>
>> The will to stick as much as possible to the libpq functions has been
>> the reason I had not written the above parser before (releasing
>> 2.4.0). Unfortunately the bytea problem has proven trickier to handle
>> for many psycopg users. I've changed my mind as I think psycopg has
>> the responsibility to provide a set of feature in a robust way, and if
>> the libpq is just not reliable for bytea parsing (for me the hex
>> format should have been backported to the the client libraries of the
>> previous versions) I think we have to provide a solution, not just to
>> propagate the problem.
>
> I think I agree with Harald here. In my opinion, this shouldn't be
> done at the driver level.

No? At which level do you think this should be done?

> There never has been a guarantee from the
> database side that applications compiled against older libpq will be
> able to communicate with newer versions. Emulating this in the driver
> only propagates this mis-conception. What has been the problem in the
> past?

There have been several examples in the not so long story of this
mailing list, other were in the previous ml. Here's a couple. You can
google yourself to discover that GnuMed, Drupal and who knows how many
not public projects have been bitten by the transition to hex.

http://archives.postgresql.org/psycopg/2011-02/msg00020.php
http://archives.postgresql.org/pgsql-general/2010-10/msg00146.php
(from Harald, interesting)

> Maybe the documentation should be improved so that people are
> sure to build against the appropriate version of libpq for the version
> of the server they intend to communicate with?

The documentation does a pretty good job in describing the problem.
Unfortunately not everybody is willing/able/allowed to install an
up-to-date client library. PG 9 is not so widespread on the clients
yet (e.g. Natty still deploys 8.4, so you will have to wait at least
Oct 2011 for a libpq 9 on the most widespread linux distro).

Half of Psycopg job is to convert the textual representation of
postgres data type into python object. The libpq provides no
assistance in doing that for any other format than bytea. There is no
PQunescapeDateTime whatsoever: should we ask Python users to parse the
dates from string themselves? Not to mention composite types, hstore
and other delicacies, for which we provide parsers to Python objects.
The bytea is in a sense "blessed" for having a dedicated unescape
function, but if using it causes more trouble than it solves I think a
less problematic parser is a good answer.

-- Daniele