Thread: plpython crash
Hi there, plpython crashes on me on various 64-bit Ubuntu hosts, see the gdb backtrace at: https://gist.github.com/1140005 Do you believe there was recent bugfixes regarding PLyMapping_ToTuple() ? This is PG 9.0.4 with HEAD of plpython taken in march 2011 and backported. Please tell me if you need more information. -- Jean-Baptiste Quenot
On 11/08/11 18:01, Jean-Baptiste Quenot wrote: > Hi there, > > plpython crashes on me on various 64-bit Ubuntu hosts, see the gdb > backtrace at: https://gist.github.com/1140005 > > Do you believe there was recent bugfixes regarding PLyMapping_ToTuple() ? > > This is PG 9.0.4 with HEAD of plpython taken in march 2011 and backported. > > Please tell me if you need more information. Hi, there were no changes to that area of plpython after March 2011. Could you try to see if the error also appears if you run your app with current PostgreSQL HEAD (both the server and plpython)? Which Python version is that? You can get that info by running: do $$ import sys; plpy.info(sys.version) $$ language plpythonu; Could you try to extract a self-contained example of how to reproduce it? If the bug only appears under your application's specific workload, perhaps you could try running it with Postgres compiled with -O0, because compiling with -O2 causes the gdb backtrace to be missing optimised out values and inlined functions? Cheers, Jan
Here is the same with -O0: https://gist.github.com/1140005 sys.version reports this: INFO: 2.6.6 (r266:84292, Sep 15 2010, 16:41:53) [GCC 4.4.5] -- Jean-Baptiste Quenot
On 12/08/11 13:55, Jean-Baptiste Quenot wrote: > Here is the same with -O0: > > https://gist.github.com/1140005 > > sys.version reports this: > > INFO: 2.6.6 (r266:84292, Sep 15 2010, 16:41:53) > [GCC 4.4.5] I'm still at a loss. Did you reproduce it with git HEAD? I see that the query being execute is "select * from myfunc()"; would it be possible to share the code of myfunc? Jan
After backporting plpython.c from HEAD, this is the error message I get: ERROR: key "........pg.dropped.6........" not found in mapping HINT: To return null in a column, add the value None to the mapping with the key named after the column. CONTEXT: while creating return value PL/Python function "myfunc" What does it mean? -- Jean-Baptiste Quenot
On 16/08/11 16:52, Jean-Baptiste Quenot wrote: > After backporting plpython.c from HEAD, this is the error message I get: > > ERROR: key "........pg.dropped.6........" not found in mapping > HINT: To return null in a column, add the value None to the mapping > with the key named after the column. > CONTEXT: while creating return value > PL/Python function "myfunc" > > What does it mean? Ah, interesting, I think that this means that you are returning a table type and that table has a dropped column. The code should skip over dropped columns, but apparently it does not and tries to find a value for that column in the mapping you are returning. I'll try to reproduce it here. Jan
On 16/08/11 17:06, Jan Urbański wrote: > On 16/08/11 16:52, Jean-Baptiste Quenot wrote: >> After backporting plpython.c from HEAD, this is the error message I get: >> >> ERROR: key "........pg.dropped.6........" not found in mapping >> HINT: To return null in a column, add the value None to the mapping >> with the key named after the column. >> CONTEXT: while creating return value >> PL/Python function "myfunc" >> >> What does it mean? I did a couple of simple tests and can't see how can the code not skip dropped columns. It seems like you're missing this commit: http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=41282111e6cc73aca4b63dffe950ba7a63e4bd8a Could you try running this query? (assuming your function is called 'myfync') select proname, relname, attname, attisdropped from pg_proc join pg_class on (prorettype = reltype) join pg_attribute on (attrelid = pg_class.oid) where proname = 'myfunc'; Cheers, Jan
Dear Jan, Sorry I typed the wrong git commands. With latest plpython from branch master I got the same gdb backtrace as reported before. I managed to wrap up a testcase that fails 100% of times on my setup: https://gist.github.com/1149512 Hope it crashes on your side too :-) This is the result on PG 9.0.4: https://gist.github.com/1149543 This is the result on PG 9.0.4 with plpython.c backported from HEAD: https://gist.github.com/1149558 Cheers, -- Jean-Baptiste Quenot
On 16/08/11 19:07, Jean-Baptiste Quenot wrote: > Dear Jan, > > Sorry I typed the wrong git commands. With latest plpython from > branch master I got the same gdb backtrace as reported before. I > managed to wrap up a testcase that fails 100% of times on my setup: > https://gist.github.com/1149512 > > Hope it crashes on your side too :-) Awesome, it segfaults for me with HEAD ;) Now it's just a simple matter of programming... I'll take a look at it this evening. Jan
On 16/08/11 19:12, Jan Urbański wrote: > On 16/08/11 19:07, Jean-Baptiste Quenot wrote: >> Dear Jan, >> >> Sorry I typed the wrong git commands. With latest plpython from >> branch master I got the same gdb backtrace as reported before. I >> managed to wrap up a testcase that fails 100% of times on my setup: >> https://gist.github.com/1149512 >> >> Hope it crashes on your side too :-) > > Awesome, it segfaults for me with HEAD ;) > > Now it's just a simple matter of programming... I'll take a look at it > this evening. Found it, we're invalidating the compiled functions cache when input composite arguments change, but not when output composite arguments change and the function gets called with pointers to invalid I/O routines. I'll have a patch ready soon. Jan
On 17/08/11 11:40, Jan Urbański wrote: > On 16/08/11 19:12, Jan Urbański wrote: >> On 16/08/11 19:07, Jean-Baptiste Quenot wrote: >>> >>> [plpython is buggy] > I'll have a patch ready soon. Here are two patches that fix two separate bugs that you found simultaneously. Because they're actually separate issues, it turned out fixing them was a bit more tricky than I expected (fixing one was unmasking the other one etc). Thanks for the report! Jan
Attachment
Jan Urbański <wulczer@wulczer.org> writes:
>> On 16/08/11 19:07, Jean-Baptiste Quenot wrote:
>>> [plpython is buggy]
> Here are two patches that fix two separate bugs that you found
> simultaneously. Because they're actually separate issues, it turned out
> fixing them was a bit more tricky than I expected (fixing one was
> unmasking the other one etc).
These look generally sane although I have some minor stylistic gripes.
Will clean them up and apply in a few hours (I have to leave for an
appointment shortly).
regards, tom lane
Jan Urbański <wulczer@wulczer.org> writes:
> Here are two patches that fix two separate bugs that you found
> simultaneously. Because they're actually separate issues, it turned out
> fixing them was a bit more tricky than I expected (fixing one was
> unmasking the other one etc).
Applied with one non-cosmetic change: I got rid of the test on
TransactionIdIsValid(arg->typrel_xmin) in PLy_input_tuple_funcs,
as well as where you'd copied that logic in PLy_output_tuple_funcs.
AFAICS skipping the update on the xmin/tid, if we're coming through
there a second time, would be simply wrong.
regards, tom lane
On 17/08/11 23:10, Tom Lane wrote: > Jan Urbański <wulczer@wulczer.org> writes: >> Here are two patches that fix two separate bugs that you found >> simultaneously. Because they're actually separate issues, it turned out >> fixing them was a bit more tricky than I expected (fixing one was >> unmasking the other one etc). > > Applied with one non-cosmetic change: I got rid of the test on > TransactionIdIsValid(arg->typrel_xmin) in PLy_input_tuple_funcs, > as well as where you'd copied that logic in PLy_output_tuple_funcs. > AFAICS skipping the update on the xmin/tid, if we're coming through > there a second time, would be simply wrong. Thanks! The way things are set up now I think you never go through PLy_input_tuple_funcs twice, unless the cache is determined to be invalid and then you recreate the function from scratch. But of course it's better to be safe than sorry and even if I'm right and it was never executed twice, any refactoring effort might have broken it easily. Cheers, Jan