Thread: further explain changes
Per recent discussion on pgsql-performance, and per discussion on -hackers that it might not be too late for small patches after all, here is a patch (as yet without documentation) which adds some additional instrumentation to EXPLAIN for hashes: number of buckets, number of batches, original number of batches, and peak memory utilization. Thoughts? I was also thinking about the possibility of adding a new option called "output" and making that control whether the "Output" line gets printed. It's kind of annoying to use EXPLAIN (ANALYZE, VERBOSE) right now (and moreso with this patch) specifically because of that line, which is quite... verbose. If we're going to change it ever we should do it for 9.0, since we've made a lot of other changes that people will be adjusting for anyhow. Also, do we want to change the schema URL? The existing URL was suggested by Peter but IIRC there was some thought that it might not be the best choice. http://www.postgresql.org/2009/explain ...Robert
Attachment
On Sat, Jan 23, 2010 at 10:08 PM, Robert Haas <robertmhaas@gmail.com> wrote: > > I was also thinking about the possibility of adding a new option > called "output" and making that control whether the "Output" line gets > printed. It's kind of annoying to use EXPLAIN (ANALYZE, VERBOSE) why not let it go in ANALYZE, just as the sort info -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157
Le 24/01/2010 06:06, Jaime Casanova a écrit : > On Sat, Jan 23, 2010 at 10:08 PM, Robert Haas <robertmhaas@gmail.com> wrote: >> >> I was also thinking about the possibility of adding a new option >> called "output" and making that control whether the "Output" line gets >> printed. It's kind of annoying to use EXPLAIN (ANALYZE, VERBOSE) > > why not let it go in ANALYZE, just as the sort info > Yes, it would be more consistent. Other than that, this patch is quite interesting. -- Guillaume.http://www.postgresqlfr.orghttp://dalibo.com
<p>isn't that line pretty much the main point of "verbose"? I would assume anything new like this would get its own optionwhich might default to on.<p>greg<p><blockquote type="cite">On 24 Jan 2010 03:08, "Robert Haas" <<a href="mailto:robertmhaas@gmail.com">robertmhaas@gmail.com</a>>wrote:<br /><br />Per recent discussion on pgsql-performance,and per discussion on<br /> -hackers that it might not be too late for small patches after all,<br /> hereis a patch (as yet without documentation) which adds some<br /> additional instrumentation to EXPLAIN for hashes: numberof buckets,<br /> number of batches, original number of batches, and peak memory<br /> utilization. Thoughts?<br /><br/> I was also thinking about the possibility of adding a new option<br /> called "output" and making that control whetherthe "Output" line gets<br /> printed. It's kind of annoying to use EXPLAIN (ANALYZE, VERBOSE)<br /> right now (andmoreso with this patch) specifically because of that<br /> line, which is quite... verbose. If we're going to changeit ever we<br /> should do it for 9.0, since we've made a lot of other changes that<br /> people will be adjustingfor anyhow.<br /><br /> Also, do we want to change the schema URL? The existing URL was<br /> suggested by Peterbut IIRC there was some thought that it might not<br /> be the best choice.<br /><br /><a href="http://www.postgresql.org/2009/explain"target="_blank">http://www.postgresql.org/2009/explain</a><br /><font color="#888888"><br/> ...Robert<br /></font><br /><br /> --<br /> Sent via pgsql-hackers mailing list (<a href="mailto:pgsql-hackers@postgresql.org">pgsql-hackers@postgresql.org</a>)<br/> To make changes to your subscription:<br/><a href="http://www.postgresql.org/mailpref/pgsql-hackers" target="_blank">http://www.postgresql.org/mailpref/pgsql-hackers</a><br/><br /></blockquote>
On Sun, Jan 24, 2010 at 12:06 AM, Jaime Casanova <jcasanov@systemguards.com.ec> wrote: > On Sat, Jan 23, 2010 at 10:08 PM, Robert Haas <robertmhaas@gmail.com> wrote: >> >> I was also thinking about the possibility of adding a new option >> called "output" and making that control whether the "Output" line gets >> printed. It's kind of annoying to use EXPLAIN (ANALYZE, VERBOSE) > > why not let it go in ANALYZE, just as the sort info It's kinda long-winded - it adds like 4 extra lines for each hash join. I don't think I want to add that much clutter to regular E-A output. If people don't like making it controlled by VERBOSE, then I vote for a new option, as Greg Stark suggested. ...Robert
Robert Haas <robertmhaas@gmail.com> writes:
> On Sun, Jan 24, 2010 at 12:06 AM, Jaime Casanova
>> why not let it go in ANALYZE, just as the sort info
> It's kinda long-winded - it adds like 4 extra lines for each hash
> join. I don't think I want to add that much clutter to regular E-A
> output.
Well, that would only happen if you're deliberately obtuse about the
formatting. The sort code manages to fit all the extra on one line,
and I don't see why hash couldn't.
I'd vote for just adding it in the exact same cases that sort adds extra
info. -1 for either adding a new option or changing the meaning of the
ones that are there.
regards, tom lane
On Sun, Jan 24, 2010 at 12:30 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> On Sun, Jan 24, 2010 at 12:06 AM, Jaime Casanova >>> why not let it go in ANALYZE, just as the sort info > >> It's kinda long-winded - it adds like 4 extra lines for each hash >> join. I don't think I want to add that much clutter to regular E-A >> output. > > Well, that would only happen if you're deliberately obtuse about the > formatting. The sort code manages to fit all the extra on one line, > and I don't see why hash couldn't. > > I'd vote for just adding it in the exact same cases that sort adds extra > info. -1 for either adding a new option or changing the meaning of the > ones that are there. Care to suggest a format? ...Robert
Robert Haas <robertmhaas@gmail.com> writes:
> Care to suggest a format?
As much like the sort case as possible ...
regards, tom lane
On Sun, Jan 24, 2010 at 12:30 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> On Sun, Jan 24, 2010 at 12:06 AM, Jaime Casanova >>> why not let it go in ANALYZE, just as the sort info > >> It's kinda long-winded - it adds like 4 extra lines for each hash >> join. I don't think I want to add that much clutter to regular E-A >> output. > > Well, that would only happen if you're deliberately obtuse about the > formatting. The sort code manages to fit all the extra on one line, > and I don't see why hash couldn't. OK, here's a new version. ...Robert
Attachment
On Thu, Jan 28, 2010 at 4:18 PM, Robert Haas <robertmhaas@gmail.com> wrote: > On Sun, Jan 24, 2010 at 12:30 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> Robert Haas <robertmhaas@gmail.com> writes: >>> On Sun, Jan 24, 2010 at 12:06 AM, Jaime Casanova >>>> why not let it go in ANALYZE, just as the sort info >> >>> It's kinda long-winded - it adds like 4 extra lines for each hash >>> join. I don't think I want to add that much clutter to regular E-A >>> output. >> >> Well, that would only happen if you're deliberately obtuse about the >> formatting. The sort code manages to fit all the extra on one line, >> and I don't see why hash couldn't. > > OK, here's a new version. > OK, i have 3 hashes in a query and i got these 3 lines in an EXPLAIN ANALYZE with your patch """ -> Hash (cost=3878.94..3878.94 rows=83594 width=34) (actual time=504.648..504.648 rows=83594 loops=1) Buckets: 2048 Batches: 8 MemoryUsage: 589kB [...] -> Hash (cost=14.49..14.49 rows=649 width=15) (actual time=2.901..2.901 rows=649 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 32kB [...] -> Hash (cost=977.62..977.62 rows=6555 width=16) (actual time=60.913..60.913 rows=6556 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 308kB """ I guess Memory Usage is per Batch, right? is this an average? What is the unit measure for Bucket? there are 14 temp files generated for this query and the only Sort says it was on memory so these files should come from the hashes, can we say that in the explain analyze? mmm... that memory usage, could be Disk actually? -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157
On Sat, Jan 30, 2010 at 12:26 PM, Jaime Casanova <jcasanov@systemguards.com.ec> wrote: > On Thu, Jan 28, 2010 at 4:18 PM, Robert Haas <robertmhaas@gmail.com> wrote: >> On Sun, Jan 24, 2010 at 12:30 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >>> Robert Haas <robertmhaas@gmail.com> writes: >>>> On Sun, Jan 24, 2010 at 12:06 AM, Jaime Casanova >>>>> why not let it go in ANALYZE, just as the sort info >>> >>>> It's kinda long-winded - it adds like 4 extra lines for each hash >>>> join. I don't think I want to add that much clutter to regular E-A >>>> output. >>> >>> Well, that would only happen if you're deliberately obtuse about the >>> formatting. The sort code manages to fit all the extra on one line, >>> and I don't see why hash couldn't. >> >> OK, here's a new version. >> > > OK, i have 3 hashes in a query and i got these 3 lines in an EXPLAIN > ANALYZE with your patch > """ > -> Hash (cost=3878.94..3878.94 rows=83594 > width=34) (actual time=504.648..504.648 rows=83594 loops=1) > Buckets: 2048 Batches: 8 Memory Usage: 589kB > [...] > -> Hash (cost=14.49..14.49 rows=649 width=15) > (actual time=2.901..2.901 rows=649 loops=1) > Buckets: 1024 Batches: 1 Memory Usage: 32kB > [...] > -> Hash (cost=977.62..977.62 rows=6555 width=16) > (actual time=60.913..60.913 rows=6556 loops=1) > Buckets: 1024 Batches: 1 Memory Usage: 308kB > """ > > I guess Memory Usage is per Batch, right? is this an average? It's the maximum memory every used by that hash. > What is the unit measure for Bucket? There is no unit - it's just how many buckets. > there are 14 temp files generated for this query and the only Sort > says it was on memory so these files should come from the hashes, can > we say that in the explain analyze? I think maybe it's 2 temp files per batch in excess of 1, thus (8 - 1) x 2 = 14 for the 8-batch join and none for the other two. Perhaps you'd care to test that hypothesis? > mmm... that memory usage, could be > Disk actually? No. ...Robert