== PostgreSQL Weekly News - September 30, 2018 == - Mailing list pgsql-announce
| From | David Fetter |
|---|---|
| Subject | == PostgreSQL Weekly News - September 30, 2018 == |
| Date | |
| Msg-id | 20180930223004.GA15382@fetter.org Whole thread Raw |
| List | pgsql-announce |
== PostgreSQL Weekly News - September 30, 2018 == PGDay.IT 2019 will take place May 16th and May 17th in Bologna, Italy. The CfP is open at https://2019.pgday.it/en/blog/cfp and the Call for Workshops is at https://2019.pgday.it/en/blog/cfw until January 15, 2019. https://2019.pgday.it/en/ pgDay Paris 2019 will be held in Paris, France on March 12, 2019 at 199bis rue Saint-Martin. The CfP is open until November 30, 2018. http://2019.pgday.paris/callforpapers/ == PostgreSQL Product News == Ora2Pg 19.1, a tool for migrating Oracle databases to PostgreSQL, released. http://ora2pg.darold.net/ PostGIS 2.5.0 the industry standard geographic information system package for PostgreSQL, released. http://postgis.net/2018/09/23/postgis-2.5.0/ == PostgreSQL Jobs for September == http://archives.postgresql.org/pgsql-jobs/2018-9/ == PostgreSQL Local == PostgresConf South Africa 2018 will take place in Johannesburg on October 9, 2018 https://postgresconf.org/conferences/SouthAfrica2018 PostgreSQL Conference Europe 2018 will be held on October 23-26, 2018 at the Lisbon Marriott Hotel in Lisbon, Portugal. https://2018.pgconf.eu/ 2Q PGConf will be on December 4-5, 2018 in Chicago, IL. http://www.2qpgconf.com/ PGConf.ASIA 2018 will take place on December 10-12, 2018 in Akihabara, Tokyo, Japan. http://www.pgconf.asia/EN/2018/ == PostgreSQL in the News == Planet PostgreSQL: http://planet.postgresql.org/ PostgreSQL Weekly News is brought to you this week by David Fetter Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org. == Applied Patches == Tom Lane pushed: - Fix failure in WHERE CURRENT OF after rewinding the referenced cursor. In a case where we have multiple relation-scan nodes in a cursor plan, such as a scan of an inheritance tree, it's possible to fetch from a given scan node, then rewind the cursor and fetch some row from an earlier scan node. In such a case, execCurrent.c mistakenly thought that the later scan node was still active, because ExecReScan hadn't done anything to make it look not-active. We'd get some sort of failure in the case of a SeqScan node, because the node's scan tuple slot would be pointing at a HeapTuple whose t_self gets reset to invalid by heapam.c. But it seems possible that for other relation scan node types we'd actually return a valid tuple TID to the caller, resulting in updating or deleting a tuple that shouldn't have been considered current. To fix, forcibly clear the ScanTupleSlot in ExecScanReScan. Another issue here, which seems only latent at the moment but could easily become a live bug in future, is that rewinding a cursor does not necessarily lead to *immediately* applying ExecReScan to every scan-level node in the plan tree. Upper-level nodes will think that they can postpone that call if their child node is already marked with chgParam flags. I don't see a way for that to happen today in a plan tree that's simple enough for execCurrent.c's search_plan_tree to understand, but that's one heck of a fragile assumption. So, add some logic in search_plan_tree to detect chgParam flags being set on nodes that it descended to/through, and assume that that means we should consider lower scan nodes to be logically reset even if their ReScan call hasn't actually happened yet. Per bug #15395 from Matvey Arye. This has been broken for a long time, so back-patch to all supported branches. Discussion: https://postgr.es/m/153764171023.14986.280404050547008575@wrigleys.postgresql.org https://git.postgresql.org/pg/commitdiff/89b280e139c463c98eb33592216a123e89052d08 - Doc: warn against using parallel restore with --load-via-partition-root. This isn't terribly safe, and making it so doesn't seem like a small project, so for the moment just warn against it. Discussion: https://postgr.es/m/13624.1535486019@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/73a60051379b35a0bec399edfe369c59e50cc775 - Fix over-allocation of space for array_out()'s result string. array_out overestimated the space needed for its output, possibly by a very substantial amount if the array is multi-dimensional, because of wrong order of operations in the loop that counts the number of curly-brace pairs needed. While the output string is normally short-lived, this could still cause problems in extreme cases. An additional minor error was that it counted one more delimiter than is actually needed. Repair those errors, add an Assert that the space is now correctly calculated, and make some minor improvements in the comments. I also failed to resist the temptation to get rid of an integer modulus operation per array element; a simple comparison is sufficient. This bug dates clear back to Berkeley days, so back-patch to all supported versions. Keiichi Hirobe, minor additional work by me Discussion: https://postgr.es/m/CAH=EFxE9W0tRvQkixR2XJRRCToUYUEDkJZk6tnADXugPBRdcdg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/87d9bbca13f9c6b8f6ee986f0e399cb83bd731d4 - Use ppoll(2), if available, to wait for input in pgbench. Previously, pgbench always used select(2) for this purpose, but that's problematic for very high client counts, because select() can't deal with file descriptor numbers larger than FD_SETSIZE. It's pretty common for that to be only 1024 or so, whereas modern OSes can allow many more open files than that. Using poll(2) would surmount that problem, but it creates another one: poll()'s timeout resolution is only 1ms, which is poor enough to cause problems with --rate specifications approaching or exceeding 1K TPS. On platforms that have ppoll(2), which includes Linux and recent FreeBSD, we can use that to avoid the FD_SETSIZE problem without any loss of timeout resolution. Hence, add configure logic to test for ppoll(), and use it if available. This patch introduces an abstraction layer into pgbench that could be extended to support other kernel event-wait APIs such as kevents. But actually adding such support is a matter for some future patch. Doug Rady, reviewed by Robert Haas and Fabien Coelho, and whacked around a good bit more by me Discussion: https://postgr.es/m/23D017C9-81B7-484D-8490-FD94DEC4DF59@amazon.com https://git.postgresql.org/pg/commitdiff/60e612b602999e670f2d57a01e52799eaa903ca9 - Sync our Snowball stemmer dictionaries with current upstream. We haven't touched these since text search functionality landed in core in 2007 :-(. While the upstream project isn't a beehive of activity, they do make additions and bug fixes from time to time. Update our copies of these files. Also update our documentation about how to keep things in sync, since they're not making distribution tarballs these days. Fortunately, their source code turns out to be a breeze to build. Notable changes: * The non-UTF8 version of the hungarian stemmer now works in LATIN2 not LATIN1. * New stemmers have appeared for arabic, indonesian, irish, lithuanian, nepali, and tamil. These all work in UTF8, and the indonesian and irish ones also work in LATIN1. (There are some new stemmers that I did not incorporate, mainly because their names don't match the underlying languages, suggesting that they're not to be considered mainstream.) Worth noting: the upstream Nepali dictionary was contributed by Arthur Zakirov. initdb forced because the contents of snowball_create.sql have changed. Still TODO: see about updating the stopword lists. Arthur Zakirov, minor mods and doc work by me Discussion: https://postgr.es/m/20180626122025.GA12647@zakirov.localdomain Discussion: https://postgr.es/m/20180219140849.GA9050@zakirov.localdomain https://git.postgresql.org/pg/commitdiff/fd582317e10e26083b8c720598bfcdbf89787112 - Avoid unnecessary precision loss for pgbench's --rate target. It's fairly silly to truncate the throttle_delay to integer when the only math we ever do with it requires converting back to double. Furthermore, given that people are starting to complain about restrictions like only supporting 1K client connections, I don't think we're very far away from situations where the precision loss matters. As the code stood, for example, there's no difference between --rate 100001 and --rate 111111; both get converted to throttle_delay = 9. Somebody trying to run 100 threads and have each one dispatch around 1K TPS would find this lack of precision rather surprising, especially since the required per-thread delays are around 1ms, well within the timing precision of modern systems. https://git.postgresql.org/pg/commitdiff/5b7e036707ccd93506731da82a56b07023d13e30 - Make some fixes to allow building Postgres on macOS 10.14 ("Mojave"). Apple's latest rearrangements of the system-supplied headers have broken building of PL/Perl and PL/Tcl. The only practical way to fix PL/Tcl is to start using the "-isysroot" compiler flag to point to SDK-supplied headers, as Apple expects. We must also start distinguishing where to find Perl's headers from where to find its shared library; but that seems like good cleanup anyway. Extensions that formerly did something like -I$(perl_archlibexp)/CORE should now do -I$(perl_includedir)/CORE instead. perl_archlibexp is still the place to look for libperl.so, though. If for some reason you don't like the default -isysroot setting, you can override that by setting PG_SYSROOT in configure's arguments. I don't currently think people would need to do so, unless maybe for cross-version build purposes. In addition, teach configure where to find tclConfig.sh. Our traditional method of searching $auto_path hasn't worked for the last couple of macOS releases, and it now seems clear that Apple's not going to change that. The workaround of manually specifying --with-tclconfig was annoying already, but Mojave's made it a lot more so because the sysroot path now has to be included as well. Let's just wire the knowledge into configure instead. To avoid breaking builds against non-default Tcl installations (e.g. MacPorts) wherein the $auto_path method probably still works, arrange to try the additional case only after all else has failed. Back-patch to all supported versions, since at least the buildfarm cares about that. The changes are set up to not do anything on macOS releases that are old enough to not have functional sysroot trees. https://git.postgresql.org/pg/commitdiff/5e22171310f8d7c82219a6b978440e5144e88683 - Convert elog.c's useful_strerror() into a globally-used strerror wrapper. elog.c has long had a private strerror wrapper that handles assorted possible failures or deficiencies of the platform's strerror. On Windows, it also knows how to translate Winsock error codes, which the native strerror does not. Move all this code into src/port/strerror.c and define strerror() as a macro that invokes it, so that both our frontend and backend code will have all of this behavior. I believe this constitutes an actual bug fix on Windows, since AFAICS our frontend code did not report Winsock error codes properly before this. However, the main point is to lay the groundwork for implementing %m in src/port/snprintf.c: the behavior we want %m to have is this one, not the native strerror's. Note that this throws away the prior use of src/port/strerror.c, which was to implement strerror() on platforms lacking it. That's been dead code for nigh twenty years now, since strerror() was already required by C89. We should likewise cause strerror_r to use this behavior, but I'll tackle that separately. Patch by me, reviewed by Michael Paquier Discussion: https://postgr.es/m/2975.1526862605@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/26e9d4d4ef16b5e2be96319f89ea6ba7f63a4d73 - Always use our own versions of *printf(). We've spent an awful lot of effort over the years in coping with platform-specific vagaries of the *printf family of functions. Let's just forget all that mess and standardize on always using src/port/snprintf.c. This gets rid of a lot of configure logic, and it will allow a saner approach to dealing with %m (though actually changing that is left for a follow-on patch). Preliminary performance testing suggests that as it stands, snprintf.c is faster than the native printf functions for some tasks on some platforms, and slower for other cases. A pending patch will improve that, though cases with floating-point conversions will doubtless remain slower unless we want to put a *lot* of effort into that. Still, we've not observed that *printf is really a performance bottleneck for most workloads, so I doubt this matters much. Patch by me, reviewed by Michael Paquier Discussion: https://postgr.es/m/2975.1526862605@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/96bf88d52711ad3a0a4cc2d1d9cb0e2acab85e63 - Implement %m in src/port/snprintf.c, and teach elog.c to rely on that. I started out with the idea that we needed to detect use of %m format specs in contexts other than elog/ereport calls, because we couldn't rely on that working in *printf calls. But a better answer is to fix things so that it does work. Now that we're using snprintf.c all the time, we can implement %m in that and we've fixed the problem. This requires also adjusting our various printf-wrapping functions so that they ensure "errno" is preserved when they call snprintf.c. Remove elog.c's handmade implementation of %m, and let it rely on snprintf to support the feature. That should provide some performance gain, though I've not attempted to measure it. There are a lot of places where we could now simplify 'printf("%s", strerror(errno))' into 'printf("%m")', but I'm not in any big hurry to make that happen. Patch by me, reviewed by Michael Paquier Discussion: https://postgr.es/m/2975.1526862605@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/d6c55de1f99a9028540516316b95321a7b12a540 - Fix link failures due to snprintf/strerror changes. snprintf.c requires isnan(), which requires -lm on some platforms. libpq never bothered with -lm before, but now it needs it. strerror.c tries to translate a string or two, which requires -lintl. We'd managed never to need that anywhere in ecpg/pgtypeslib/ before, but now we do. Per buildfarm and a report from Peter Eisentraut. Discussion: https://postgr.es/m/20180926190934.ea4xvzhkayuw7gkx@alap3.anarazel.de Discussion: https://postgr.es/m/f67b5008-9f01-057f-2bff-558cb53af851@2ndquadrant.com https://git.postgresql.org/pg/commitdiff/a6b88d682cbec73474a73c9782fb7096e9440a8b - Clean up *printf macros to avoid conflict with format archetypes. We must define the macro "printf" with arguments, else it can mess up format archetype attributes in builds where PG_PRINTF_ATTRIBUTE is just "printf". Fortunately, that's easy to do now that we're requiring C99; we can use __VA_ARGS__. On the other hand, it's better not to use __VA_ARGS__ for the rest of the *printf crew, so that one can take the addresses of those functions without surprises. I'd proposed doing this some time ago, but forgot to make it happen; buildfarm failures subsequent to 96bf88d52 reminded me. Discussion: https://postgr.es/m/22709.1535135640@sss.pgh.pa.us Discussion: https://postgr.es/m/20180926190934.ea4xvzhkayuw7gkx@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/8b91d258844afa58e856ac354f9ba9745ff9ffb2 - Try another way to detect the result type of strerror_r(). The method we've traditionally used, of redeclaring strerror_r() to see if the compiler complains of inconsistent declarations, turns out not to work reliably because some compilers only report a warning, not an error. Amazingly, this has gone undetected for years, even though it certainly breaks our detection of whether strerror_r succeeded. Let's instead test whether the compiler will take the result of strerror_r() as a switch() argument. It's possible this won't work universally either, but it's the best idea I could come up with on the spur of the moment. We should probably back-patch this once the dust settles, but first let's see what the buildfarm thinks of it. Discussion: https://postgr.es/m/10877.1537993279@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/751f532b9766fb5d3c334758abea95b7bb085c5a - Fix another portability issue from commit 758ce9b77. strerror.c now requires strlcpy() in some cases, and a couple of the ecpg libraries did not have that at hand. Pull it in from src/port/ following the usual recipe. Per buildfarm. https://git.postgresql.org/pg/commitdiff/ce4887bd025b95c7b455fefd817a418844c6aad3 - Incorporate strerror_r() into src/port/snprintf.c, too. This provides the features that used to exist in useful_strerror() for users of strerror_r(), too. Also, standardize on the GNU convention that strerror_r returns a char pointer that may not be NULL. I notice that libpq's win32.c contains a variant version of strerror_r that probably ought to be folded into strerror.c. But lacking a Windows environment, I should leave that to somebody else. Discussion: https://postgr.es/m/2975.1526862605@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/758ce9b7794845f95473c569155d29fcf0e2751b - Fix assorted bugs in pg_get_partition_constraintdef(). It failed if passed a nonexistent relation OID, or one that was a non-heap relation, because of blindly applying heap_open to a user-supplied OID. This is not OK behavior for a SQL-exposed function; we have a project policy that we should return NULL in such cases. Moreover, since pg_get_partition_constraintdef ought now to work on indexes, restricting it to heaps is flat wrong anyway. The underlying function generate_partition_qual() wasn't on board with indexes having partition quals either, nor for that matter with rels having relispartition set but yet null relpartbound. (One wonders whether the person who wrote the function comment blocks claiming that these functions allow a missing relpartbound had ever tested it.) Fix by testing relispartition before opening the rel, and by using relation_open not heap_open. (If any other relkinds ever grow the ability to have relispartition set, the code will work with them automatically.) Also, don't reject null relpartbound in generate_partition_qual. Back-patch to v11, and all but the null-relpartbound change to v10. (It's not really necessary to change generate_partition_qual at all in v10, but I thought s/heap_open/relation_open/ would be a good idea anyway just to keep the code in sync with later branches.) Per report from Justin Pryzby. Discussion: https://postgr.es/m/20180927200020.GJ776@telsasoft.com https://git.postgresql.org/pg/commitdiff/aaf10f32a308bc5f53772c773edf3f345f59bb74 - Build src/port files as a library with -fPIC, and use that in libpq. libpq and ecpg need shared-library-friendly versions of assorted src/port/ and src/common/ modules. Up to now, they got those by symlinking the individual source files and compiling them locally. That's baroque, and a pain to maintain, and it results in some amount of duplicated compile work. It might've made sense when only a couple of files were needed, but the list has grown and grown and grown :-( It makes more sense to have the originating directory build a third variant of libpgport.a/libpgcommon.a containing modules built with $(CFLAGS_SL), and just link that into the shared library. Unused files won't get linked, so the end result should be the same. This patch makes a down payment on that idea by having src/port/ build such a library and making libpq use it. If the buildfarm doesn't expose fatal problems with the approach, I'll extend it to the other cases. Discussion: https://postgr.es/m/13022.1538003440@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/ea53100d5671b5b243f77898b0b04d23c38b2820 - Remove pqsignal() from libpq's official exports list. Client applications should get this function, if they need it, from libpgport. The fact that it's exported from libpq is a hack left over from before we set up libpgport. It's never been documented, and there's no good reason for non-PG code to be calling it anyway, so hopefully this won't cause any problems. Moreover, with the previous setup it was not real clear whether our clients that use the function were getting it from libpgport or libpq, so this might actually prevent problems. The reason for changing it now is that in the wake of commit ea53100d5, some linkers won't export the symbol, apparently because it's coming from a .a library instead of a .o file. We could get around that by continuing to symlink pqsignal.c into libpq as before; but unless somebody complains very hard, I don't want to adopt such a kluge. Discussion: https://postgr.es/m/13022.1538003440@sss.pgh.pa.us Discussion: https://postgr.es/m/E1g5Y8r-0006vs-QA@gemulon.postgresql.org https://git.postgresql.org/pg/commitdiff/f7ab802855200df5529a6e1e7b748d7926acace8 - Build src/common files as a library with -fPIC. Build a third version of libpgcommon.a, with -fPIC and -DFRONTEND, as commit ea53100d5 did for src/port. Use that in libpq to avoid symlinking+rebuilding source files retail. Also adjust ecpg to use the new src/port and src/common libraries. Arrange to install these libraries, too, to simplify out-of-tree builds of shared libraries that need any of these modules. Discussion: https://postgr.es/m/13022.1538003440@sss.pgh.pa.us Discussion: https://postgr.es/m/E1g5Y8r-0006vs-QA@gemulon.postgresql.org https://git.postgresql.org/pg/commitdiff/7143b3e82136d2941b3394168940d895a29b4f36 - Tweak MSVC build system to match changes in 7143b3e82. Also try to make the comment suggesting that this might be needed more intelligible. Per buildfarm. https://git.postgresql.org/pg/commitdiff/97c6852ff7c54d5b44426ceae9e3215d13157c10 - Tweak MSVC build system to match changes in 7143b3e82. Looks like we need to pull in $libpgcommon in a couple more places than before. Per buildfarm. https://git.postgresql.org/pg/commitdiff/61f14cc8c85b5e6186c3b86c2c929821d7b33589 - Improve error reporting for unsupported effective_io_concurrency setting. Give a specific error complaining about lack of posix_fadvise() when someone tries to set effective_io_concurrency > 0 on platforms without that. This probably isn't worth extensive back-patching, but I (tgl) felt cramming it into v11 was reasonable. James Robinson Discussion: https://postgr.es/m/153771876450.14994.560017943128223619@wrigleys.postgresql.org Discussion: https://postgr.es/m/A3942987-5BC7-4F05-B54D-2A0EC2914B33@jlr-photo.com https://git.postgresql.org/pg/commitdiff/2b04dfc4724970231ac338aa71e529c823fc5ff6 - Create an RTE field to record the query's lock mode for each relation. Add RangeTblEntry.rellockmode, which records the appropriate lock mode for each RTE_RELATION rangetable entry (either AccessShareLock, RowShareLock, or RowExclusiveLock depending on the RTE's role in the query). This patch creates the field and makes all creators of RTE nodes fill it in reasonably, but for the moment nothing much is done with it. The plan is to replace assorted post-parser logic that re-determines the right lockmode to use with simple uses of rte->rellockmode. For now, just add Asserts in each of those places that the rellockmode matches what they are computing today. (In some cases the match isn't perfect, so the Asserts are weaker than you might expect; but this seems OK, as per discussion.) This passes check-world for me, but it seems worth pushing in this state to see if the buildfarm finds any problems in cases I failed to test. catversion bump due to change of stored rules. Amit Langote, reviewed by David Rowley and Jesper Pedersen, and whacked around a bit more by me Discussion: https://postgr.es/m/468c85d9-540e-66a2-1dde-fec2b741e688@lab.ntt.co.jp https://git.postgresql.org/pg/commitdiff/fdba460a26af919c0b366755d119f384706e670a Noah Misch pushed: - Initialize random() in bootstrap/stand-alone postgres and in initdb. This removes a difference between the standard IsUnderPostmaster execution environment and that of --boot and --single. In a stand-alone backend, "SELECT random()" always started at the same seed. On a system capable of using posix shared memory, initdb could still conclude "selecting dynamic shared memory implementation ... sysv". Crashed --boot or --single postgres processes orphaned shared memory objects having names that collided with the not-actually-random names that initdb probed. The sysv fallback appeared after ten crashes of --boot or --single postgres. Since --boot and --single are rare in production use, systems used for PostgreSQL development are the principal candidate to notice this symptom. Back-patch to 9.3 (all supported versions). PostgreSQL 9.4 introduced dynamic shared memory, but 9.3 does share the "SELECT random()" problem. Reviewed by Tom Lane and Kyotaro HORIGUCHI. Discussion: https://postgr.es/m/20180915221546.GA3159382@rfd.leadboat.com https://git.postgresql.org/pg/commitdiff/d18f6674bd60e923bcdbf0fd916685b0a250c60f Joe Conway pushed: - Document aclitem functions and operators. aclitem functions and operators have been heretofore undocumented. Fix that. While at it, ensure the non-operator aclitem functions have pg_description strings. Does not seem worthwhile to back-patch. Author: Fabien Coelho, with pg_description from John Naylor, and significant refactoring and editorialization by me. Reviewed by: Tom Lane Discussion: https://postgr.es/m/flat/alpine.DEB.2.21.1808010825490.18204%40lancre https://git.postgresql.org/pg/commitdiff/c62dd80cdf149e2792b13c13777a539f5abb0370 Andrew Dunstan pushed: - Fast default trigger and expand_tuple fixes. Ensure that triggers get properly filled in tuples for the OLD value. Also fix the logic of detecting missing null values. The previous logic failed to detect a missing null column before the first missing column with a default. Fixing this has simplified the logic a bit. Regression tests are added to test changes. This should ensure better coverage of expand_tuple(). Original bug reports, and some code and test scripts from Tomas Vondra Backpatch to release 11. https://git.postgresql.org/pg/commitdiff/7636e5c60fea83a9f3cd2ad278c0819b98941c74 Andres Freund pushed: - Make EXPLAIN output for JIT compilation more dense. A discussion about also reporting JIT compilation overhead on workers brought unhappiness with the verbosity of the current explain format to light. Make the text format more dense, and restructure the structured output to mirror that more closely. As we're re-jiggering the output format anyway: The denser format allows us to report all flags for JIT compilation (now also reporting PGJIT_EXPR and PGJIT_DEFORM), and report the total time in addition to the individual times. Per complaint from Tom Lane. Author: Andres Freund Discussion: https://postgr.es/m/27812.1537221015@sss.pgh.pa.us Backpatch: 11-, where JIT compilation was introduced https://git.postgresql.org/pg/commitdiff/52050ad8ebec8d831902f587314aa4f6aaa6d2c5 - auto_explain: Include JIT information if applicable. Due to my (Andres') omission auto_explain did not include information about JIT compilation. Fix that. Author: Lukas Fittl Discussion: https://postgr.es/m/CAP53PkzgSyoTCau0-5FNaM484B=uO8nLzma7L1ncWLb1=oVJQA@mail.gmail.com Backpatch: 11-, where JIT compilation was introduced https://git.postgresql.org/pg/commitdiff/b076eb7669d7279d0f446305c2e12dffd6bc3347 - Collect JIT instrumentation from workers. Previously, when using parallel query, EXPLAIN (ANALYZE)'s JIT compilation timings did not include the overhead from doing so on the workers. Fix that. We do so by simply aggregating the cost of doing JIT compilation on workers and the leader together. Arguably that's not quite accurate, because the total time spend doing so is spent in parallel - but it's hard to do much better. For additional detail, when VERBOSE is specified, the stats for workers are displayed separately. Author: Amit Khandekar and Andres Freund Discussion: https://postgr.es/m/CAJ3gD9eLrz51RK_gTkod+71iDcjpB_N8eC6vU2AW-VicsAERpQ@mail.gmail.com Backpatch: 11- https://git.postgresql.org/pg/commitdiff/33001fd7a7072d483272115a9376478fdc007fb9 - Change TupleTableSlot->tts_nvalid to type AttrNumber. Previously it was an int / 4 bytes. The maximum number of attributes in a tuple is restricted by the maximum value Var->varattno, which is an AttrNumber/int16. Hence use the same data type for TupleTableSlot->tts_nvalid. Author: Ashutosh Bapat Discussion: https://postgr.es/m/20180220224318.gw4oe5jadhpmcdnm@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/a598708ffa8eb72a22eeee4e6f30bc26e4984acd - Remove function list from prologue of execTuples.c. That section is never in sync with the actual routines available and their functionality. Author: Ashutosh Bapat Discussion: https://postgr.es/m/20180220224318.gw4oe5jadhpmcdnm@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/bbdfbb9154fccf5b58ecbbdf4e8989e2fed206f8 - Split ExecStoreTuple into ExecStoreHeapTuple and ExecStoreBufferHeapTuple. Upcoming changes introduce further types of tuple table slots, in preparation of making table storage pluggable. New storage methods will have different representation of tuples, therefore the slot accessor should refer explicitly to heap tuples. Instead of just renaming the functions, split it into one function that accepts heap tuples not residing in buffers, and one accepting ones in buffers. Previously one function was used for both, but that was a bit awkward already, and splitting will allow us to represent slot types for tuples in buffers and normal memory separately. This is split out from the patch introducing abstract slots, as this largely consists out of mechanical changes. Author: Ashutosh Bapat Reviewed-By: Andres Freund Discussion: https://postgr.es/m/20180220224318.gw4oe5jadhpmcdnm@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/29c94e03c7d05d2b29afa1de32795ce178531246 - Remove absolete function TupleDescGetSlot(). TupleDescGetSlot() was kept around for backward compatibility for user-written SRFs. With the TupleTableSlot abstraction work, that code will need to be version specific anyway, so there's no point in keeping the function around any longer. Author: Ashutosh Bapat Reviewed-By: Andres Freund Discussion: https://postgr.es/m/20180220224318.gw4oe5jadhpmcdnm@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/10763358c3f0df48d2ae39b49b0c93be149cceab - Clean up in the wake of TupleDescGetSlot() removal / 10763358c3f. The previous commit wasn't careful enough to remove all traces of TupleDescGetSlot(). Besides fixing the oversight of not removing TupleDescGetSlot()'s declaration, this also removes FuncCallContext->slot. That was documented to be for use in combination with TupleDescGetSlot(), a cursory search over extensions finds no users, and there doesn't seem to be convincing reasons to keep it around. If we later in the v12 release cycle find users, we can re-consider this part of the commit. Reported-By: Michael Paquier Discussion: https://postgr.es/m/20180926000413.GC1659@paquier.xyz https://git.postgresql.org/pg/commitdiff/27e082b0c6e564facfbf54b56090fdcc4bf44cca - Correct overflow handling in pgbench. This patch attempts, although it's quite possible there are a few holes, to properly detect and reported signed integer overflows in pgbench. Author: Fabien Coelho Reviewed-By: Andres Freund Discussion: https://postgr.es/m/20171212052943.k2hlckfkeft3eiio@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/92a0342a90b38b0b007f079d33286f9aefabfe40 Michaël Paquier pushed: - Revoke pg_stat_statements_reset() permissions. Commit 25fff40 has granted execute permission of the function pg_stat_statements_reset() to default role "pg_read_all_stats", but this role is meant to read statistics, and not to reset them. The permissions on this function are revoked from "pg_read_all_stats". The version of pg_stat_statements is bumped up in consequence. Author: Haribabu Kommi Reviewed-by: Michael Paquier, Amit Kapila Discussion: https://postgr.es/m/CAJrrPGf5fCnKqXObpwGN9nMyD--tzOf-7LFCJiz59Z1wJ5qj9A@mail.gmail.com https://git.postgresql.org/pg/commitdiff/edb9797660541b217d23ae7c02b96b496d34fec4 - Ignore publication tables when --no-publications is used. 96e1cb4 has added support for --no-publications in pg_dump, pg_dumpall and pg_restore, but forgot the fact that publication tables also need to be ignored when this option is used. Author: Gilles Darold Reviewed-by: Michael Paquier Discussion: https://postgr.es/m/3f48e812-b0fa-388e-2043-9a176bdee27e@dalibo.com Backpatch-through: 10, where publications have been added. https://git.postgresql.org/pg/commitdiff/08c9917e24683e36dca35201723e47cdc3fa62db - Rework activation of commit timestamps during recovery. The activation and deactivation of commit timestamp tracking has not been handled consistently for a primary or standbys at recovery. The facility can be activated at three different moments of recovery: - The beginning, where a primary would use the GUC value for the decision-making, and where a standby relies on the contents of the control file. - When replaying a XLOG_PARAMETER_CHANGE record at redo. - The end, where both primary and standby rely on the GUC value. Using the GUC value for a primary at the beginning of recovery causes problems with commit timestamp access when doing crash recovery. Particularly, when replaying transaction commits, it could be possible that an attempt to read commit timestamps is done for a transaction which committed at a moment when track_commit_timestamp was disabled. A test case is added to reproduce the failure. The test works down to v11 as it takes advantage of transaction commits within procedures. Reported-by: Hailong Li Author: Masahiko Sawasa, Michael Paquier Reviewed-by: Kyotaro Horiguchi Discussion: https://postgr.es/m/11224478-a782-203b-1f17-e4797b39bdf0@qunar.com Backpatch-through: 9.5, where commit timestamps have been introduced. https://git.postgresql.org/pg/commitdiff/8d28bf500f6536e295e9c3d7b85cdfec1c4dc913 - Add basic regression tests for default monitoring roles. The following default roles gain some coverage: - pg_read_all_stats - pg_read_all_settings Author: Alexandra Ryzhevich Discussion: https://postgr.es/m/CAOt4E5S5WJmDc9YpS1BfyAMQ5C1NEmiYynD6nUz42qVxphqkpA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/f535d5f0c13cb48d85565017e7d56f2075c59978 - Switch flags tracking pending interrupts to sig_atomic_t. Those previously used bool, which should be safe on any modern platforms, however the C standard is clear that it is better to use sig_atomic_t for variables manipulated in signal handlers. This commit adds at the same time PGDLLIMPORT to ClientConnectionLost. Author: Michael Paquier Reviewed-by: Tom Lane, Chris Travers, Andres Freund Discussion: https://postgr.es/m/20180925011311.GD1354@paquier.xyz https://git.postgresql.org/pg/commitdiff/ba16aade337b16028d1e9e156d83417097c13817 - Fix WAL recycling on standbys depending on archive_mode. A restart point or a checkpoint recycling WAL segments treats segments marked with neither ".done" (archiving is done) or ".ready" (segment is ready to be archived) in archive_status the same way for archive_mode being "on" or "always". While for a primary this is fine, a standby running a restart point with archive_mode = on would try to mark such a segment as ready for archiving, which is something that will never happen except after the standby is promoted. Note that this problem applies only to WAL segments coming from the local pg_wal the first time archive recovery is run. Segments part of a self-contained base backup are the most common case where this could happen, however even in this case normally the .done markers would be most likely part of the backup. Segments recovered from an archive are marked as .ready or .done by the startup process, and segments finished streaming are marked as such by the WAL receiver, so they are handled already. Reported-by: Haruka Takatsuka Author: Michael Paquier Discussion: https://postgr.es/m/15402-a453c90ed4cf88b2@postgresql.org Backpatch-through: 9.5, where archive_mode = always has been added. https://git.postgresql.org/pg/commitdiff/78ea8b5daab9237fd42d7a8a836c1c451765499f Thomas Munro pushed: - Constify dsa_size_class_map and use a better type. Author: Mark G Reviewed-by: Tom Lane Discussion: https://postgr.es/m/CAEeOP_Zy_FvVwcAU0UX9nkOhnoR5KN%3D0B6LWX_kv0ZuSc4wbGw%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/64171b32069adcce7f57840143eaca9fbe28ba7e Álvaro Herrera pushed: - Remove fmgr.h inclusion from partition.h. It's not needed anymore. https://git.postgresql.org/pg/commitdiff/62e533d3f129b59794c256eabad44011c9d2b951 - Remove obsolete comment. The documented shortcoming was actually fixed in 4c728f3829 so the comment is not true anymore. https://git.postgresql.org/pg/commitdiff/5913b9bbf351b421141b300e37752e9ab8d85163 Tomáš Vondra: - Improve test coverage of geometric types. This commit significantly increases test coverage of geo_ops.c, adding tests for various issues addressed by 2e2a392de3 (which went undetected for a long time, at least partially due to not being covered). This also removes alternative results expecting -0 on some platforms. Instead the functions are should return the same results everywhere, transforming -0 to 0 if needed. The tests are added to geometric.sql file, sorted by the left hand side of the operators. There are many cross datatype operators, so this seems like the best solution. Author: Emre Hasegeli Reviewed-by: Tomas Vondra Discussion: https://www.postgresql.org/message-id/CAE2gYzxF7-5djV6-cEvqQu-fNsnt%3DEqbOURx7ZDg%2BVv6ZMTWbg%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/a3d2844852dc664718320b15cbc6d6bfa264e66e - Fix problems in handling the line data type. According to the source history, the internal format of line data type has changed, but various functions working with it did were not updated and thus were producing wrong results. This patch addresses various such issues, in particular: * Reject invalid specification A=B=0 on receive * Reject same points on line_construct_pp() * Fix perpendicular operator when negative values are involved * Avoid division by zero on perpendicular operator * Fix intersection and distance operators when neither A nor B are 1 * Return NULL for closest point when objects are parallel * Check whether closest point of line segments is the intersection point * Fix closest point of line segments being on the wrong segment Aside from handling those issues, the patch also aims to make operators more symmetric and less sen to precision loss. The EPSILON interferes with even minor changes, but the least we can do is applying it to both sides of the operators equally. Author: Emre Hasegeli Reviewed-by: Tomas Vondra Discussion: https://www.postgresql.org/message-id/CAE2gYzxF7-5djV6-cEvqQu-fNsnt%3DEqbOURx7ZDg%2BVv6ZMTWbg%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/2e2a392de391a9f4ef4221fccbd00c43ba5c9b40 Peter Eisentraut pushed: - Update dummy CREATE ASSERTION grammar. While we are probably still far away from fully implementing assertions, all patch proposals appear to take issue with the existing dummy grammar CREATE/DROP ASSERTION productions, so update those a little bit. Rename the rule, use any_name instead of name, and remove some unused code. Also remove the production for DROP ASSERTION, since that would most likely be handled via the generic DROP support. extracted from a patch by Joe Wildish https://git.postgresql.org/pg/commitdiff/a49ceda6a044c2fc104b3d1397fe0bef8679d1aa - Recurse to sequences on ownership change for all relkinds. When a table ownership is changed, we must apply that also to any owned sequences. (Otherwise, it would result in a situation that cannot be restored, because linked sequences must have the same owner as the table.) But this was previously only applied to regular tables and materialized views. But it should also apply to at least foreign tables. This patch removes the relkind check altogether, because it doesn't save very much and just introduces the possibility of similar omissions. Bug: #15238 Reported-by: Christoph Berg <christoph.berg@credativ.de> https://git.postgresql.org/pg/commitdiff/0320ddaf3c08ea17d616549d0e7f45239592fc76 Alexander Korotkov pushed: - Remove extra usage of BoxPGetDatum() macro. Author: Mark Dilger Discussion: https://postgr.es/m/B2AEFCD0-836D-4654-9D59-3DF616E0A6F3%40gmail.com https://git.postgresql.org/pg/commitdiff/0f6459589494a4b4ff6c707594f8d308b9da88f8 - Minor formatting cleanup for 2a6368343f. https://git.postgresql.org/pg/commitdiff/4ec90f53f10141867d8b86f58d72990a13ff267b Amit Kapila pushed: - Fix assertion failure when updating full_page_writes for checkpointer. When the checkpointer receives a SIGHUP signal to update its configuration, it may need to update the shared memory for full_page_writes and need to write a WAL record for it. Now, it is quite possible that the XLOG machinery has not been initialized by that time and it will lead to assertion failure while doing that. Fix is to allow the initialization of the XLOG machinery outside critical section. This bug has been introduced by the commit 2c03216d83 which added the XLOG machinery initialization in RecoveryInProgress code path. Reported-by: Dilip Kumar Author: Dilip Kumar Reviewed-by: Michael Paquier and Amit Kapila Backpatch-through: 9.5 Discussion: https://postgr.es/m/CAFiTN-u4BA8KXcQUWDPNgaKAjDXC=C2whnzBM8TAcv=stckYUw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/a86bf6057edf7bc344f1ab6ba7824b561ed9068a Stephen Frost pushed: - Add application_name to connection authorized msg. The connection authorized message has quite a bit of useful information in it, but didn't include the application_name (when provided), so let's add that as it can be very useful. Note that at the point where we're emitting the connection authorized message, we haven't processed GUCs, so it's not possible to get this by using log_line_prefix (which pulls from the GUC). There's also something to be said for having this included in the connection authorized message and then not needing to repeat it for every line, as having it in log_line_prefix would do. The GUC cleans the application name to pure-ascii, so do that here too, but pull out the logic for cleaning up a string into its own function in common and re-use it from those places, and check_cluster_name which was doing the same thing. Author: Don Seiler <don@seiler.us> Discussion: https://postgr.es/m/CAHJZqBB_Pxv8HRfoh%2BAB4KxSQQuPVvtYCzMg7woNR3r7dfmopw%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/8bddc864000f56d396621d4ad0f13e8e1872ddf5 == Pending Patches == Thomas Munro sent in two more revisions of a patch to enable SERIALIZABLE READ ONLY DEFERRABLE for hot standbys. Thomas Munro sent in another revision of a patch to implement a synchronous replay mode for avoiding stale reads on hot standbys. Andrey Borodin sent in another revision of a patch to add a GiST verification function for amcheck. Peter Eisentraut sent in a patch to clarify CREATE TABLESPACE documentation by being more specific about when and how to create the directory and what permissions it should have. Elvis Pranskevichus sent in another revision of a patch to add and report a new GUC: session_read_only. This GUC is reported via a ParameterStatus message. Haribabu Kommi sent in a patch to revoke the pg_stat_statements_reset() permissions from the pg_read_all_stats role, which shouldn't have had them in the first place. Haribabu Kommi sent in three more revisions of a patch to add parameters userid, dbid and queryid to pg_stat_statements_reset(). Jim Nasby sent in a patch to remove is_wraparound and move the check for anti-wraparound VACUUM from vacuum_rel() to lazy_vacuum_rel(). Pavan Deolasee sent in another revision of a patch to implement MERGE per SQL:2016. Thomas Munro sent in a patch to add DNS SRV support for LDAP authentication. Thomas Munro sent in a patch to de-support and remove dsm_resize() and dsm_remap(). Surafel Temesgen sent in another revision of a patch to add a PERCENT option to FETCH FIRST. Michaël Paquier sent in another revision of a patch to fix a bug in track_commit_timestamp. Dmitry Dolgov and Michaël Paquier traded patches to fix a bug that manifested as a segfault when creating partition with a primary key and sql_drop trigger exists. Sergei Kornilov sent in another revision of a patch to refactor the recovery.conf API. David Rowley sent in another revision of a patch to calculate total_table_pages after set_base_rel_sizes. Richard Guo sent in another revision of a patch to implement predicate propagation for non-equivalence clauses. Haozhou Wang sent in another revision of a patch to implement disk quotas. Amul Sul sent in a patch to add a 64-bit hash function for the hstore and citext data types. Fabien COELHO sent in another revision of a patch to add a pseudo-random permutation function to pgbench. Liudmila Mantrova sent in another revision of a patch to clarify the handling of letters and digits in to_date() and to_timestamp(). Nikita Glukhov sent in another revision of a patch to implement kNN searches for btree indexes. Tom Lane and Daniel Gustafsson traded patches to add support for Secure Transport SSL library on macOS as an alternative to OpenSSL. Michael Banck sent in two more revisions of a patch to add online verification of page checksums. Tom Lane sent in another revision of a patch to speed up snprintf. Michaël Paquier sent in a patch to refactor vacuum open. Kyotaro HORIGUCHI sent in another revision of a patch to implement a shared-memory based stats collector. Thomas Munro sent in another revision of a patch to Use pread()/pwrite() instead of lseek() + read()/write() in systems that have the former. Nikita Glukhov sent in a patch to fix a memory leak in PLyObject_ToJsonbValue(). David Rowley sent in a patch to fix some incorrect comments and an outdated README for run-time pruning. Jimmy Yih sent in a patch to get a more consistent view definition when a UNION subquery contains undecorated constants. Peter Eisentraut sent in another revision of a patch to enable using file cloning in pg_upgrade where available. David Rowley and Amit Kapila traded patches to avoid locking range table relations in the executor, remove some useless fields from planner nodes, prune PlanRowMark of relations that are pruned from a plan, and revise executor range table relation opening/closing. Haribabu Kommi sent in another revision of a patch to move declaration of the synchronize_seqscans GUC from src/backend/access/heap/heapam.c to src/backend/access/table/tableam.c, add a check_default_table_access_method hook to verify the access method, and validate the index scan hook addition. Masahiko Sawada sent in a patch to fix how autovacuum is logged. Edmund Horner sent in two more revisions of a patch to improve TID scanning by adding support for range quals to estimating the cost of the path. Haribabu Kommi sent in another revision of a patch to allow target_session_attrs in libpq to accept a prefer-read option. Thomas Munro sent in another revision of a patch to fix some corner case problems with fsync. Amit Khandekar sent in another revision of a patch to allocate a dedicated slot for each partitions that requires it and slotify partition tuple conversion. Peter Eisentraut sent in a patch to advance the transaction timestamp in intra-procedure transactions. David Rowley sent in two more revisions of a patch to fix an incorrect setting of heap insert options for partitioned tables. Michael Banck sent in another revision of a patch to add an option for progress reporting to pg_verify_checksums. Thomas Munro sent in two more revisions of a patch to add kqueue(2) support for WaitEventSet. Michaël Paquier sent in two more revisions of a patch to use durable_unlink for .ready and .done files for WAL segment removal. Andres Freund sent in another revision of a patch to remove abstime and things that depend on it. David Hedberg sent in a patch to add pipe support to pg_dump and pg_restore. Pavel Stěhule sent in two more revisions of a patch to implement schema variables. Marco Atzeri sent in a patch to fix the Cygwin linking rules. Andres Freund sent in a patch to remove WITH OIDs support and convert catalog table oids to explicit oid columns. Amit Kapila sent in another revision of a patch to fix datum serialization. Dmitry Dolgov sent in another revision of a patch to add generic type subscripting and use same for arrays and JSONB. John Naylor sent in a patch to sync the ECPG scanner with core. Tom Lane sent in two revisions of a patch to make executor relation handling better by ensuring that the caller holds a lock *at least as strong* as the one being recorded in the range table entry. Tom Lane sent in a patch to fix a problem where relations are being opened without any lock whatever. Christoph Moench-Tegeder sent in a patch to implement pg_ls_archive_status().
pgsql-announce by date: