Re: RFC: adding pytest as a supported test framework - Mailing list pgsql-hackers

Hi everyone, it's been a while for this thread.

= Status =

Since the last email, quite a bit of offline chatter has happened,
particularly at FOSDEM [1] and PGConf.dev [2]. The dev-meeting
audiences, at least, appear to have reached a critical mass of nodding
heads and cautious optimism around the use of a pytest suite. Some are
more optimistic than others, some are quite skeptical, some are
completely silent, but no one has -1'd it yet.

I think I've been solidly outvoted in my attempt to support Python and
Perl suites side-by-side in perpetuity. Many, many people have
expressed that they'd feel disappointed if that were a permanent
situation, so I've been trying to adapt my approach to the idea that
we would need to port existing Perl tests at _some_ point in the
timeline.

Simultaneously, I've been approached by some people offline who are
pretty clearly waiting for me to make the first move here. I'm a bit
uncomfortable with that -- no one needs my permission to post patches
in this vein! -- but I get that this entire topic is a balancing act,
so it makes some sense. Thank you all for your patience. Here is a
patchset.

= Patches =

This is dev-quality, not targeted for commit yet, but hopefully enough
to spark some opinions and conversation.

0001: Prepares for pytest support, by adding Meson summary lines
indicating when the Perl TAP tests are enabled.

Peter E mentioned to me that it was hard to figure out which suites
had been enabled, so hopefully this helps a bit. I'm not sure what to
do on the Autoconf side.

0002: Adds Meson and Autoconf support for running pytest, via
--enable-pytest or -Dpytest=enabled.

This is a skeleton. There is no Postgres-specific test logic in this
patch. So if you like pytest, but hate how I write tests in it, you
have the option of basing your alternative on top of this patch to
show us. I also added a failing test so that you can all see what that
looks like in the CI. (I'll put this in the Draft CF [3].)

When enabling the feature, the check_pytest.py script checks that the
configured `PYTHON` executable has all of pytest-requirements.txt
installed. Peter pointed out that this is incorrect: what we actually
want to check is that the interpreter used by pytest has all of the
required packages, and the two could be different.

One way to solve that problem is to just get rid of the check script,
and assume that if pytest is installed, we're good to go. But I ran
into enough installation weirdness in the CI (especially for MinGW,
ugh) that it was really helpful to have a script say "no, you're still
missing this." I'd like to help buildfarm operators in a similar way.
Unfortunately, check_pytest.py is pretty complex, and it doesn't make
as much sense to pay that cost unless and until we add more package
dependencies, as 0003 does later in the set. Thoughts?

For Autoconf, I'm just running pytest directly rather than translating
its output through pgtap + prove, mostly because I think that's a
better dev experience. It has color and everything. Is that okay?

0003: Ports a tiny subset of the SSL client tests.

This is enough to show off
- pytest fixtures and caching via scopes
- FFI binding to libpq via ctypes
- generation of TLS certs via the py-cryptography package
- low-level protocol testing without a "real" server

Andres requested a while back that we control concurrency via Meson
rather than pytest's global discovery. So there is no use of
pytest-xdist here. (I don't think we're in much danger of having CPUs
go idle during a `meson test` run, anyway.)

0004: Ports a tiny subset of the SSL server tests.

This additionally shows off
- more complicated test parameterization
- my vision of how we can cache and configure a running server
instance via pytest fixtures

Basically, server configuration changes are pushed and popped off of a
stack as fixtures are entered and exited. (C++ folks will be
completely unimpressed, but I'm still proud of it.) We'd need to
discuss the proper use of scopes and establish project-wide fixture
conventions if we go in this direction, but I hope we do something
like it; I really want to be able to run each and every test in
isolation.

There are some Win32 syscall examples as well, for those who are
interested, but there's nothing that uses that code at the moment.

0005: Adds a new variable to Cirrus to control the default Meson suites.

0006: Sets that new variable to just the pytest suites, so that I
don't burn too much Cirrus time for this proof of concept, and turns
on all OSes.

= Other General Thoughts =

You might immediately complain that we don't want to open-code every
single byte in the protocol for a test, as I have done here. I don't
want to, either; I want to use Construct [4] for that. I also want us
to have end-to-end tests that combine the fixtures in 0003 and 0004 --
but I want everyone to be able to choose between end-to-end tests and
mocked client/server tests, and I'm most excited about the latter, so
that's what I focused on here.

The distinction between the fixture scopes "session" and "module" is
useful in the Autoconf case (which runs all tests in a pyt/ folder at
once) and useless in the Meson case (which runs each pyt/*.py test in
an individual pytest instance). I suspect this will cause some
friction, but I don't know how much. I haven't been very careful in my
use of "module" scope yet.

I think devs should be able to run pytest directly from the source
root. So I've put settings specific to Autoconf/Meson into
Makefile.global/meson.build instead of pytest.ini. (For instance, I
don't imagine anyone wants to read the pgtap output directly.)

Formatting of the patchset has been provided by isort + black. I hear
ruff is the new kid on the block, but I don't see raw speed being
incredibly important for us. And for this particular use case, I
really like the idea of an "unconfigurable" PEP8 formatter so we can
do other things with our time.

Huge thanks to Andres and Bilal for helping me debug Windows problems in the CI.

Wow, thanks for reading this far. WDYT?

--Jacob

[1] https://wiki.postgresql.org/wiki/FOSDEM/PGDay_2025_Developer_Meeting#Additional_toolchains_in_the_test_suite
[2] https://wiki.postgresql.org/wiki/PGConf.dev_2025_Developer_Meeting
[3] https://commitfest.postgresql.org/54/
[4] https://construct.readthedocs.io/en/latest/