Re: Adding CI to our tree - Mailing list pgsql-hackers

Hi,

Attached is an updated version of the CI patches. An example of a test run
with the attached version of this
https://cirrus-ci.com/build/6501998521483264

I again included the commit allowing crash dumps to be collected on windows,
but I don't think it can be merged as-is, and should be left for later.


Changes since v2:
- Address review comments

- Build with further optional features enabled. I think just about everything
  reasonable is now enabled on freebsd, linux and macos. There's quite a bit
  more that could be done on windows, but I think it's good enough for now.

- I added cross-compilation to windows from linux, to the "warnings"
  task. Occasionally there are build-system issues specific to
  cross-compilation, and the set of warnings are different.

- Docs are now built as part of the 'CompilerWarnings' task.

- I improved the CI README a bit more, in particular I added docs for the
  'ci-os-only' tag I added to the CI logic, which lets one select which
  operating systems test get to run on.

- Some of the 'Warnings' tasks now build with --enable-dtrace (once with
  optimizations, once without). It's pretty easy to break probes without
  seeing the problem locally.

- Switched to using PG_TEST_USE_UNIX_SOCKETS for windows. Without that I was
  seeing occasional spurious test failures due to the use of PROVE_FLAGS=
  -j10, to make the otherwise serial execution of tests on windows bearable.

- switch macos task to use monterey

- plenty smaller changes / cleanups


There of course is a lot more that can be done [1], but I am pretty happy with
what this covers.


I'd like to commit this soon. There's two aspects that perhaps deserve a bit
more discussion before doing so though:

One I explicitly brought up before:

On 2021-10-01 15:27:52 -0700, Andres Freund wrote:
> One policy discussion that we'd have to have is who should control the images
> used for CI. Right now that's on my personal google cloud account - which I am
> happy to do, but medium - long term that'd not be optimal.

The proposed CI script uses custom images to run linux and freebsd tests. They
are automatically built every day from the repository https://github.com/anarazel/pg-vm-images/

These images have all the prerequisites pre-installed. For Linux something
similar can be achieved by using dockerfiles referenced the .cirrus.yml file,
but for FreeBSD that's not available. Installing the necessary dependencies on
every run is too time intensive. For linux, the tests run a lot faster in
full-blown VMs than in docker, and full VMs allow a lot more control /
debugging.

I think this may be OK for now, but I also could see arguments for wanting to
transfer the image specifications and the google account to PG properties.


The second attention-worthy point is the choice of a new toplevel ci/
directory as the location for resources referencenced by CI. A few other
projects also use ci/, but I can also see arguments for moving the contents to
e.g. src/tools/ci or such?

Greetings,

Andres Freund


[1] Some ideas for what could make sense to extend CI to in the future:

- also test with msys / mingw on windows

- provide more optional dependencies for windows build

- Extend the set of compiler warnings - as the compiler version is controlled,
  we could be more aggressive than we can be via configure.ac.

- Add further distributions / platforms. Possibly as "manual" tasks - the
  amount resources one CI user gets is limited, so running tests on all
  platforms all the time would make tests take longer. Interesting things
  could be:

  - further linux distributions, particularly long-term supported ones

  - Some of the other BSDs. There currently are no pre-made images for
    openbsd/netbsd, but it shouldn't be too hard to script building them.

  - running some tests on ARM could be interesting, cirrus supports that for
    container based builds now

- run checks like cpluspluscheck as part of the CompilerWarnings task

- add tasks for running tests with tools like asan / ubsan (valgrind will be
  too slow).

- consider enable compile-time debugging options like COPY_PARSE_PLAN_TREES,
  and run-time force_parallel_mode = regress on some platforms. They seem to
  catch a lot of problems during development and are likely affordable enough.