On Fri, Jul 4, 2025 at 9:49 AM Erik Johnston <erikj@element.io> wrote:
Hi, a quick update:
- We have discovered that the corruption was present from before libicu update. - We ran `pg_amcheck --index state_groups_state_type_idx --heapallindexed matrix`, which returned nothing - We believe that means that (and matches what we see sampling) the index has gained extra entries, i.e. that for a given state group it does return all the relevant rows in the table plus extra rows.
We are also seeing old state groups starting to point at rows that have only just been inserted. For example, querying for 353864583 on the primary it returns that row plus four rows that have been inserted today, but on the backup from last week an index only scan for 353864583 only returns one row. This makes it feel like the corruption is ongoing? Nothing should have modified that state group in the interim (they are generally immutable).
This naively feels like when inserting a new row we sometimes add the row to the index twice: once pointing from the correct state group to the new row, and once from an old state group to the new row?
