recurring SSL errors with streaming replication in 9.4.1 - Mailing list pgsql-bugs

Hello,

We are getting period FATALs from our recently upgraded to 9.4.1 Hot Standby:

015-02-17 03:01:26.251 CET,,,30247,,54e1bfe0.7627,2,,2015-02-16
11:01:04 CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL
error: unexpected record
",,,,,,,,,""
2015-02-17 03:01:26.280 CET,,,26192,,54dd0fb1.6650,12,,2015-02-12
21:40:17 CET,1/0,0,LOG,00000,"unexpected pageaddr 219/8B000000 in log
segment 00000004000002190000008F, offset 0",,,,,,,,,""
2015-02-17 03:01:26.295 CET,,,56544,,54e2a0f6.dce0,1,,2015-02-17
03:01:26 CET,,0,LOG,00000,"started streaming WAL from primary at
219/8F000000 on timeline 4",,,,,,,,,""

The issues happen roughly couple of times a day, but sometimes even
more frequently:

postgresql-2015-02-16_000000.csv:2015-02-16 11:01:04.093
CET,,,61385,,54de3bb7.efc9,2,,2015-02-13 19:00:23
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL error:
unexpected record
postgresql-2015-02-17_000000.csv:2015-02-17 03:01:26.251
CET,,,30247,,54e1bfe0.7627,2,,2015-02-16 11:01:04
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL error:
unexpected record
postgresql-2015-02-17_000000.csv:2015-02-17 19:01:39.693
CET,,,56544,,54e2a0f6.dce0,2,,2015-02-17 03:01:26
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL error:
unexpected record
postgresql-2015-02-18_000000.csv:2015-02-18 03:01:47.176
CET,,,17898,,54e38203.45ea,2,,2015-02-17 19:01:39
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL error:
unexpected record
postgresql-2015-02-18_000000.csv:2015-02-18 08:21:50.111
CET,,,29858,,54e3f28b.74a2,2,,2015-02-18 03:01:47
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL error:
unexpected record
postgresql-2015-02-18_000000.csv:2015-02-18 11:01:52.193
CET,,,38081,,54e43d8e.94c1,2,,2015-02-18 08:21:50
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL error:
unexpected record

postgresql-2015-02-18_000000.csv:2015-02-18 19:02:04.862
CET,,,43159,,54e46310.a897,2,,2015-02-18 11:01:52
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL SYSCALL
error: Success
postgresql-2015-02-19_000000.csv:2015-02-19 11:02:21.195
CET,,,55104,,54e4d39c.d740,2,,2015-02-18 19:02:04
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL error:
unexpected record

We have another replica of a different cluster also on 9.4.1 running
into the same problem, but the timings are different:

postgresql-2015-02-16_000000.csv:2015-02-16 00:32:13.931
CET,,,3800,,54df0d9a.ed8,2,,2015-02-14 09:55:54
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL error:
unexpected record
postgresql-2015-02-16_000000.csv:2015-02-16 01:13:27.923
CET,,,4053,,54e12c7d.fd5,2,,2015-02-16 00:32:13
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL error:
unexpected record
postgresql-2015-02-16_000000.csv:2015-02-16 01:17:10.406
CET,,,5175,,54e13627.1437,2,,2015-02-16 01:13:27
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL SYSCALL
error: Success
postgresql-2015-02-16_000000.csv:2015-02-16 01:17:21.421
CET,,,5307,,54e13706.14bb,2,,2015-02-16 01:17:10
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL SYSCALL
error: Success
postgresql-2015-02-16_000000.csv:2015-02-16 06:27:24.274
CET,,,5316,,54e13711.14c4,2,,2015-02-16 01:17:21
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL error:
unexpected record
postgresql-2015-02-16_000000.csv:2015-02-16 10:33:22.927
CET,,,14275,,54e17fbc.37c3,2,,2015-02-16 06:27:24
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL error:
unexpected record
postgresql-2015-02-17_000000.csv:2015-02-17 09:37:23.703
CET,,,22809,,54e1b962.5919,2,,2015-02-16 10:33:22
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL error:
unexpected record
postgresql-2015-02-17_000000.csv:2015-02-17 23:10:03.386
CET,,,2287,,54e2fdc3.8ef,2,,2015-02-17 09:37:23
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL error:
unexpected record
postgresql-2015-02-17_000000.csv:2015-02-17 23:26:48.320
CET,,,26373,,54e3bc3b.6705,2,,2015-02-17 23:10:03
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL SYSCALL
error: Success
postgresql-2015-02-18_000000.csv:2015-02-18 08:06:20.050
CET,,,26800,,54e3c028.68b0,2,,2015-02-17 23:26:48
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL error:
unexpected record
postgresql-2015-02-18_000000.csv:2015-02-18 10:58:37.773
CET,,,41501,,54e439ec.a21d,2,,2015-02-18 08:06:20
CET,,0,FATAL,XX000,"could not send data to WAL stream: SSL error:
unexpected record

We also see this problem on other replicas of the clusters  running
with ssl enabled, with 9.3 the particular flavor of this error is:

postgresql-2015-02-17_000000.csv:2015-02-17 05:13:46.953
CET,,,11997,,54e256e9.2edd,2,,2015-02-16 21:45:29
CET,,0,FATAL,XX000,"could not receive data from WAL stream: SSL error:
sslv3 alert unexpected message

ssl_renegotiation_limit is to the default of '512MB'. I will disable
it on one of the clusters and see if the problem is still there, but
meanwhile, is there anything we can provide to help resolve this
issue? I see in the release notes for 9.4  that the SSL renegotiation
code has been improved, but apparently some issues are still there.

Regards,
--
Alexey Klyukin