Thread: Proposal for implementing OCSP Stapling in PostgreSQL

Hello PostgreSQL Hackers,

This proposal suggests implementing OCSP Stapling in PostgreSQL as an 
alternative and more efficient method for checking certificate 
revocation, aligning with the trend shift from Certificate Revocation 
Lists (CRL).


1. benefits
OCSP Stapling offers several advantages over traditional CRL checks, 
including:

*) enhances user trust and real-time certificate verification without 
relying on potentially outdated CRLs.
*) helps address privacy concerns associated with traditional OCSP 
checks, where the client contacts the OCSP responder directly.
*) reduces latency by eliminating the need for the client to perform an 
additional round-trip to the OCSP responder.
*) efficient resource utilization by allowing the server to cache and 
reuse OCSP responses.



2. a POC patch with below changes:
*) a new configuration option 'ssl_ocsp_file' to enable/disable OCSP 
Stapling and specify OCSP responses for PostgreSQL servers. For 
instance, ssl_ocsp_file = '_server.resp'

*) a server-side callback function responsible for generating OCSP 
stapling responses. This function comes into play only when a client 
requests the server's certificate status during the SSL/TLS handshake.

*) a new connection parameter 'ssl_ocsp_stapling' on the client side. 
For example, when 'ssl_ocsp_stapling=1', the psql client will send a 
certificate status request to the PostgreSQL server.

*) a client-side callback function within the libpq interface to 
validate and check the stapled OCSP response received from the server. 
If the server's certificate status is valid, the TLS handshake 
continues; otherwise, the connection is rejected.



3.  test cases for 'make check' are not yet ready as they could be 
complicated, but basic tests can be performed as demonstrated below:
To run the tests, OpenSSL tools are required to simulate the OCSP 
responder for generating OCSP responses. Additionally, initial 
certificate generation, including a self-signed root CA, OCSP response 
signing certificate, and PostgreSQL server certificate, is needed.

*) add ocsp atrributes to openssl.cnf
$ openssl version
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

$ diff openssl-ocsp.cnf /etc/ssl/openssl.cnf
204d203
< authorityInfoAccess = OCSP;URI:http://127.0.0.1:6655
232,235d230
< [ v3_ocsp ]
< basicConstraints = CA:FALSE
< keyUsage = nonRepudiation, digitalSignature, keyEncipherment
< extendedKeyUsage = OCSPSigning
255c250
< keyUsage = critical, cRLSign, digitalSignature, keyCertSign
---
 >


*) prepare OCSP responder for generating OCSP response
$ mkdir -p demoCA/newcerts
$ touch demoCA/index.txt
$ echo '01' > demoCA/serial

# create a self-signed root CA
$ openssl req -new -nodes -out rootCA.csr -keyout rootCA.key -subj 
"/C=CA/ST=BC/L=VAN/O=IDO/OU=DEV/CN=rootCA"
$ openssl x509 -req -in rootCA.csr -days 3650 -extfile openssl-ocsp.cnf 
-extensions v3_ca -signkey rootCA.key -out rootCA.crt

# create a certificate for OCSP responder
$ openssl req -new -nodes -out ocspSigning.csr -keyout ocspSigning.key 
-subj "/C=CA/ST=BC/L=VAN/O=IDO/OU=DEV/CN=ocspSigner"
$ openssl ca -keyfile rootCA.key -cert rootCA.crt -in ocspSigning.csr 
-out ocspSigning.crt -config openssl-ocsp.cnf -extensions v3_ocsp
     Sign the certificate? [y/n]:y
     1 out of 1 certificate requests certified, commit? [y/n]y

# create a certificate for PostgreSQL server
$ openssl req -new -nodes -out server.csr -keyout server.key -subj 
"/C=CA/ST=BC/L=VAN/O=IDO/OU=DEV/CN=server"
$ openssl ca -batch -days 365 -keyfile rootCA.key -cert rootCA.crt 
-config openssl-ocsp.cnf -out server.crt -infiles server.csr


# start OCSP responder
$ openssl ocsp -index demoCA/index.txt -port 6655 -rsigner 
ocspSigning.crt -rkey ocspSigning.key -CA rootCA.crt -text

# make sure PostgreSQL server's certificate is 'good'
$ openssl ocsp -issuer rootCA.crt -url http://127.0.0.1:6655 -resp_text 
-noverify -cert server.crt

# generate OCSP response when certificate status is 'good' and save as 
_server.resp-good:
$ openssl ocsp -issuer rootCA.crt -cert server.crt -url 
http://127.0.0.1:6655 -respout _server.resp-good


# revoke PostgreSQL server's certificate
$ openssl ca -keyfile rootCA.key -cert rootCA.crt -revoke server.crt

# make sure PostgreSQL server's certificate is 'revoked'
$ openssl ocsp -issuer rootCA.crt -url http://127.0.0.1:6655 -resp_text 
-noverify -cert server.crt

### generate OCSP response when certificate status is 'revoked' and save 
as _server.resp-revoked:
$ openssl ocsp -issuer rootCA.crt -cert server.crt -url 
http://127.0.0.1:6655 -respout _server.resp-revoked


*) setup OCSP stapling on PostgreSQL server side
copy 'rootCA.crt, server.key, server.crt, _server.resp-good, and 
_server.resp-revoked' to pgdata folder and update PostgreSQL server 
configuration by specifying ssl_ocsp_file = '_server.resp', where 
'_server.resp' is either a copy of '_server.resp-good' or 
'_server.resp-revoked' depending on the test case, for example:

listen_addresses = '*'
ssl = on
ssl_ca_file = 'rootCA.crt'
ssl_cert_file = 'server.crt'
ssl_key_file = 'server.key'
ssl_ocsp_file = '_server.resp'



*) test with psql client
3.1) PostgreSQL server's certificate status is 'good'
$ cp -pr _server.resp-good _server.resp
$ psql -d "sslmode=verify-ca sslrootcert=rootCA.crt user=david 
dbname=postgres ssl_ocsp_stapling=1" -h 127.0.0.1 -p 5432
psql (17devel)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, 
compression: off)
Type "help" for help.

postgres=#


3.2) PostgreSQL server's certificate status is 'revoked'
$ cp -pr _server.resp-revoked _server.resp
$ psql -d "sslmode=verify-ca sslrootcert=rootCA.crt user=david 
dbname=postgres ssl_ocsp_stapling=1" -h 127.0.0.1 -p 5432
psql: error: connection to server at "127.0.0.1", port 5432 failed: SSL 
error: ocsp callback failure


3.3) PostgreSQL server's certificate status is 'revoked' but OCSP 
stapling is not required by psql client:
$ psql -d "sslmode=verify-ca sslrootcert=rootCA.crt user=david 
dbname=postgres ssl_ocsp_stapling=0" -h 127.0.0.1 -p 5432
psql (17devel)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, 
compression: off)
Type "help" for help.

postgres=#


This is a highly experimental proof of concept, and any comments or 
feedback would be greatly appreciated!


Best regards,
David Zhang

===============
Highgo Software Canada
www.highgo.ca

Hi Hackers,

This is the 2nd version patch with following updates:

1) Changed the frontend SSL parameter from `ssl_ocsp_stapling` to 
`sslocspstapling` to align with other SSL parameters.

2) Documented both the backend parameter `ssl_ocsp_file` and the 
frontend parameter `sslocspstapling`.

3) Implemented a check for the `nextUpdate` field in the OCSP response. 
If it is present but expired, the TLS connection will fail."

To add the test cases for OCSP Stapling, I think I should 1) add one 
section to conf/cas.config to generate `ocsp_ca.crt`; 2) use this 
`ocsp_ca` to sign some OCSP responses for server side certificates with 
`good`, `revoked` and `unknown`, and then 3) add some test cases to 
t/001_ssltests.pl.

Any comments or feedback would be greatly appreciated!

Thank you,

David


On 2024-02-05 3:51 p.m., David Zhang wrote:
> Hello PostgreSQL Hackers,
>
> This proposal suggests implementing OCSP Stapling in PostgreSQL as an 
> alternative and more efficient method for checking certificate 
> revocation, aligning with the trend shift from Certificate Revocation 
> Lists (CRL).
>
>
> 1. benefits
> OCSP Stapling offers several advantages over traditional CRL checks, 
> including:
>
> *) enhances user trust and real-time certificate verification without 
> relying on potentially outdated CRLs.
> *) helps address privacy concerns associated with traditional OCSP 
> checks, where the client contacts the OCSP responder directly.
> *) reduces latency by eliminating the need for the client to perform 
> an additional round-trip to the OCSP responder.
> *) efficient resource utilization by allowing the server to cache and 
> reuse OCSP responses.
>
>
>
> 2. a POC patch with below changes:
> *) a new configuration option 'ssl_ocsp_file' to enable/disable OCSP 
> Stapling and specify OCSP responses for PostgreSQL servers. For 
> instance, ssl_ocsp_file = '_server.resp'
>
> *) a server-side callback function responsible for generating OCSP 
> stapling responses. This function comes into play only when a client 
> requests the server's certificate status during the SSL/TLS handshake.
>
> *) a new connection parameter 'ssl_ocsp_stapling' on the client side. 
> For example, when 'ssl_ocsp_stapling=1', the psql client will send a 
> certificate status request to the PostgreSQL server.
>
> *) a client-side callback function within the libpq interface to 
> validate and check the stapled OCSP response received from the server. 
> If the server's certificate status is valid, the TLS handshake 
> continues; otherwise, the connection is rejected.
>
>
>
> 3.  test cases for 'make check' are not yet ready as they could be 
> complicated, but basic tests can be performed as demonstrated below:
> To run the tests, OpenSSL tools are required to simulate the OCSP 
> responder for generating OCSP responses. Additionally, initial 
> certificate generation, including a self-signed root CA, OCSP response 
> signing certificate, and PostgreSQL server certificate, is needed.
>
> *) add ocsp atrributes to openssl.cnf
> $ openssl version
> OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
>
> $ diff openssl-ocsp.cnf /etc/ssl/openssl.cnf
> 204d203
> < authorityInfoAccess = OCSP;URI:http://127.0.0.1:6655
> 232,235d230
> < [ v3_ocsp ]
> < basicConstraints = CA:FALSE
> < keyUsage = nonRepudiation, digitalSignature, keyEncipherment
> < extendedKeyUsage = OCSPSigning
> 255c250
> < keyUsage = critical, cRLSign, digitalSignature, keyCertSign
> ---
> >
>
>
> *) prepare OCSP responder for generating OCSP response
> $ mkdir -p demoCA/newcerts
> $ touch demoCA/index.txt
> $ echo '01' > demoCA/serial
>
> # create a self-signed root CA
> $ openssl req -new -nodes -out rootCA.csr -keyout rootCA.key -subj 
> "/C=CA/ST=BC/L=VAN/O=IDO/OU=DEV/CN=rootCA"
> $ openssl x509 -req -in rootCA.csr -days 3650 -extfile 
> openssl-ocsp.cnf -extensions v3_ca -signkey rootCA.key -out rootCA.crt
>
> # create a certificate for OCSP responder
> $ openssl req -new -nodes -out ocspSigning.csr -keyout ocspSigning.key 
> -subj "/C=CA/ST=BC/L=VAN/O=IDO/OU=DEV/CN=ocspSigner"
> $ openssl ca -keyfile rootCA.key -cert rootCA.crt -in ocspSigning.csr 
> -out ocspSigning.crt -config openssl-ocsp.cnf -extensions v3_ocsp
>     Sign the certificate? [y/n]:y
>     1 out of 1 certificate requests certified, commit? [y/n]y
>
> # create a certificate for PostgreSQL server
> $ openssl req -new -nodes -out server.csr -keyout server.key -subj 
> "/C=CA/ST=BC/L=VAN/O=IDO/OU=DEV/CN=server"
> $ openssl ca -batch -days 365 -keyfile rootCA.key -cert rootCA.crt 
> -config openssl-ocsp.cnf -out server.crt -infiles server.csr
>
>
> # start OCSP responder
> $ openssl ocsp -index demoCA/index.txt -port 6655 -rsigner 
> ocspSigning.crt -rkey ocspSigning.key -CA rootCA.crt -text
>
> # make sure PostgreSQL server's certificate is 'good'
> $ openssl ocsp -issuer rootCA.crt -url http://127.0.0.1:6655 
> -resp_text -noverify -cert server.crt
>
> # generate OCSP response when certificate status is 'good' and save as 
> _server.resp-good:
> $ openssl ocsp -issuer rootCA.crt -cert server.crt -url 
> http://127.0.0.1:6655 -respout _server.resp-good
>
>
> # revoke PostgreSQL server's certificate
> $ openssl ca -keyfile rootCA.key -cert rootCA.crt -revoke server.crt
>
> # make sure PostgreSQL server's certificate is 'revoked'
> $ openssl ocsp -issuer rootCA.crt -url http://127.0.0.1:6655 
> -resp_text -noverify -cert server.crt
>
> ### generate OCSP response when certificate status is 'revoked' and 
> save as _server.resp-revoked:
> $ openssl ocsp -issuer rootCA.crt -cert server.crt -url 
> http://127.0.0.1:6655 -respout _server.resp-revoked
>
>
> *) setup OCSP stapling on PostgreSQL server side
> copy 'rootCA.crt, server.key, server.crt, _server.resp-good, and 
> _server.resp-revoked' to pgdata folder and update PostgreSQL server 
> configuration by specifying ssl_ocsp_file = '_server.resp', where 
> '_server.resp' is either a copy of '_server.resp-good' or 
> '_server.resp-revoked' depending on the test case, for example:
>
> listen_addresses = '*'
> ssl = on
> ssl_ca_file = 'rootCA.crt'
> ssl_cert_file = 'server.crt'
> ssl_key_file = 'server.key'
> ssl_ocsp_file = '_server.resp'
>
>
>
> *) test with psql client
> 3.1) PostgreSQL server's certificate status is 'good'
> $ cp -pr _server.resp-good _server.resp
> $ psql -d "sslmode=verify-ca sslrootcert=rootCA.crt user=david 
> dbname=postgres ssl_ocsp_stapling=1" -h 127.0.0.1 -p 5432
> psql (17devel)
> SSL connection (protocol: TLSv1.2, cipher: 
> ECDHE-RSA-AES256-GCM-SHA384, compression: off)
> Type "help" for help.
>
> postgres=#
>
>
> 3.2) PostgreSQL server's certificate status is 'revoked'
> $ cp -pr _server.resp-revoked _server.resp
> $ psql -d "sslmode=verify-ca sslrootcert=rootCA.crt user=david 
> dbname=postgres ssl_ocsp_stapling=1" -h 127.0.0.1 -p 5432
> psql: error: connection to server at "127.0.0.1", port 5432 failed: 
> SSL error: ocsp callback failure
>
>
> 3.3) PostgreSQL server's certificate status is 'revoked' but OCSP 
> stapling is not required by psql client:
> $ psql -d "sslmode=verify-ca sslrootcert=rootCA.crt user=david 
> dbname=postgres ssl_ocsp_stapling=0" -h 127.0.0.1 -p 5432
> psql (17devel)
> SSL connection (protocol: TLSv1.2, cipher: 
> ECDHE-RSA-AES256-GCM-SHA384, compression: off)
> Type "help" for help.
>
> postgres=#
>
>
> This is a highly experimental proof of concept, and any comments or 
> feedback would be greatly appreciated!
>
>
> Best regards,
> David Zhang
>
> ===============
> Highgo Software Canada
> www.highgo.ca
>

Hi Hackers,

This is the third version patch for "Certificate status check using OCSP 
Stapling" with ssl regression test cases added.

Here is how I run the ssl regression test:
     ./configure --enable-tap-tests --with-openssl
     make -j
     cd src/test/ssl
     make sslfiles
     make check PG_TEST_EXTRA=ssl

expected results:
     # +++ tap check in src/test/ssl +++
     t/001_ssltests.pl .. ok
     t/002_scram.pl ..... ok
     t/003_sslinfo.pl ... ok
     All tests successful.
     Files=3, Tests=279, 17 wallclock secs ( 0.05 usr  0.01 sys + 2.32 
cusr  2.16 csys =  4.54 CPU)

     Result: PASS

Notes, before executing the SSL regression tests with the command `make 
check PG_TEST_EXTRA=ssl`, it is necessary to wait for 1 minute after 
running `make sslfiles`. This delay is required because the newly 
generated OCSP responses for the 'expired' test cases need 1 minute to 
pass the nextUpdate period. Once the stapled OCSP response files for the 
tests are committed as test input, there is no need to wait, similar to 
certificate files.

Any comments or feedback would be greatly appreciated!

Thank you,

David

On Tue, Mar 5, 2024 at 4:12 PM David Zhang <david.zhang@highgo.ca> wrote:
> Any comments or feedback would be greatly appreciated!

Hi David -- I haven't had time to get to this for the 17 release
cycle, but I'm interested in this feature and I intend to review it at
some point for 18. I think OCSP will be especially helpful for anyone
relying on sslrootcert=system. Thanks for working on it!

--Jacob

On Tue, Mar 5, 2024 at 4:12 PM David Zhang <david.zhang@highgo.ca> wrote:
> This is the third version patch for "Certificate status check using OCSP
> Stapling" with ssl regression test cases added.

Hi David,

Thanks again for working on this! So far I've taken a look at the
design and tests. I've only skimmed the callback implementations; I
plan to review those in more detail once the architecture is nailed
down.

= Design =

It looks like this design relies on the DBA to manually prefetch OCSP
responses for their cert chain, and cache them in the local
ssl_ocsp_file. This is similar to Nginx's ssl_stapling_file directive
[1]. I think this may make sense for a v1 (much less code!), but it's
going to take a lot of good documentation on what, exactly, an admin
has to do to make sure their response file is fresh. IIUC it will
depend on the CA's policy and how they operate their responder.

We should probably be prepared for complaints that we don't run the
client ourselves. A v2 could maybe include an extension for running
the OCSP client in a background worker? Or maybe that's overkill, and
an existing job scheduler extension could do it, but having a custom
worker that could periodically check the response file and provide
warnings when it gets close to expiring might be helpful for admins.

I am worried about the multi-stapling that is being done in the tests,
for example the server-ca-ocsp-good response file. I think those tests
are cheating a bit. Yes, for this particular case, we can issue a
single signed response for both the intermediate and the leaf -- but
that's only because we issued both of them. What if your intermediate
and your root use different responders [2]? What if your issuer's
responder doesn't even support multiple certs per request [3]?

(Note that OpenSSL still doesn't support multi-stapling [4], even in
TLS 1.3 where we were supposed to get it "for free".)

I think it would be good for the sslocspstapling directive to 1) maybe
have a shorter name (cue bikeshed!) and 2) support more than a binary
on/off. For example, the current implementation could use
"disable/require" options, and then a v2 could add "prefer" which
simply sends the status request and honors must-staple extensions on
the certificate. (That should be safe to default to, I think, and it
would let DBAs control the stapling rollout more easily.)

A question from ignorance: how does the client decide that the
signature on the OCSP response is actually valid for the specific
chain in use?

= Tests =

I think the tests should record the expected_stderr messages for
failures (see the Code section below for why).

If it turns out that multi-stapling is safe, then IMO the tests should
explicitly test both TLSv1.2 and v1.3, since those protocols differ in
how they handle per-certificate status.

If it's okay with you, I'd like to volunteer to refactor out the
duplicated recipes in sslfiles.mk. I have some particular ideas in
mind and don't want to make you play fetch-a-rock. (No pressure to use
what I come up with, if you don't like it.)

Because a CRL is a valid fallback for OCSP in practice, I think there
should be some tests for their interaction. (For v1, maybe that's as
simple as "you're not allowed to use both yet", but it should be
explicit.)

= Code =

(this is a very shallow review)

+#define OCSP_CERT_STATUS_OK    1
+#define OCSP_CERT_STATUS_NOK   (-1)

Returning f -1 from the callback indicates an internal error, so we're
currently sending the wrong alerts for OCSP failures ("internal error"
rather than "bad certificate status response") and getting unhelpful
error messages from libpq. For cases where the handshake proceeds
correctly but we don't accept the OCSP response status, I think we
should be returning zero.

Also, we should not stomp on the OCSP_ namespace (I thought these
macros were part of the official <openssl/ocsp.h> API at first).

+   /* set up OCSP stapling callback */
+   SSL_CTX_set_tlsext_status_cb(SSL_context, ocsp_stapling_cb);

It seems like this should only be done if ssl_ocsp_file is set?

Thanks again!
--Jacob

[1] https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_stapling_file
[2] as appears to be the case for postgresql.org
[3] https://community.letsencrypt.org/t/bulk-ocsp-requests/168156/2
[4] https://github.com/openssl/openssl/pull/20945

> = Design =
>
> It looks like this design relies on the DBA to manually prefetch OCSP
> responses for their cert chain, and cache them in the local
> ssl_ocsp_file. This is similar to Nginx's ssl_stapling_file directive
> [1]. I think this may make sense for a v1 (much less code!), but it's
> going to take a lot of good documentation on what, exactly, an admin
> has to do to make sure their response file is fresh. IIUC it will
> depend on the CA's policy and how they operate their responder.
>
> We should probably be prepared for complaints that we don't run the
> client ourselves. A v2 could maybe include an extension for running
> the OCSP client in a background worker? Or maybe that's overkill, and
> an existing job scheduler extension could do it, but having a custom
> worker that could periodically check the response file and provide
> warnings when it gets close to expiring might be helpful for admins.
Totally agree. Either Implementing OCSP requests over HTTP, then parsing 
the response and then saving the results to a file, or using an OpenSSL 
client with a cron job to periodically update the file should work. 
Using a cron job would likely have less impact on PostgreSQL.
> I am worried about the multi-stapling that is being done in the tests,
> for example the server-ca-ocsp-good response file. I think those tests
> are cheating a bit. Yes, for this particular case, we can issue a
> single signed response for both the intermediate and the leaf -- but
> that's only because we issued both of them. What if your intermediate
> and your root use different responders [2]? What if your issuer's
> responder doesn't even support multiple certs per request [3]?
>
> (Note that OpenSSL still doesn't support multi-stapling [4], even in
> TLS 1.3 where we were supposed to get it "for free".)

Totally agree, then we should limit OCSP stapling check for the 
leaf/PostgreSQL server certificate only in v1.

> I think it would be good for the sslocspstapling directive to 1) maybe
> have a shorter name (cue bikeshed!) and 2) support more than a binary
> on/off. For example, the current implementation could use
> "disable/require" options, and then a v2 could add "prefer" which
> simply sends the status request and honors must-staple extensions on
> the certificate. (That should be safe to default to, I think, and it
> would let DBAs control the stapling rollout more easily.)

This will definitely give end users more options, especially during the 
transition period.

> A question from ignorance: how does the client decide that the
> signature on the OCSP response is actually valid for the specific
> chain in use?

If I understand correctly, the certificate used by the OCSP responder to 
sign the OCSP response must be valid for the specific chain in use, or 
the admins allow to load a new chain to validate the certificate used to 
sign the OCSP response. I think it would be better to make this part to 
be more open.

>
> = Tests =
>
> I think the tests should record the expected_stderr messages for
> failures (see the Code section below for why).
+1
> If it turns out that multi-stapling is safe, then IMO the tests should
> explicitly test both TLSv1.2 and v1.3, since those protocols differ in
> how they handle per-certificate status.
>
> If it's okay with you, I'd like to volunteer to refactor out the
> duplicated recipes in sslfiles.mk. I have some particular ideas in
> mind and don't want to make you play fetch-a-rock. (No pressure to use
> what I come up with, if you don't like it.)
That would be great, thanks a lot in advance!
> Because a CRL is a valid fallback for OCSP in practice, I think there
> should be some tests for their interaction. (For v1, maybe that's as
> simple as "you're not allowed to use both yet", but it should be
> explicit.)
+1
> = Code =
>
> (this is a very shallow review)
>
> +#define OCSP_CERT_STATUS_OK    1
> +#define OCSP_CERT_STATUS_NOK   (-1)
>
> Returning f -1 from the callback indicates an internal error, so we're
> currently sending the wrong alerts for OCSP failures ("internal error"
> rather than "bad certificate status response") and getting unhelpful
> error messages from libpq. For cases where the handshake proceeds
> correctly but we don't accept the OCSP response status, I think we
> should be returning zero.
+1
> Also, we should not stomp on the OCSP_ namespace (I thought these
> macros were part of the official <openssl/ocsp.h> API at first).
+1
> +   /* set up OCSP stapling callback */
> +   SSL_CTX_set_tlsext_status_cb(SSL_context, ocsp_stapling_cb);
>
> It seems like this should only be done if ssl_ocsp_file is set?

+1

Thanks a lot for reviewing and providing all your feedback!


Best regards,

David Zhang

On Wed, Jul 17, 2024 at 3:42 PM David Zhang <idrawone@gmail.com> wrote:
> Totally agree. Either Implementing OCSP requests over HTTP, then parsing
> the response and then saving the results to a file, or using an OpenSSL
> client with a cron job to periodically update the file should work.
> Using a cron job would likely have less impact on PostgreSQL.

Yeah, my preference would be to farm this job out to OpenSSL entirely.
Implementing OCSP-over-HTTP ourselves seems unlikely to buy us much,
for all the work it would give us.

> Totally agree, then we should limit OCSP stapling check for the
> leaf/PostgreSQL server certificate only in v1.

Sounds good. Maybe a future version can implement a check of the full
chain, but I imagine we'll have to follow someone else's lead.

> > A question from ignorance: how does the client decide that the
> > signature on the OCSP response is actually valid for the specific
> > chain in use?
>
> If I understand correctly, the certificate used by the OCSP responder to
> sign the OCSP response must be valid for the specific chain in use, or
> the admins allow to load a new chain to validate the certificate used to
> sign the OCSP response. I think it would be better to make this part to
> be more open.

Okay. That part needs more design work IMO, and solid testing.

> > If it's okay with you, I'd like to volunteer to refactor out the
> > duplicated recipes in sslfiles.mk. I have some particular ideas in
> > mind and don't want to make you play fetch-a-rock. (No pressure to use
> > what I come up with, if you don't like it.)
> That would be great, thanks a lot in advance!

No problem! I've attached two patches that can be optionally applied
on top of yours:
- 0001 simplifies the response generation into a single recipe.
- 0002 is a bigger change that uses `openssl ca` to generate index
files, as opposed to constructing them manually ourselves.

(The makefile will get even smaller without multi-stapling support,
but I didn't want to combine that with the refactor.)

For 0002, I'm wiping the new CA index for each recipe and rebuilding
it from scratch, then copying it into place (this relies on the
.NOTPARALLEL setting up top for correctness). I think there's probably
an even simpler approach, which would be to generate a single index
that can produce all of our desired responses. I can give that a try
once multi-stapling support is pulled out.

> Thanks a lot for reviewing and providing all your feedback!

You're very welcome, thanks for working on this feature!

--Jacob

Thanks a lot Jacob for helping update the tests and sorry for the late reply.

Based on previous discussion, I remove the document patch, and start to focus on the v1 simple OCSP logic by checking the leaf/Postgres server certificate's status only (0001-v1-WIP-OCSP-support-certificate-status-check.patch). I also merge your changes and simplify the test by testing the Postgres server certificate's status only (0002-v1-WIP-OCSP-simplify-.res-generation-and-regress-tes.patch).

A question from ignorance: how does the client decide that the
signature on the OCSP response is actually valid for the specific
chain in use?

If I understand correctly, the certificate used by the OCSP responder to
sign the OCSP response must be valid for the specific chain in use, or
the admins allow to load a new chain to validate the certificate used to
sign the OCSP response. I think it would be better to make this part to
be more open.

Okay. That part needs more design work IMO, and solid testing.

Based on the RFC here, https://datatracker.ietf.org/doc/html/rfc6960#section-4.2.2.2. My understanding is that the OCSP responder's certificate should be directly signed by the same CA which signed the Postgres Server's certificate. It looks the openssl 3.0.14 implements in this way. In other words, it the OCSP responder's certificate is signed by a different branch of the chain, then openssl will report some errors. Unless the end user explicitly provides the OCSP responder's certificate with trust_other option. In this case it will skip the some certificate verification. I think it should be simple enough for v1 by set `OCSP_basic_verify(basic_resp, NULL, trusted_store, 0)`.  The key function OCSP_basic_verify is documented at here, https://docs.openssl.org/3.0/man3/OCSP_resp_find_status/



Thank you,
David

While I have only skimmed the patch so far and need more review before I can
comment on it, I do have a question on the expected use of OCSP support in
postgres.  With OCSP becoming optional [0], and big providers like Let's
Encrypt deprecating OCSP [1], is this mainly targeting organizations running
their own CA with in-house OCSP?

--
Daniel Gustafsson

[0] https://lists.cabforum.org/pipermail/servercert-wg/2023-September/003998.html
[1] https://letsencrypt.org/2024/07/23/replacing-ocsp-with-crls.html

Hi Daniel,

Thank you for all the information.

On 2024-08-07 12:20 a.m., Daniel Gustafsson wrote:
> While I have only skimmed the patch so far and need more review before I can
> comment on it, I do have a question on the expected use of OCSP support in
> postgres.  With OCSP becoming optional [0], and big providers like Let's
> Encrypt deprecating OCSP [1], is this mainly targeting organizations running
> their own CA with in-house OCSP?
>
> --
> Daniel Gustafsson
>
> [0] https://lists.cabforum.org/pipermail/servercert-wg/2023-September/003998.html
> [1] https://letsencrypt.org/2024/07/23/replacing-ocsp-with-crls.html
Regarding the privacy concert, the OCSP Stapling works in a very similar 
way as CRL but provide a more "real-time" certificate status checking. 
When the Client/psql needs to check the Certificate Status using OCSP 
Stapling, it doesn't connect to any 3rd party server, such as CA or OCSP 
Responder. It only requires the Server/Postgres to provide one extra 
piece of information about the status of Server's certificate. OCSP 
Stapling was designed to avoid the privacy concern and a single point of 
failure issues.

When the Client/psql needs to check the certificate revocation status:
option 1 using CRL: CA generates the CRL and then upload it to somewhere 
or distribute it to each Client/psql. I think Postgres only support the 
preloaded CRL.
option 2 using OCSP Stapling: Postgres server retrieves the certificate 
status periodically from OCSP responder, the Client/psql will do the 
certificate status check during each handshake.

I think it is still necessary to provide the 2nd option/flexibility to 
end users if they are allowed to check the Postgres server's certificate 
revocation status through CRL.

Thank you,
David

On Wed, Aug 7, 2024 at 12:20 AM Daniel Gustafsson <daniel@yesql.se> wrote:
>
> While I have only skimmed the patch so far and need more review before I can
> comment on it, I do have a question on the expected use of OCSP support in
> postgres.  With OCSP becoming optional [0], and big providers like Let's
> Encrypt deprecating OCSP [1], is this mainly targeting organizations running
> their own CA with in-house OCSP?

That announcement took me by surprise (and, it looks like, a number of
other people [1, 2]). I get that OCSP is expensive and painful for
Let's Encrypt, based on previous outages and blog posts, but I also
figured that Must-Staple was basically the best you could do without
being a browser. It already seemed pretty clear that we shouldn't
build a client-side OCSP check. Throwing server-side stapling under
the bus with it was unexpected.

Some of the LE quotes on the matter are giving me cart-before-horse vibes:

> But it is clear to me OCSP is an ineffective technical dead-end, and we are all better served by moving on to figure
outwhat else we can do. 
>
> We may keep OCSP running for some time for certificates that have the must-staple extension, to help smooth the
transition,but at this time we don’t have a plan for how to actually deprecate OCSP: just an intent, publicized to
ensurewe can all begin to plan for a future without it. 

It's pretty frustrating to hear about a "transition" when there is
nothing to transition to.

I honestly wonder if they're going to end up walking some of this
back. The messaging reminds me of "that one project" that every
company seems to have, where it's expensive and buggy as heck, all the
maintainers want to see it deleted, and they unilaterally declare over
clients' objections that they will, only to find at the last second
that the cure is worse than the disease and then finally resign
themselves to supporting it. Tears are shed, bridges burned.

Anyways, I look forward to seeing how broken my crystal ball is this
time. The timing is awful for this patchset in particular.

--Jacob

[1] https://community.letsencrypt.org/t/sunsetting-of-ocsp-in-favor-of-older-technology/222589
[2] https://community.letsencrypt.org/t/what-will-happen-to-must-staple/222397

> On 15 Aug 2024, at 00:42, Jacob Champion <jacob.champion@enterprisedb.com> wrote:

> It's pretty frustrating to hear about a "transition" when there is
> nothing to transition to.

I guess they prefer that orgs transition back to just using CRL's.

> Anyways, I look forward to seeing how broken my crystal ball is this
> time. The timing is awful for this patchset in particular.

It is indeed, if it ends up deprecated server-side among the big providers then
support for it risks being very hard to use.  Not sure what is the best course
of action here is.

--
Daniel Gustafsson

On Mon, Sep 2, 2024 at 5:55 AM Daniel Gustafsson <daniel@yesql.se> wrote:
> I guess they prefer that orgs transition back to just using CRL's.

From a practical perspective, I don't think anyone but browsers can do
that right now. Best I can tell, there's no CRLite client other than
Firefox, and Google's CRLSets look like a manual emergency system
rather than a general-purpose tool.

I don't think we could do it manually even if we wanted to (and we
don't want to, IMHO, for a whole host of reasons). As one specific
example, take the certificate for postgresql.org. There's no CRL
distribution point listed, and an LE blog post [1] from a couple years
back implies that they have no plans to make those available to us:

    Although we will be producing CRLs which cover all certificates that we
    issue, we will not be including those URLs in the CRL Distribution Point
    extension of our certificates. [...] Our new CRL URLs will be disclosed
    only in CCADB, so that the Apple and Mozilla root programs can consume
    them without exposing them to potentially large download traffic from
    the rest of the internet at large.

Frankly, it looks like they have no plan for non-browser clients. It's
feeling like one of those "Web" vs. "Internet" splits.

--Jacob

[1] https://letsencrypt.org/2022/09/07/new-life-for-crls.html