From 2898a174bba1895428e010001f6e576ff73e2e39 Mon Sep 17 00:00:00 2001 From: Masahiko Sawada Date: Tue, 19 May 2020 15:06:38 +0900 Subject: [PATCH v36 8/9] Documentation update. Co-authored-by: Masahiko Sawada, Ashutosh Bapat --- doc/src/sgml/catalogs.sgml | 135 ++++++++++++ doc/src/sgml/config.sgml | 144 +++++++++++++ doc/src/sgml/distributed-transaction.sgml | 158 ++++++++++++++ doc/src/sgml/fdwhandler.sgml | 245 ++++++++++++++++++++++ doc/src/sgml/filelist.sgml | 1 + doc/src/sgml/func.sgml | 147 +++++++++++++ doc/src/sgml/monitoring.sgml | 42 ++++ doc/src/sgml/postgres.sgml | 1 + doc/src/sgml/storage.sgml | 6 + src/backend/access/transam/README.fdwxact | 134 ++++++++++++ 10 files changed, 1013 insertions(+) create mode 100644 doc/src/sgml/distributed-transaction.sgml create mode 100644 src/backend/access/transam/README.fdwxact diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 6d06ad22b9..6db7a7ba8c 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -9394,6 +9394,11 @@ SCRAM-SHA-256$<iteration count>:&l summary of configuration file contents + + pg_foreign_xacts + foreign transactions + + pg_group groups of database users @@ -11268,6 +11273,136 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx + + <structname>pg_foreign_xacts</structname> + + + pg_foreign_xacts + + + + The view pg_foreign_xacts displays + information about foreign transactions that are opened on + foreign servers for atomic distributed transaction commit (see + for details). + + + + pg_foreign_xacts contains one row per foreign + transaction. An entry is removed when the foreign transaction is + committed or rolled back. + + + + <structname>pg_foreign_xacts</structname> Columns + + + + + Name + Type + References + Description + + + + + dbid + oid + pg_database.oid + + OID of the database which the foreign transaction resides in + + + + xid + xid + + + Numeric transaction identifier with which this foreign transaction + associates + + + + serverid + oid + pg_foreign_server.oid + + The OID of the foreign server on which the foreign transaction is prepared + + + + userid + oid + pg_user.oid + + The OID of the user that prepared this foreign transaction. + + + + status + text + + + Status of foreign transaction. Possible values are: + + + + preparing : This foreign transaction is being prepared. + + + + + prepared : This foreign transaction has been prepared. + + + + + comitting : This foreign transcation has been + prepared to commit or being committed. + + + + + abortin : This foreign transaction has been + prepared to abort or being aborted. + + + + + + + locker_pid + int + + + Process ID of the locker currently processing. + + + + identifier + text + + + The identifier of the prepared foreign transaction. + + + + +
+ + + When the pg_foreign_xacts view is accessed, the + internal transaction manager data structures are momentarily locked, and + a copy is made for the view to display. This ensures that the + view produces a consistent set of results, while not blocking + normal operations longer than necessary. Nonetheless + there could be some impact on database performance if this view is + frequently accessed. + + + + <structname>pg_publication_tables</structname> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 45bd1f1b7e..431f1fb796 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -9552,6 +9552,150 @@ dynamic_library_path = 'C:\tools\postgresql;H:\my_project\lib;$libdir' + + Distributed Transaction Management + + + Setting + + + + foreign_twophase_commit (enum) + + foreign_twophase_commit configuration parameter + + + + + Specifies whether distributed transaction commits ensures that all + involved changes on foreign servers are committed or not. Valid + values are required and disabled. + The default setting is disabled. Setting to + disabled don't use two-phase commit protocol to + commit or rollback distributed transactions. When set to + required distributed transactions strictly requires + that all written servers can use two-phase commit protocol. That is, + the distributed transaction cannot commit if even one server does not + support the prepare callback routine + (described in ). + In required case, distributed transaction commit will + wait for all involving foreign transaction to be committed before the + command return a "success" indication to the client. + + + + This parameter can be changed at any time; the behavior for any one + transaction is determined by the setting in effect when it commits. + + + + + When disabled there can be risk of database + consistency if one or more foreign servers crashes while committing + the distributed transactions. + + + + + + + max_prepared_foreign_transactions (integer) + + max_prepared_foreign_transactions configuration parameter + + + + + Sets the maximum number of foreign transactions that can be prepared + simultaneously. A single local transaction can give rise to multiple + foreign transaction. If a user expects N local + transactions and each of those involves K foreign + servers, this value need to be set N * K, not + just N. This parameter can only be set at server + start. + + + When running a standby server, you must set this parameter to the + same or higher value than on the master server. Otherwise, queries + will not be allowed in the standby server. + + + + + + + + + Foreign Transaction Resolvers + + + These settings control the behavior of a foreign transaction resolver. + + + + + max_foreign_transaction_resolvers (int) + + max_foreign_transaction_resolvers configuration parameter + + + + + Specifies maximum number of foreign transaction resolution workers. A foreign transaction + resolver is responsible for foreign transaction resolution on one database. + + + Foreign transaction resolution workers are taken from the pool defined by + max_worker_processes. + + + The default value is 0. + + + + + + foreign_transaction_resolution_retry_interval (integer) + + foreign_transaction_resolution_interval configuration parameter + + + + + Specify how long the foreign transaction resolver should wait when the last resolution + fails before retrying to resolve foreign transaction. This parameter can only be set in the + postgresql.conf file or on the server command line. + + + The default value is 10 seconds. + + + + + + foreign_transaction_resolver_timeout (integer) + + foreign_transaction_resolver_timeout configuration parameter + + + + + Terminate foreign transaction resolver processes that don't have any foreign + transactions to resolve longer than the specified number of milliseconds. + A value of zero disables the timeout mechanism, meaning it connects to one + database until stopping manually by pg_stop_foreign_xact_resovler(). + This parameter can only be set in the postgresql.conf + file or on the server command line. + + + The default value is 60 seconds. + + + + + + + Version and Platform Compatibility diff --git a/doc/src/sgml/distributed-transaction.sgml b/doc/src/sgml/distributed-transaction.sgml new file mode 100644 index 0000000000..1106fe00c9 --- /dev/null +++ b/doc/src/sgml/distributed-transaction.sgml @@ -0,0 +1,158 @@ + + + + Distributed Transaction + + + A distributed transaction is a transaction in which two or more network hosts + are involved. PostgreSQL's global transaction + manager supports distributed transactions that access foreign servers using + Foreign Data Wrappers. + + + + Atomic Commit + + + Formerly, transactions on foreign servers were simply committed or rolled + back one by one. Therefore, when one foreign server had a problem during + commit, it was possible that transactions on only part of foreign servers + are committed while other transactions are rolled back. This used to leave + database data in an inconsistent state in terms of a federated database. + Atomic commit of distributed transactions is an operation that applies a set + of changes as a single operation globally. This guarantees all-or-nothing + results for the changes on all remote hosts involved in. + PostgreSQL provides a way to perform read-write + transactions with foreign resources using foreign data wrappers. + Using PostgreSQL's atomic commit ensures that + all changes on foreign servers are either committed or rolled back using the + transaction callback routines + (see ). + + + + Atomic Commit Using Two-phase Commit Protocol + + + To achieve commit among all foreign servers automatically, + PostgreSQL employs two-phase commit protocol, + which is a type of atomic commitment protocol (ACP). Using two-phase + commit protocol, the commit sequence of distributed transaction performs + with the following steps: + + + + Prepare all transactions on foreign servers. + PostgreSQL's distributed transaction manager + prepares all transactions on the foreign servers if two-phase commit is + required. Two-phase commit is required when the transaction modifies + data on two or more servers including the local server itself and + is + required. If the preparation on all foreign servers + is successful then go to the next step. If there is any failure in the + prepare phase, the server will rollback all the transactions on both + local and foreign servers. + + + + + Commit the local transaction. The server commits the transaction locally. + Once the local transaction gets committed, we will never rollback any + involved transactions. + + + + + Resolve all prepared transactions on foreign servers. Prepared transactions + are committed or rolled back according to the result of the local transaction. + This step is performed by a foreign transaction resolver process. + + + + + + + The above sequence is executed transparently to the users at transaction commit. + The transaction returns an acknowledgment of the successful commit of the + distributed transaction to the client after step 2. After that, all + prepared transactions are resolved asynchronously by a foreign transaction + resolver process. + + + + When the user executes PREPARE TRANSACTION, the transaction + prepares the local transactions as well as all involved transactions on the + foreign servers. Likewise, when COMMIT PREPARED or + ROLLBACK PREPARED all prepared transactions are resolved + asynchronously after committing or rolling back the local transaction. + + + + + In-Doubt Transactions + + + Distributed transaction can become in-doubt state + after preparing the all involved transactions until the all involved + transaction are resolved. In case where the local node crashes during + preparing transactions, the distributed transaction becomes in-doubt + state. The information of involved foreign transactions is recovered + during crash recovery and these are resolved in background. Until all + in-doubt state transactions are resolved, other transactions might see + an inconsistent results on the foreign servers on reading. + + + + The foreign transaction resolver processes automatically resolve the + transactions associated with the in-doubt distributed transaction. Or you + can use pg_resolve_foriegn_xact function to resolve + it manually. + + + + + Foreign Transaction Resolver Processes + + + Foreign transaction resolver processes are auxiliary processes that are + responsible for resolving in-doubt distributed transactions. They commit or + rollback prepared transactions on all foreign servers involved with the + distributed transaction according to the result of the corresponding local + transaction. + + + + One foreign transaction resolver is responsible for transaction resolutions + on the database to which it is connected. On failure during resolution, they + retry to resolve at an interval of + foreign_transaction_resolution_interval time. + + + + + During a foreign transaction resolver process connecting to the database, + the database cannot be dropped without immediate shutdown. You can call + pg_stop_foreign_xact_resovler function to stop the + particular resolver process before dropping the database. + + + + + + Configuration Settings + + + Atomic commit requires several configuration options to be set. + On the local node, and + must be non-zero value, + and must be enabled. Additionally, + the max_worker_processes may need to be adjusted + to accommodate for foreign transaction resolver workers, at least + (max_foreign_transaction_resolvers + 1). + Note that other PostgreSQL features such as parallel + queries, logical replication, etc., also take worker slots from + max_worker_processes. + + + + diff --git a/doc/src/sgml/fdwhandler.sgml b/doc/src/sgml/fdwhandler.sgml index 8aa7edfe4a..bbf7c0b488 100644 --- a/doc/src/sgml/fdwhandler.sgml +++ b/doc/src/sgml/fdwhandler.sgml @@ -1657,6 +1657,117 @@ ReparameterizeForeignPathByChild(PlannerInfo *root, List *fdw_private, + + FDW Routines For Transaction Management + + + Transaction management callbacks are used to commit, rollback, and + prepare the foreign transaction. If an FDW wishes that its foreign + transaction is managed by PostgreSQL's global + transaction manager it must provide both + CommitForeignTransaction and + RollbackForeignTransaction. In addition, if an FDW + wishes to support atomic commit (as described in + ), it must provide + PrepareForeignTransaction as well and can provide + GetPrepareId callback optionally. + + + + +void +PrepareForeignTransaction(FdwXactInfo *finfo); + + Prepare the transaction on the foreign server. This function is called at the + pre-commit phase of the local transactions if foreign twophase commit is + required. This function is used only for distributed transaction management + (see ). + + + + Note that this callback function is always executed by backend processes. + + + +bool +CommitForeignTransaction(FdwXactInfo *finfo); + + Commit the foreign transaction. This function is called either at + the pre-commit phase of the local transaction if the transaction + can be committed in one-phase or at the post-commit phase if + two-phase commit is required. If finfo->flags has + the flag FDW_XACT_FLAG_ONEPHASE the transaction + can be committed in one-phase, this function must commit the prepared + transaction identified by finfo->identifier. + + + + Note that all cases except for calling pg_resolve_fdwxact + SQL function, this callback function is executed by foreign transaction + resolver processes. + + + +bool +RollbackForeignTransaction(FdwXactInfo *finfo); + + Rollback the foreign transaction. This function is called either at + the end of the local transaction after rolled back locally. The foreign + transactions are rolled back when a user requested rollbacking or when + an error occurs during the transaction. This function must be tolerant to + being called recursively if any error occurs during rollback of the foreign + transaction. So you would need to track recursion and prevent being called + infinitely. If finfo->flags has the flag + FDW_XACT_FLAG_ONEPHASE the transaction can be rolled + back in one-phase, otherwise this function must rollback the prepared + transaction identified by finfo->identifier. + + + + The foreign transaction identified by finfo->identifier + might not exist on the foreign servers. This can happen when, for instance, + there is a failure during preparing the foreign transaction. Therefore, this + function needs to tolerate the undefined object error + (ERRCODE_UNDEFINED_OBJECT) rather than raising an error. + + + + Note that in all cases except for calling pg_resolve_fdwxact + SQL function, this callback function is executed by foreign transaction + resolver processes. + + + +char * +GetPrepareId(TransactionId xid, Oid serverid, Oid userid, int *prep_id_len); + + Return null-terminated string that represents prepared transaction identifier + with its length *prep_id_len. + This optional function is called during executor startup for once per the + foreign server. Note that the transaction identifier must be a string literal, + less than NAMEDATALEN bytes long and should not be same + as any other concurrent prepared transaction id. If this callback routine + is not supported, PostgreSQL's distributed + transaction manager generates a unique identifier in the form of + fx_<random value up to 231>_<xid>_<user mapping oid>. + + + + Note that this callback function is always executed by backend processes. + + + + + Functions PrepareForeignTransaction, + CommitForeignTransaction and + RolblackForeignTransaction are called + outside of a valid transaction state. So please note that + you cannot use functions that use the system catalog cache + such as Foreign Data Wrapper helper functions described in + . + + + @@ -2136,4 +2247,138 @@ GetForeignServerByName(const char *name, bool missing_ok); + + Transaction managements for Foreign Data Wrappers + + If a server used by an FDW supports transactions, it is usually worthwhile + for the FDW to manage transactions opened on the foreign server. The FDW + callback function CommitForeignTransaction, + RollbackForeignTransaction and + PrepareForeignTransaction are used for transaction + management and must fit into the working of the + PostgreSQL transaction processing. + + + + The information in FdwXactInfo can be used to get + information of foreign server being processed such as + ForeignServer and UserMapping + The flags has contains flag bit describing the + foreign transaction state for transaction management. + + + + The foreign transaction needs to be registered to + PostgreSQL global transaction manager. + Registration and unregistration are done by calling + FdwXactRegisterXact and + FdwXactUnregisterXact respectively. + The FDW can pass a boolean modified along with + UserMapping to FdwXactRegisterXact + indicating that writes are going to happen on the foreign server. Such + foreign servers are taken into account for the decision of two-phase + commit protocol being required or not. + + + + The FDW callback function CommitForeignTransaction + and RollbackForeignTransaction are used to commit + and rollback foreign transactions. During transaction commit, the global + transaction manager calls CommitForeignTransaction function + in the pre-commit phase and calls + RollbackForeignTransaction function in the post-rollback + phase. + + + + In addition to simply commit and rollback foreign transactions, + PostgreSQL global transaction manager enables + distributed transactions to atomically commit and rollback among all foreign + servers, which is as known as atomic commit in literature. To achieve atomic + commit, PostgreSQL employs two-phase commit + protocol, which is a type of atomic commitment protocol. Every FDWs that wish + to support two-phase commit protocol are required to have the FDW callback + function PrepareForeignTransaction and optionally + GetPrepareId, in addition to + CommitForeignTransaction and + RollbackForeignTransaction + (see for details). + + + + An example of a distributed transaction is as follows + +BEGIN; +UPDATE ft1 SET col = 'a'; +UPDATE ft2 SET col = 'b'; +COMMIT; + + ft1 and ft2 are foreign tables on different foreign servers may be using different + Foreign Data Wrappers. + + + + When the core executor access the foreign servers, foreign servers whose FDW + supports transaction management callback routines is registered as a participant. + During registration, GetPrepareId is called if provided to + generate a unique transaction identifier. + + + + During pre-commit phase of the local transaction, the foreign transaction manager + persists the foreign transaction information to the disk and WAL, and then + prepare all foreign transactions by calling + PrepareForeignTransaction if two-phase commit protocol + is required. Two-phase commit is required when the transaction modified data + on more than one server including the local server itself and a user requests + foreign twophase commit (see ). + + + + PostgreSQL commits locally and go to the next + step if and only if all foreign transactions are prepared successfully. + If any failure happens or a user requests to cancel during preparation, + the global transaction manager changes over rollback and calls + RollbackForeignTransaction. + + + + When changing over rollback due to any failure, it calls + RollbackForeignTransaction with + FDWXACT_FLAG_ONEPHASE for foreign transactions which are not + closed yet and calls RollbackForeignTransaction without + that flag for foreign transactions which are already prepared. For foreign + transactions which are being prepared, it does both because it's not sure that + the preparation has been completed on the foreign server. Therefore, + RollbackForeignTransaction needs to tolerate the undefined + object error. + + + + Note that when (finfo->flags & FDWXACT_FLAG_ONEPHASE) + is true, both CommitForeignTransaction function and + RollbackForeignTransaction function should commit and + rollback directly, rather than processing prepared transactions. This can + happen when two-phase commit is not required or a foreign server is not + modified within the transaction. + + + + Once all foreign transactions are prepared, the core transaction manager commits + locally. After that the transaction commit waits for all prepared foreign + transaction to be committed before completion. After all prepared foreign + transactions are resolved the transaction commit completes. + + + + One foreign transaction resolver process is responsible for foreign + transaction resolution on a database. The foreign transaction resolver process + calls either CommitForeignTransaction or + RollbackForeignTransaction to resolve the foreign + transaction identified by finfo->identifier. If failed + to resolve, the resolver process will exit with an error message. The foreign + transaction launcher will launch the resolver process again at + interval. + + diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml index 45b701426b..3751d734c6 100644 --- a/doc/src/sgml/filelist.sgml +++ b/doc/src/sgml/filelist.sgml @@ -49,6 +49,7 @@ + diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 4d1f1794ca..49a8b13f57 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -27285,6 +27285,153 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8'); + + Data Sanity Functions + + + The functions shown in + provide ways to check the sanity of data files in the cluster. + + + + Data Sanity Functions + + + + + Function + + + Description + + + + + + + + + pg_relation_check_pages + + pg_relation_check_pages ( relation regclass [, fork text ] ) + setof record + ( path text, + failed_block_num bigint ) + + + Checks the pages of the specified relation to see if they are valid + enough to safely be loaded into the server's shared buffers. If + given, fork specifies that only the pages of + the given fork are to be verified. fork can + be main for the main data + fork, fsm for the free space + map, vm for the visibility map, + or init for the initialization fork. The + default of NULL means that all forks of the + relation should be checked. The function returns a list of block + numbers that appear corrupted along with the path names of their + files. Use of this function is restricted to superusers by + default, but access may be granted to others + using GRANT. + + + + +
+ + + + + Foreign Transaction Management Functions + + + pg_resolve_foreign_xact + + + pg_remove_foreign_xact + + + + shows the functions + available for foreign transaction management. + These functions cannot be executed during recovery. Use of these function + is restricted to superusers. + + + + Foreign Transaction Management Functions + + + Name Return Type Description + + + + + + pg_resolve_foreign_xact(transaction xid, userid oid, userid oid) + + bool + + Resolve a foreign transaction. This function searches for foreign + transaction matching the arguments and resolves it. Once the foreign + transaction is resolved successfully, this function removes the + corresponding entry from . + This function won't resolve a foreign transaction which is being + processed. + + + + + pg_remove_foreign_xact(transaction xid, serverid oid, userid oid) + + void + + This function works the same as pg_resolve_foreign_xact + except that this removes the foreign transaction entry without resolution. + This function is useful to remove a foreign transaction entry whose foreign + server is no longer available. + + + + +
+ + + The function shown in + control the foreign transaction resolvers. + + + + Foreign Transaction Resolver Control Functions + + + + Name Return Type Description + + + + + + + pg_stop_fdwxact_resolver(dbid oid) + + bool + + Stop the foreign transaction resolver running on the given database. + This function is useful for stopping a resolver process on the database + that you want to drop. + + + + +
+ + + pg_stop_fdwxact_resolver is useful to be used before + dropping the database to that the foreign transaction resolver is connecting. + + + diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index dcbb10fb6f..94006d0b2a 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -1094,6 +1094,18 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser CheckpointerMain Waiting in main loop of checkpointer process. + + FdwXactLauncherMain + Waiting in main loop of foreign transaction resolution launcher process. + + + FdwXactResolverMain + Waiting in main loop of foreign transaction resolution worker process. + + + LogicalLauncherMain + Waiting in main loop of logical launcher process. + LogicalApplyMain Waiting in main loop of logical replication apply process. @@ -1318,6 +1330,18 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser DataFileWrite Waiting for a write to a relation data file. + + FdwXactFileRead + Waiting for a read of a foreign transaction state file. + + + FdwXactFileSync + Waiting for a foreign transaction state file to reach stable storage. + + + FdwXactFileWrite + Waiting for a write of a foreign transaction state file. + LockFileAddToDataDirRead Waiting for a read while adding a line to the data directory lock @@ -1624,6 +1648,11 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser Waiting for activity from a child process while executing a Gather plan node. + + FdwXactResolution + Waiting for all foreign transaction participants to be resolved during + atomic commit among foreign servers. + HashBatchAllocate Waiting for an elected Parallel Hash participant to allocate a hash @@ -1942,6 +1971,19 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser Waiting to read or update dynamic shared memory allocation information. + + FdwXactLock + Waiting to read or update the state of foreign transactions. + + + FdwXactResolutionLock + Waiting to read or update information of foreign transaction + resolution. + + + LogicalRepWorkerLock + Waiting for action on logical replication worker to finish. + LockFastPath Waiting to read or update a process' fast-path lock diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml index d453be3909..eca35c4a84 100644 --- a/doc/src/sgml/postgres.sgml +++ b/doc/src/sgml/postgres.sgml @@ -171,6 +171,7 @@ break is not needed in a wider output rendering. &wal; &logical-replication; &jit; + &distributed-transaction; ®ress; diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml index bfccda77af..22cd494366 100644 --- a/doc/src/sgml/storage.sgml +++ b/doc/src/sgml/storage.sgml @@ -83,6 +83,12 @@ Item subsystem + + pg_fdwxact + Subdirectory containing files used by the distributed transaction + manager subsystem + + pg_logical Subdirectory containing status data for logical decoding diff --git a/src/backend/access/transam/README.fdwxact b/src/backend/access/transam/README.fdwxact new file mode 100644 index 0000000000..8da9030689 --- /dev/null +++ b/src/backend/access/transam/README.fdwxact @@ -0,0 +1,134 @@ +src/backend/access/transam/README.fdwxact + +Atomic Commit for Distributed Transactions +=========================================== + +The atomic commit feature enables us to commit and rollback either all of +foreign servers or nothing. This ensures that the database data is always left +in a conssitent state in term of federated database. + + +Commit Sequence of Global Transactions +-------------------------------- + +We employee two-phase commit protocol to achieve commit among all foreign +servers atomically. The sequence of distributed transaction commit consisnts +of the following four steps: + +1. Foriegn Server Registration +During executor node initialization, accessed foreign servers are registered +to the list FdwXactParticipant, which is maintained by PostgreSQL's the global +transaction manager (GTM), as a distributed transaction participant The +registered foreign transactions are tracked until the end of transaction. + +2. Pre-Commit phase (1st phase of two-phase commit) +we record the corresponding WAL indicating that the foreign server is involved +with the current transaction before doing PREPARE all foreign transactions. +Thus, in case we loose connectivity to the foreign server or crash ourselves, +we will remember that we might have prepared tranascation on the foreign +server, and try to resolve it when connectivity is restored or after crash +recovery. + +The two-phase commit is required only if the transaction modified two or more +servers including the local node. + +After that we prepare all foreign transactions by calling +PrepareForeignTransaction() API. If we failed on any of them we change to +rollback, therefore at this time some participants might be prepared whereas +some are not prepared. The former foreign transactions are resolved by +the resolver process asynchronusly or can be resolved using by +pg_resolve_foreign_xact() manually, and the latter ends transaction +in one-phase by calling RollbackForeignTransaction() API. + +3. Commit locally +Once we've prepared all of them, commit the transaction locally. + +4. Post-Commit Phase (2nd phase of two-phase commit) +The steps so far are done by the backend process committing the transaction but +this resolution step(commit or rollback) is done by the foreign transaction +resolver process. + + +Identifying Foreign Transactions In GTM +--------------------------------------- + +To identify foreign transaction participants (as well as FdwXact entries) there +are two ways: using {server OID, user OID} and using user mapping OID. The same +is true for FDWs to identify the connections (and transactions upon) to the +foreign server. We need to consider the case where the way to identify the +transactions is not matched between GTM and FDWs, because the problem might occur +when the user modifies the same foreign server by different roles within the +transaction. For example, consider the following execution: + +BEGIN; +SET ROLE user_A; +INSERT INTO ft1 VALUES (1); +SET ROLE user_B; +INSERT INTO ft1 VALUES (1); +COMMIT; + +For example, suppose that an FDW identifies the connection by {server OID, user OID} +and GTM identifies the transactions by user mapping OID, and user_A and user_B use +the public user mapping to connect server_X. In the FDW, there are two +connections: {user_A, sever_X} and {user_B, server_X}, and therefore opens two +transactions on each connection, while GTM has only one FdwXact entry because the two +connections refer to the same user mapping OID. As a result, at the end of the +transaction, GTM ends only one foreign transaction, leaving another one. + +On the other hand, suppose that an FDW identifies the connection by user mapping OID +and GTM does that by {server OID, user OID}, the FDW uses only one connection and opens +a transaction since both users refer to the same user mapping OID (we expect FDWs +not to register the foreign transaction when not starting a new transaction on the +foreign server). Since GTM also has one entry it can end the foreign transaciton +properly. The downside would be that the user OID of FdwXact (i.g., FdwXact->userid) +is the user who registered the foreign transaction for the first time, necessarily +not the user who executed COMMIT. For example in the above case, FdwXact->userid +will be user_A, not user_B. But it’s not big problem in practice. + +Therefore, in fdwxact.c, we identify the foreign transaction by +{server OID, user OID}. + +Foreign Transactions Status +---------------------------- + +Every foreign transaction has an FdwXact entry. When preparing a foreign +transaction a FdwXact entry of which status starts from FDWXACT_STATUS_PREPARING +are created with WAL logging. The status changes to FDWXACT_STATUS_PREPARED +after the foreign transaction is prepared. And the status changes to +FDWXACT_STATUS_COMMITTING and FDWXACT_STATUS_ABORTING before committing and +aborting respectively. FdwXact entry is removed with WAL logging after resolved. + +FdwXact entries recovered during the recovery are marked as in-doubt if the +corresponding local transaction is not prepared transaction. The initial +status for those entries is FDWXACT_STATUS_PREPARED if they are recovered +from WAL. Because we WAL logs only when preparing the foreign transaction we +cannot know the exact fate of the foreign transaction from the recovery. + +The foreign transaction status transition is illustrated by the following +graph describing the FdwXact->status: + + +----------------------------------------------------+ + | INVALID | + +----------------------------------------------------+ + | | | + | v | + | +---------------------+ | + (*1) | PREPARING | (*1) + | +---------------------+ | + | | | + v v v + +----------------------------------------------------+ + | PREPARED | + +----------------------------------------------------+ + | | + v v + +--------------------+ +--------------------+ + | COMMITTING | | ABORTING | + +--------------------+ +--------------------+ + | | + v v + +----------------------------------------------------+ + | END | + +----------------------------------------------------+ + +(*1) Paths for recovered FdwXact entries -- 2.24.3 (Apple Git-128)