From 8ce45564b0b44e847915b8550f0b8ad9f5f233fb Mon Sep 17 00:00:00 2001 From: Masahiko Sawada Date: Tue, 19 May 2020 15:06:38 +0900 Subject: [PATCH v34 10/11] Documentation update. Co-authored-by: Masahiko Sawada, Ashutosh Bapat --- doc/src/sgml/catalogs.sgml | 135 ++++++++++++ doc/src/sgml/config.sgml | 144 ++++++++++++ doc/src/sgml/distributed-transaction.sgml | 158 ++++++++++++++ doc/src/sgml/fdwhandler.sgml | 254 ++++++++++++++++++++++ doc/src/sgml/filelist.sgml | 1 + doc/src/sgml/func.sgml | 147 +++++++++++++ doc/src/sgml/monitoring.sgml | 42 ++++ doc/src/sgml/postgres.sgml | 1 + doc/src/sgml/storage.sgml | 6 + src/backend/access/transam/README.fdwxact | 134 ++++++++++++ 10 files changed, 1022 insertions(+) create mode 100644 doc/src/sgml/distributed-transaction.sgml create mode 100644 src/backend/access/transam/README.fdwxact diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 865e826fb0..aaeebdd34a 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -9299,6 +9299,11 @@ SCRAM-SHA-256$<iteration count>:&l summary of configuration file contents + + pg_foreign_xacts + foreign transactions + + pg_group groups of database users @@ -11152,6 +11157,136 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx + + <structname>pg_foreign_xacts</structname> + + + pg_foreign_xacts + + + + The view pg_foreign_xacts displays + information about foreign transactions that are opened on + foreign servers for atomic distributed transaction commit (see + for details). + + + + pg_foreign_xacts contains one row per foreign + transaction. An entry is removed when the foreign transaction is + committed or rolled back. + + + + <structname>pg_foreign_xacts</structname> Columns + + + + + Name + Type + References + Description + + + + + dbid + oid + pg_database.oid + + OID of the database which the foreign transaction resides in + + + + xid + xid + + + Numeric transaction identifier with which this foreign transaction + associates + + + + serverid + oid + pg_foreign_server.oid + + The OID of the foreign server on which the foreign transaction is prepared + + + + userid + oid + pg_user.oid + + The OID of the user that prepared this foreign transaction. + + + + status + text + + + Status of foreign transaction. Possible values are: + + + + preparing : This foreign transaction is being prepared. + + + + + prepared : This foreign transaction has been prepared. + + + + + comitting : This foreign transcation has been + prepared to commit or being committed. + + + + + abortin : This foreign transaction has been + prepared to abort or being aborted. + + + + + + + locker_pid + int + + + Process ID of the locker currently processing. + + + + identifier + text + + + The identifier of the prepared foreign transaction. + + + + +
+ + + When the pg_foreign_xacts view is accessed, the + internal transaction manager data structures are momentarily locked, and + a copy is made for the view to display. This ensures that the + view produces a consistent set of results, while not blocking + normal operations longer than necessary. Nonetheless + there could be some impact on database performance if this view is + frequently accessed. + + + + <structname>pg_publication_tables</structname> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 82864bbb24..032801658c 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -9336,6 +9336,150 @@ dynamic_library_path = 'C:\tools\postgresql;H:\my_project\lib;$libdir' + + Distributed Transaction Management + + + Setting + + + + foreign_twophase_commit (enum) + + foreign_twophase_commit configuration parameter + + + + + Specifies whether distributed transaction commits ensures that all + involved changes on foreign servers are committed or not. Valid + values are required and disabled. + The default setting is disabled. Setting to + disabled don't use two-phase commit protocol to + commit or rollback distributed transactions. When set to + required distributed transactions strictly requires + that all written servers can use two-phase commit protocol. That is, + the distributed transaction cannot commit if even one server does not + support the prepare callback routine + (described in ). + In required case, distributed transaction commit will + wait for all involving foreign transaction to be committed before the + command return a "success" indication to the client. + + + + This parameter can be changed at any time; the behavior for any one + transaction is determined by the setting in effect when it commits. + + + + + When disabled there can be risk of database + consistency if one or more foreign servers crashes while committing + the distributed transactions. + + + + + + + max_prepared_foreign_transactions (integer) + + max_prepared_foreign_transactions configuration parameter + + + + + Sets the maximum number of foreign transactions that can be prepared + simultaneously. A single local transaction can give rise to multiple + foreign transaction. If a user expects N local + transactions and each of those involves K foreign + servers, this value need to be set N * K, not + just N. This parameter can only be set at server + start. + + + When running a standby server, you must set this parameter to the + same or higher value than on the master server. Otherwise, queries + will not be allowed in the standby server. + + + + + + + + + Foreign Transaction Resolvers + + + These settings control the behavior of a foreign transaction resolver. + + + + + max_foreign_transaction_resolvers (int) + + max_foreign_transaction_resolvers configuration parameter + + + + + Specifies maximum number of foreign transaction resolution workers. A foreign transaction + resolver is responsible for foreign transaction resolution on one database. + + + Foreign transaction resolution workers are taken from the pool defined by + max_worker_processes. + + + The default value is 0. + + + + + + foreign_transaction_resolution_retry_interval (integer) + + foreign_transaction_resolution_interval configuration parameter + + + + + Specify how long the foreign transaction resolver should wait when the last resolution + fails before retrying to resolve foreign transaction. This parameter can only be set in the + postgresql.conf file or on the server command line. + + + The default value is 10 seconds. + + + + + + foreign_transaction_resolver_timeout (integer) + + foreign_transaction_resolver_timeout configuration parameter + + + + + Terminate foreign transaction resolver processes that don't have any foreign + transactions to resolve longer than the specified number of milliseconds. + A value of zero disables the timeout mechanism, meaning it connects to one + database until stopping manually by pg_stop_foreign_xact_resovler(). + This parameter can only be set in the postgresql.conf + file or on the server command line. + + + The default value is 60 seconds. + + + + + + + Version and Platform Compatibility diff --git a/doc/src/sgml/distributed-transaction.sgml b/doc/src/sgml/distributed-transaction.sgml new file mode 100644 index 0000000000..bae3ee0f2a --- /dev/null +++ b/doc/src/sgml/distributed-transaction.sgml @@ -0,0 +1,158 @@ + + + + Distributed Transaction + + + A distributed transaction is a transaction in which two or more network hosts + are involved. PostgreSQL's global Transaction + manager supports distributed transactions that access foreign servers using + Foreign Data Wrappers. The global transaction manager is responsible for + managing transactions on foreign servers. + + + + Atomic Commit + + + Formerly, transactions on foreign server were simply committed or rolled + back one by one. Therefore, when one foreign server had a problem during + commit, it was possible that transactions on only part of foreign servers + are committed while other transactions are rolled back. This used to leave + database data in an inconsistent state in term of federated database. + Atomic commit of distributed transaction is an operation that applies a set + of changes as a single operation globally. This guarantees all-or-nothing + results for the changes on all remote hosts involved in. + PostgreSQL provides a way to perform read-write + transactions with foreign resources using foreign data wrappers. + Using the PostgreSQL's atomic commit ensures that + all the changes on foreign servers are either committed or rolled back using + the transaction callback routines + (see ). + + + + Atomic Commit Using Two-phase Commit Protocol + + + To achieve commit among all foreign servers automatically, + PostgreSQL employs two-phase commit protocol, + which is a type of atomic commitment protocol (ACP). Using two-phase + commit protocol, the commit sequence of distributed transaction performs + with the following steps: + + + + Prepare all transactions on foreign servers. + PostgreSQL's distributed transaction manager + prepares all transaction on the foreign servers if two-phase commit is + required. Two-phase commit is required when the transaction modifies + data on two or more servers including the local server itself and + is + required. If the prepare on all foreign servers is + successful then go to the next step. If there is any failure in the + prepare phase, the server will rollback all the transactions on both + local and foreign servers. + + + + + Commit the local transaction. The server commits transaction locally. + Any failure happens in this step the server changes to rollback, then + rollback all transactions on both local and foreign servers. + + + + + Resolve all prepared transaction on foreign servers. Prepared transactions + are committed or rolled back according to the result of the local transaction. + This step is performed by a foreign transaction resolver process. + + + + + + + The above sequence is executed transparently to the users at transaction commit. + The transaction returns acknowledgement of the successful commit of the + distributed transaction to the client after the step 2. After that, the all + prepared transactions are resolved asynchronously by a foreign transaction + resolver process. + + + + When the user executes PREPARE TRANSACTION, the transaction + prepares the local transactions as well as all involved transactions on the + foreign servers. Likewise, when COMMIT PREPARED or + ROLLBACK PREPARED all prepared transactions are resolved + asynchronously after committing or rolling back the local transaction. + + + + + In-Doubt Transactions + + + Distributed transaction can become in-doubt state + after preparing the all involved transactions until the all involved + transaction are resolved. During that, the transaction might see the different + results on the foreign servers on reading. In case where the local node + crashes during preparing transactions, the distributed transaction becomes + in-doubt state. The information of involved foreign transactions is + recovered during crash recovery and these are resolved in background. + + + + The foreign transaction resolver processes automatically resolves the + transactions associated with the in-doubt distributed transaction. Or you can + use pg_resolve_foriegn_xact function to resolve it + manually. + + + + + Foreign Transaction Resolver Processes + + + Foreign transaction resolver processes are auxiliary processes that are + responsible for resolving in-doubt distributed transactions. They commit or + rollback prepared transactions on all foreign servers involved with the + distributed transaction according to the result of the corresponding local + transaction. + + + + One foreign transaction resolver is responsible for transaction resolutions + on the database to which it is connected. On failure during resolution, they + retry to resolve at an interval of + foreign_transaction_resolution_interval time. + + + + + During a foreign transaction resolver process connecting to the database, + database cannot be dropped without immediate shutdown. You can call + pg_stop_foreign_xact_resovler function to stop the + particular resolver process before dropping the database. + + + + + + Configuration Settings + + + Atomic commit requires several configuration options to be set. + On the local node, and + must be non-zero value, + and must be enabled. Additionally + the max_worker_processes may need to be adjusted + to accommodate for foreign transaction resolver workers, at least + (max_foreign_transaction_resolvers + 1). + Note that other PostgreSQL features such as parallel + queries, logical replication, etc., also take worker slots from + max_worker_processes. + + + + diff --git a/doc/src/sgml/fdwhandler.sgml b/doc/src/sgml/fdwhandler.sgml index 854913ae5f..3c056193f0 100644 --- a/doc/src/sgml/fdwhandler.sgml +++ b/doc/src/sgml/fdwhandler.sgml @@ -1504,6 +1504,117 @@ ReparameterizeForeignPathByChild(PlannerInfo *root, List *fdw_private, + + FDW Routines For Transaction Management + + + Transaction management callbacks are used to commit, rollback, and + prepare the foreign transaction. If an FDW wishes that its foreign + transaction is managed by PostgreSQL's global + transaction manager it must provide both + CommitForeignTransaction and + RollbackForeignTransaction. In addition, if an FDW + wishes to support atomic commit (as described in + ), it must provide + PrepareForeignTransaction as well and can provide + GetPrepareId callback optionally. + + + + +void +PrepareForeignTransaction(FdwXactInfo *finfo); + + Prepare the transaction on the foreign server. This function is called at the + pre-commit phase of the local transactions if foreign twophase commit is + required. This function is used only for distributed transaction management + (see ). + + + + Note that this callback function is always executed by backend processes. + + + +bool +CommitForeignTransaction(FdwXactInfo *finfo); + + Commit the foreign transaction. This function is called either at + the pre-commit phase of the local transaction if the transaction + can be committed in one-phase or at the post-commit phase if + two-phase commit is required. If frstate->flag has + the flag FDW_XACT_FLAG_ONEPHASE the transaction + can be committed in one-phase, this function must commit the prepared + transaction identified by frstate->fdwxact_id. + + + + Note that all cases except for calling pg_resolve_fdwxact + SQL function, this callback function is executed by foreign transaction + resolver processes. + + + +bool +RollbackForeignTransaction(FdwXactInfo *finfo); + + Rollback the foreign transaction. This function is called either at + the end of the local transaction after rolled back locally. The foreign + transactions are rolled back when user requested rollbacking or when + any error occurs during the transaction. This function must be tolerate to + being called recursively if any error occurs during rollback the foreign + transaction. So you would need to track recursion and prevent being called + infinitely. If frstate->flag has the flag + FDW_XACT_FLAG_ONEPHASE the transaction can be rolled + back in one-phase, otherwise this function must rollback the prepared + transaction identified by frstate->fdwxact_id. + + + + The foreign transaction identified by frstate->fdwxact_id + might not exist on the foreign servers. This can happen when, for instance, + there is failure during preparing the foreign tranasction. Therefore, this + function needs to tolerate the undefined object error + (ERRCODE_UNDEFINED_OBJECT) rather than raising an error. + + + + Note that all cases except for calling pg_resolve_fdwxact + SQL function, this callback function is executed by foreign transaction + resolver processes. + + + +char * +GetPrepareId(TransactionId xid, Oid serverid, Oid userid, int *prep_id_len); + + Return null terminated string that represents prepared transaction identifier + with its length *prep_id_len. + This optional function is called during executor startup for once per the + foreign server. Note that the transaction identifier must be string literal, + less than NAMEDATALEN bytes long and should not be same + as any other concurrent prepared transaction id. If this callback routine + is not supported, PostgreSQL's distributed + transaction manager generates an unique identifier with in the form of + fx_<random value up to 231>_<server oid>_<user oid>. + + + + Note that this callback function is always executed by backend processes. + + + + + Functions PrepareForeignTransaction, + CommitForeignTransaction and + RolblackForeignTransaction are called + at outside of a valid transaction state. So please note that + you cannot use functions that use the system catalog cache + such as Foreign Data Wrapper helper functions described in + . + + + @@ -1983,4 +2094,147 @@ GetForeignServerByName(const char *name, bool missing_ok); + + Transaction managements for Foreign Data Wrappers + + If a FDW's server supports transaction, it is usually worthwhile for the + FDW to manage transaction opened on the foreign server. The FDW callback + function CommitForeignTransaction, + RollbackForeignTransaction and + PrepareForeignTransaction are used for transaction + management and must fit into the working of the + PostgreSQL transaction processing. + + + + The information in FdwXactInfo can be used to get + information of foreign server being processed such as server name, OID of + server, user and user mapping. The flags has contains flag + bit describing the foreign transaction state for transaction management. + + + + Foreign Transaction Registration and Unregistration + + Foreign transaction needs to be registered to + PostgreSQL global transaction manager. + Registration and unregistration are done by calling + FdwXactRegisterXact and + FdwXactUnregisterXact respectively. + The FDW can pass a boolean modified along with + OIDs of server and user to FdwXactRegisterXact + indicating writes are going to happen on the foreign server. Such foreign + servers are taken into account for the decision of two-phase commit + protocol being required or not. + + + + + Commit and Rollback Single Foreign Transaction + + The FDW callback function CommitForeignTransaction + and RollbackForeignTransaction can be used to commit + and rollback the foreign transaction. During transaction commit, the core + transaction manager calls CommitForeignTransaction function + in the pre-commit phase and calls + RollbackForeignTransaction function in the post-rollback + phase. + + + + + Atomic Commit and Rollback Distributed Transaction + + In addition to simply commit and rollback foreign transactions described at + , + PostgreSQL global transaction manager enables + distributed transactions to atomically commit and rollback among all foreign + servers, which is as known as atomic commit in literature. To achieve atomic + commit, PostgreSQL employs two-phase commit + protocol, which is a type of atomic commitment protocol. Every FDWs that wish + to support two-phase commit protocol are required to have the FDW callback + function PrepareForeignTransaction and optionally + GetPrepareId, in addition to + CommitForeignTransaction and + RollbackForeignTransaction + (see for details). + + + + An example of distributed transaction is as follows + +BEGIN; +UPDATE ft1 SET col = 'a'; +UPDATE ft2 SET col = 'b'; +COMMIT; + + ft1 and ft2 are foreign tables on different foreign servers may be using different + Foreign Data Wrappers. + + + + When the core executor access the foreign servers, foreign servers whose FDW + supports transaction management callback routines is registered as a participant. + During registration, GetPrepareId is called if provided to + generate an unique transaction identifer. + + + + During pre-commit phase of local transaction, the foreign transaction manager + persists the foreign transaction information to the disk and WAL, and then + prepare all foreign transaction by calling + PrepareForeignTransaction if two-phase commit protocol + is required. Two-phase commit is required when the transaction modified data + on more than one servers including the local server itself and user requests + foreign twophase commit (see ). + + + + PostgreSQL can commit locally and go to the next + step if and only if all foreign transactions are prepared successfully. + If any failure happens or user requests to cancel during preparation, + the distributed transaction manager changes over rollback and calls + RollbackForeignTransaction. + + + + When changing over rollback due to any failure, it calls + RollbackForeignTransaction with + FDWXACT_FLAG_ONEPHASE for foreign transactions which are not + closed yet, and calls RollbackForeignTransaction without + that flag for foreign transactions which are already prepared. For foreign + transactions which are being prepared, it does both because it's not sure that + the preeparation has been completed on the foreign server. Therefore, + RollbackForeignTransaction needs to tolerate the undefined + object error. + + + + Note that when (frstate->flags & FDWXACT_FLAG_ONEPHASE) + is true, both CommitForeignTransaction function and + RollbackForeignTransaction function should commit and + rollback directly, rather than processing prepared transactions. This can + happen when two-phase commit is not required or foreign server is not + modified with in the transaction. + + + + Once all foreign transaction is prepared, the core transaction manager commits + locally. After that the transaction commit waits for all prepared foreign + transaction to be committed before completetion. After all prepared foreign + transactions are resolved the transaction commit completes. + + + + One foreign transaction resolver process is responsible for foreign + transaction resolution on a database. Foreign transaction resolver process + calls either CommitForeignTransaction or + RollbackForeignTransaction to resolve foreign + transaction identified by frstate->fdwxact_id. If failed + to resolve, resolver process will exit with an error message. The foreign + transaction launcher will launch the resolver process again at + interval. + + + diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml index 38e8aa0bbf..a5161bb22b 100644 --- a/doc/src/sgml/filelist.sgml +++ b/doc/src/sgml/filelist.sgml @@ -49,6 +49,7 @@ + diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index aa99665e2e..02a7bfa159 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -26906,6 +26906,153 @@ SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8'); + + Data Sanity Functions + + + The functions shown in + provide ways to check the sanity of data files in the cluster. + + + + Data Sanity Functions + + + + + Function + + + Description + + + + + + + + + pg_relation_check_pages + + pg_relation_check_pages ( relation regclass [, fork text ] ) + setof record + ( path text, + failed_block_num bigint ) + + + Checks the pages of the specified relation to see if they are valid + enough to safely be loaded into the server's shared buffers. If + given, fork specifies that only the pages of + the given fork are to be verified. fork can + be main for the main data + fork, fsm for the free space + map, vm for the visibility map, + or init for the initialization fork. The + default of NULL means that all forks of the + relation should be checked. The function returns a list of block + numbers that appear corrupted along with the path names of their + files. Use of this function is restricted to superusers by + default, but access may be granted to others + using GRANT. + + + + +
+ + + + + Foreign Transaction Management Functions + + + pg_resolve_foreign_xact + + + pg_remove_foreign_xact + + + + shows the functions + available for foreign transaction management. + These functions cannot be executed during recovery. Use of these function + is restricted to superusers. + + + + Foreign Transaction Management Functions + + + Name Return Type Description + + + + + + pg_resolve_foreign_xact(transaction xid, userid oid, userid oid) + + bool + + Resolve a foreign transaction. This function searches for foreign + transaction matching the arguments and resolves it. Once the foreign + transaction is resolved successfully, this function removes the + corresponding entry from . + This function won't resolve a foreign transaction which is being + processed. + + + + + pg_remove_foreign_xact(transaction xid, serverid oid, userid oid) + + void + + This function works the same as pg_resolve_foreign_xact + except that this removes the foreign transaction entry without resolution. + This function is useful to remove a foreign transaction entry whose foreign + server is no longer available. + + + + +
+ + + The function shown in + control the foreign transaction resolvers. + + + + Foreign Transaction Resolver Control Functions + + + + Name Return Type Description + + + + + + + pg_stop_fdwxact_resolver(dbid oid) + + bool + + Stop the foreign transaction resolver running on the given database. + This function is useful for stopping a resolver process on the database + that you want to drop. + + + + +
+ + + pg_stop_fdwxact_resolver is useful to be used before + dropping the database to that the foreign transaction resolver is connecting. + + + diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index 9496f76b1f..d0dd3b1341 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -1072,6 +1072,18 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser CheckpointerMain Waiting in main loop of checkpointer process. + + FdwXactLauncherMain + Waiting in main loop of foreign transaction resolution launcher process. + + + FdwXactResolverMain + Waiting in main loop of foreign transaction resolution worker process. + + + LogicalLauncherMain + Waiting in main loop of logical launcher process. + LogicalApplyMain Waiting in main loop of logical replication apply process. @@ -1301,6 +1313,18 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser DataFileWrite Waiting for a write to a relation data file. + + FdwXactFileRead + Waiting for a read of a foreign transaction state file. + + + FdwXactFileSync + Waiting for a foreign transaction state file to reach stable storage. + + + FdwXactFileWrite + Waiting for a write of a foreign transaction state file. + LockFileAddToDataDirRead Waiting for a read while adding a line to the data directory lock @@ -1594,6 +1618,11 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser Waiting for activity from a child process while executing a Gather plan node. + + FdwXactResolution + Waiting for all foreign transaction participants to be resolved during + atomic commit among foreign servers. + HashBatchAllocate Waiting for an elected Parallel Hash participant to allocate a hash @@ -1907,6 +1936,19 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser Waiting to read or update dynamic shared memory allocation information. + + FdwXactLock + Waiting to read or update the state of foreign transactions. + + + FdwXactResolutionLock + Waiting to read or update information of foreign transaction + resolution. + + + LogicalRepWorkerLock + Waiting for action on logical replication worker to finish. + LockFastPath Waiting to read or update a process' fast-path lock diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml index 730d5fdc34..a5c5619072 100644 --- a/doc/src/sgml/postgres.sgml +++ b/doc/src/sgml/postgres.sgml @@ -171,6 +171,7 @@ break is not needed in a wider output rendering. &wal; &logical-replication; &jit; + &distributed-transaction; ®ress; diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml index 3234adb639..83f30c5045 100644 --- a/doc/src/sgml/storage.sgml +++ b/doc/src/sgml/storage.sgml @@ -83,6 +83,12 @@ Item subsystem + + pg_fdwxact + Subdirectory containing files used by the distributed transaction + manager subsystem + + pg_logical Subdirectory containing status data for logical decoding diff --git a/src/backend/access/transam/README.fdwxact b/src/backend/access/transam/README.fdwxact new file mode 100644 index 0000000000..8da9030689 --- /dev/null +++ b/src/backend/access/transam/README.fdwxact @@ -0,0 +1,134 @@ +src/backend/access/transam/README.fdwxact + +Atomic Commit for Distributed Transactions +=========================================== + +The atomic commit feature enables us to commit and rollback either all of +foreign servers or nothing. This ensures that the database data is always left +in a conssitent state in term of federated database. + + +Commit Sequence of Global Transactions +-------------------------------- + +We employee two-phase commit protocol to achieve commit among all foreign +servers atomically. The sequence of distributed transaction commit consisnts +of the following four steps: + +1. Foriegn Server Registration +During executor node initialization, accessed foreign servers are registered +to the list FdwXactParticipant, which is maintained by PostgreSQL's the global +transaction manager (GTM), as a distributed transaction participant The +registered foreign transactions are tracked until the end of transaction. + +2. Pre-Commit phase (1st phase of two-phase commit) +we record the corresponding WAL indicating that the foreign server is involved +with the current transaction before doing PREPARE all foreign transactions. +Thus, in case we loose connectivity to the foreign server or crash ourselves, +we will remember that we might have prepared tranascation on the foreign +server, and try to resolve it when connectivity is restored or after crash +recovery. + +The two-phase commit is required only if the transaction modified two or more +servers including the local node. + +After that we prepare all foreign transactions by calling +PrepareForeignTransaction() API. If we failed on any of them we change to +rollback, therefore at this time some participants might be prepared whereas +some are not prepared. The former foreign transactions are resolved by +the resolver process asynchronusly or can be resolved using by +pg_resolve_foreign_xact() manually, and the latter ends transaction +in one-phase by calling RollbackForeignTransaction() API. + +3. Commit locally +Once we've prepared all of them, commit the transaction locally. + +4. Post-Commit Phase (2nd phase of two-phase commit) +The steps so far are done by the backend process committing the transaction but +this resolution step(commit or rollback) is done by the foreign transaction +resolver process. + + +Identifying Foreign Transactions In GTM +--------------------------------------- + +To identify foreign transaction participants (as well as FdwXact entries) there +are two ways: using {server OID, user OID} and using user mapping OID. The same +is true for FDWs to identify the connections (and transactions upon) to the +foreign server. We need to consider the case where the way to identify the +transactions is not matched between GTM and FDWs, because the problem might occur +when the user modifies the same foreign server by different roles within the +transaction. For example, consider the following execution: + +BEGIN; +SET ROLE user_A; +INSERT INTO ft1 VALUES (1); +SET ROLE user_B; +INSERT INTO ft1 VALUES (1); +COMMIT; + +For example, suppose that an FDW identifies the connection by {server OID, user OID} +and GTM identifies the transactions by user mapping OID, and user_A and user_B use +the public user mapping to connect server_X. In the FDW, there are two +connections: {user_A, sever_X} and {user_B, server_X}, and therefore opens two +transactions on each connection, while GTM has only one FdwXact entry because the two +connections refer to the same user mapping OID. As a result, at the end of the +transaction, GTM ends only one foreign transaction, leaving another one. + +On the other hand, suppose that an FDW identifies the connection by user mapping OID +and GTM does that by {server OID, user OID}, the FDW uses only one connection and opens +a transaction since both users refer to the same user mapping OID (we expect FDWs +not to register the foreign transaction when not starting a new transaction on the +foreign server). Since GTM also has one entry it can end the foreign transaciton +properly. The downside would be that the user OID of FdwXact (i.g., FdwXact->userid) +is the user who registered the foreign transaction for the first time, necessarily +not the user who executed COMMIT. For example in the above case, FdwXact->userid +will be user_A, not user_B. But it’s not big problem in practice. + +Therefore, in fdwxact.c, we identify the foreign transaction by +{server OID, user OID}. + +Foreign Transactions Status +---------------------------- + +Every foreign transaction has an FdwXact entry. When preparing a foreign +transaction a FdwXact entry of which status starts from FDWXACT_STATUS_PREPARING +are created with WAL logging. The status changes to FDWXACT_STATUS_PREPARED +after the foreign transaction is prepared. And the status changes to +FDWXACT_STATUS_COMMITTING and FDWXACT_STATUS_ABORTING before committing and +aborting respectively. FdwXact entry is removed with WAL logging after resolved. + +FdwXact entries recovered during the recovery are marked as in-doubt if the +corresponding local transaction is not prepared transaction. The initial +status for those entries is FDWXACT_STATUS_PREPARED if they are recovered +from WAL. Because we WAL logs only when preparing the foreign transaction we +cannot know the exact fate of the foreign transaction from the recovery. + +The foreign transaction status transition is illustrated by the following +graph describing the FdwXact->status: + + +----------------------------------------------------+ + | INVALID | + +----------------------------------------------------+ + | | | + | v | + | +---------------------+ | + (*1) | PREPARING | (*1) + | +---------------------+ | + | | | + v v v + +----------------------------------------------------+ + | PREPARED | + +----------------------------------------------------+ + | | + v v + +--------------------+ +--------------------+ + | COMMITTING | | ABORTING | + +--------------------+ +--------------------+ + | | + v v + +----------------------------------------------------+ + | END | + +----------------------------------------------------+ + +(*1) Paths for recovered FdwXact entries -- 2.27.0