WHPG 7.6 release

docs/whpg/7x/install_guide/minor_upgrade.md

@@ -131,6 +131,46 @@ Perform a final cleanup of the running database instance.
 
 Perform the following steps as the `gpadmin` user. When upgrading, the package manager will automatically remove old files in `/usr/local/greenplum-db-<version>` and install new files to `/usr/edb/whpg7`.
 
+::: warning Upgrading to 7.6.0: Behavior changes to review
+WarehousePG 7.6.0 changes the following behavior. Review each item before you upgrade.
+
+-   `pg_file_write()`, `pg_file_rename()`, `pg_file_unlink()`, and `pg_logdir_ls()` now require superuser privileges or membership in `pg_write_server_files` (`pg_read_server_files` for `pg_logdir_ls()`). This closes a critical privilege-escalation issue that let any role run these functions. Run the [REVOKE/GRANT block below](#applying-the-file-function-privilege-fix) in every database of an upgraded cluster, since a binary upgrade doesn't change privileges already stored in the catalog.
+-   `ALTER TABLE ... SET`/`RESET (compresstype, compresslevel, blocksize)` on append-optimized column-oriented (AOCO) tables now re-encodes every column that has no explicit `ENCODING` of its own, so the statement changes on-disk data where it previously only changed the catalog. A column whose explicit `ENCODING` happens to match the table default can't be distinguished from an implicit one, and gets re-encoded too.
+-   Plain-text output from [`pg_dump`](../ref_guide/utility_guide/reference/pg_dump.md), [`pg_dumpall`](../ref_guide/utility_guide/reference/pg_dumpall.md), and [`pg_restore`](../ref_guide/utility_guide/reference/pg_restore.md) now begins with `\restrict <key>` and ends with `\unrestrict <key>`. Tooling that parses dump output needs adjusting for the new markers. Both the dumper and the `psql` used to restore the dump need to be on this version or later for the protection to apply. A dump from 7.6.0 restored with an older `psql` gets `invalid command \restrict` on the first line, and stops there if `ON_ERROR_STOP` is set.
+-   [`CREATE OPERATOR`](../ref_guide/sql_commands/CREATE_OPERATOR.md) and [`ALTER OPERATOR`](../ref_guide/sql_commands/ALTER_OPERATOR.md) now require superuser privileges to attach a non-built-in selectivity estimator.
+-   SQL-level calls to functions that take or return the `internal` type, and casts to and from `internal`, are rejected.
+-   [`psql`](../ref_guide/utility_guide/reference/psql.md) now consumes in-line [`COPY ... FROM STDIN`](../ref_guide/sql_commands/COPY.md) data even when the `COPY` command fails. Scripts must terminate in-line data with `\.`.
+-   `pgcrypto` [PGP encryption](../security_guide/encryption.md#encpgp) now returns an error when cipher initialization fails, instead of writing unencrypted data. Use the new `ignore-cipher-failure=1` decryption option, passed in the `options` argument of `pgp_sym_decrypt()` and `pgp_pub_decrypt()`, to read back data written by an affected build.
+-   The resource group [`CPUSET`](../admin_guide/performance/wlmgmt/workload_mgmt_resgroups.md#cpuset) setting now parses in the documented order, `<coordinator_cores>;<segment_cores>`. Clusters that swapped the two halves of the value to compensate for the previous reversed behavior need to swap them back.
+-   [`gp_toolkit`](../ref_guide/gp_toolkit.md) is now version 1.9. Run `ALTER EXTENSION gp_toolkit UPDATE` to pick up the corrected [`gp_toolkit.gp_workfile_entries`](../ref_guide/gp_toolkit.md#topic33) view.
+-   A hot standby that receives no queries keeps `pg_distributedlog` segment files that the primary has already removed (about 1 MB per 131072 transactions of that segment) until its next query, which then drops them all.
+:::
+
+### Applying the file function privilege fix
+
+If you can't upgrade to `7.6.0-WHPG` immediately, run the following as a superuser in every database of the cluster, including `postgres` and `template1`. It takes effect immediately and needs no restart. WarehousePG dispatches the statements to all segments automatically.
+
+```sql
+REVOKE EXECUTE ON FUNCTION pg_file_write(text,text,boolean) FROM public;
+REVOKE EXECUTE ON FUNCTION pg_file_rename(text,text,text) FROM public;
+REVOKE EXECUTE ON FUNCTION pg_file_unlink(text) FROM public;
+REVOKE EXECUTE ON FUNCTION pg_logdir_ls() FROM public;
+-- keep the intended administrative access
+GRANT EXECUTE ON FUNCTION pg_file_write(text,text,boolean) TO pg_write_server_files;
+GRANT EXECUTE ON FUNCTION pg_file_rename(text,text,text) TO pg_write_server_files;
+GRANT EXECUTE ON FUNCTION pg_file_unlink(text) TO pg_write_server_files;
+GRANT EXECUTE ON FUNCTION pg_logdir_ls() TO pg_read_server_files;
+```
+
+Run the same block after upgrading an existing cluster, too. A binary upgrade doesn't change privileges already stored in the catalog, so without it, `public` keeps execute access even though the new binaries already reject the call. Confirm every row shows `f`:
+
+```sql
+select proname, has_function_privilege('public', oid, 'execute') as public_can_execute
+from pg_proc where proname in ('pg_file_write', 'pg_file_rename', 'pg_file_unlink', 'pg_logdir_ls');
+```
+
+Roles that legitimately need these functions must be superusers or hold membership in `pg_write_server_files` (or `pg_read_server_files` for `pg_logdir_ls()`). `gp_toolkit.gp_move_orphaned_files()`, the one in-tree caller, keeps working for superusers.
+
 ::: warning Upgrading to 7.5.0: Python 3.11 requirement
 WarehousePG 7.5.0 upgrades PL/Python to Python 3.11. The PL/Python interpreter now links against `/usr/bin/python3.11` and requires the following packages on every node:
 

docs/whpg/7x/install_guide/platform-requirements.md

@@ -15,9 +15,17 @@ WarehousePG 7 runs on the following operating system platforms:
 -   Oracle Linux 64-bit 8.7 or later, using the Red Hat Compatible Kernel (RHCK)
 -   Rocky Linux 8.7 or later
 
-> **Note** If you use endpoint security software on your WarehousePG hosts, it may affect your database performance and stability. See [About Endpoint Security Sofware](../security_guide/index.md#endpoint_security) for more information.
+::: info Note
+Starting with WarehousePG 7.6.0, WarehousePG also supports the ppc64le (64-bit little-endian PowerPC) architecture.
+:::
 
-> **Caution** A kernel issue in Red Hat Enterprise Linux 8.5 and 8.6 can cause I/O freezes and synchronization problems with XFS filesystems. This issue is fixed in RHEL 8.7. See [RHEL8: xfs_buf deadlock between inode deletion and block allocation](https://access.redhat.com/solutions/6984334).
+::: info Note
+If you use endpoint security software on your WarehousePG hosts, it may affect your database performance and stability. See [About Endpoint Security Software](../security_guide/index.md#endpoint_security) for more information.
+:::
+
+::: warning
+A kernel issue in Red Hat Enterprise Linux 8.5 and 8.6 can cause I/O freezes and synchronization problems with XFS filesystems. This issue is fixed in RHEL 8.7. See [RHEL8: xfs_buf deadlock between inode deletion and block allocation](https://access.redhat.com/solutions/6984334).
+:::
 
 WarehousePG server supports TLS version 1.2 and TLS version 1.3 on RHEL/CentOS systems.
 

docs/whpg/7x/ref_guide/config_params/guc-list.md

@@ -4297,6 +4297,30 @@ The value of [wal_sender_timeout](#replication_timeout) controls the time that t
 | ----------------------- | ------- | -------------------------------------- |
 | integer 0- INT_MAX/1000 | 10 sec  | coordinator, system, reload, superuser |
 
+<a id="whpg_dispatch_topology_file"></a>
+
+## whpg_dispatch_topology_file
+
+The path of the dispatch topology file. An empty value, the default, disables it.
+
+This parameter serves hot-standby dispatchers only. When set, dispatch resolves every content's address from this file instead of selecting a row from [gp_segment_configuration](../system_catalogs/system_catalogs_definitions/gp_segment_configuration.md), with fail-closed validation and a `dbid` handshake. On any other node, the component-table build refuses to start. Each row in the file must carry the node's own identity, the `gp_dbid` value from `internal.auto.conf`. A relative path is relative to the data directory.
+
+The topology file's state is exposed through the read-only [whpg_dispatch_topology_state](#whpg_dispatch_topology_state) parameter.
+
+| Value Range | Default | Set Classifications    |
+| ----------- | ------- | ----------------------- |
+| string      |         | local, system, reload   |
+
+<a id="whpg_dispatch_topology_state"></a>
+
+## whpg_dispatch_topology_state
+
+Shows the state of the [whpg_dispatch_topology_file](#whpg_dispatch_topology_file) parameter.
+
+| Value Range               | Default  | Set Classifications |
+| -------------------------- | -------- | -------------------- |
+| inactive, active, error    | inactive | read only             |
+
 <a id="work_mem"></a>
 
 ## work_mem

docs/whpg/7x/ref_guide/config_params/guc_category-list.md

@@ -593,6 +593,7 @@ The parameters in this topic control the configuration of the WarehousePG cluste
 -   [gp_enable_direct_dispatch](guc-list.md#gp_enable_direct_dispatch)
 -   [gp_segment_connect_timeout](guc-list.md#gp_segment_connect_timeout)
 -   [gp_set_proc_affinity](guc-list.md#gp_set_proc_affinity)
+-   [whpg_dispatch_topology_file](guc-list.md#whpg_dispatch_topology_file)
 
 <a id="topic52"></a>
 
@@ -623,6 +624,7 @@ The parameters in this topic control the configuration of the WarehousePG cluste
 -   [gp_session_id](guc-list.md#gp_session_id)
 -   [gp_server_version](guc-list.md#gp_server_version)
 -   [gp_server_version_num](guc-list.md#gp_server_version_num)
+-   [whpg_dispatch_topology_state](guc-list.md#whpg_dispatch_topology_state)
 
 <a id="topic55"></a>
 

docs/whpg/7x/ref_guide/sql_commands/ALTER_OPERATOR.md

@@ -32,6 +32,8 @@ ALTER OPERATOR name ( { <left_type> | NONE } , { <right_type> | NONE } )
 
 You must own the operator to use `ALTER OPERATOR`. To alter the owner, you must also be a direct or indirect member of the new owning role, and that role must have `CREATE` privilege on the operator's schema. (These restrictions enforce that altering the owner does not do anything you could not do by dropping and recreating the operator. However, a superuser can alter ownership of any operator anyway.)
 
+You must be a superuser to set a `RESTRICT` or `JOIN` estimator function that isn't one of the built-in selectivity estimators.
+
 <a id="section4"></a>
 
 ## Parameters

docs/whpg/7x/ref_guide/sql_commands/CREATE_OPERATOR.md

@@ -52,6 +52,8 @@ The other clauses specify optional operator optimization clauses. Their meaning
 
 To be able to create an operator, you must have `USAGE` privilege on the argument types and the return type, as well as `EXECUTE` privilege on the underlying function. If a commutator or negator operator is specified, you must own these operators.
 
+You must be a superuser to specify a `RESTRICT` or `JOIN` estimator function that isn't one of the built-in selectivity estimators.
+
 <a id="section4"></a>
 
 ## Parameters

docs/whpg/7x/ref_guide/sql_commands/CREATE_TABLE_AS.md

@@ -22,7 +22,8 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXI
         [ WITH [ NO ] DATA ]
         [ DISTRIBUTED BY ( <column> [<opclass>] [, ... ] ) 
            | DISTRIBUTED RANDOMLY
-           | DISTRIBUTED REPLICATED ]
+           | DISTRIBUTED REPLICATED
+           | DISTRIBUTED COORDINATOR ONLY ]
 ```
 
 <a id="section3"></a>
@@ -83,6 +84,7 @@ A [SELECT](SELECT.md), [TABLE](SELECT.md#table-command), or [VALUES](VALUES.md)
 DISTRIBUTED BY ( column \[opclass] \[, ... ] )
 DISTRIBUTED RANDOMLY
 DISTRIBUTED REPLICATED
+DISTRIBUTED COORDINATOR ONLY
 Used to declare the WarehousePG distribution policy for the table. Refer to [CREATE TABLE](CREATE_TABLE.md) for details.
 
 <a id="section5"></a>
@@ -93,7 +95,7 @@ This command is functionally similar to [SELECT INTO](SELECT_INTO.md), but it is
 
 `CREATE TABLE AS` can be used for fast data loading from external table data sources. See [CREATE EXTERNAL TABLE](CREATE_EXTERNAL_TABLE.md).
 
-`CREATE TABLE AS` does not support setting distribution to `COORDINATOR ONLY`.
+`CREATE TABLE AS` supports coordinator-only distribution via the clause `DISTRIBUTED COORDINATOR ONLY`. Refer to [CREATE TABLE](CREATE_TABLE.md#section3) for restrictions on coordinator-only distribution.
 
 <a id="section6"></a>
 

docs/whpg/7x/ref_guide/system_catalogs/system_catalogs_definitions/pg_stat_operations.md

@@ -3,11 +3,11 @@ title: pg_stat_operations
 
 ---
 
-The view `pg_stat_operations` shows details about the last operation performed on a database object (such as a table, index, view or database) or a global object (such as a role).
+The view `pg_stat_operations` shows details about the last operation performed on a database object (such as a table, index, view or database) or a global object (such as a role, resource queue, or resource group).
 
 | column       | type        | references | description                                                                                                                                                                                                                                                                             |
 | ------------ | ----------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `classname`  | text        |            | The name of the system table in the `pg_catalog` schema where the record about this object is stored (`pg_class`=relations, `pg_database`=databases,`pg_namespace`=schemas, `pg_authid`=roles)                                                                                          |
+| `classname`  | text        |            | The name of the system table in the `pg_catalog` schema where the record about this object is stored (`pg_class`=relations, `pg_database`=databases, `pg_namespace`=schemas, `pg_authid`=roles, `pg_tablespace`=tablespaces, `pg_resqueue`=resource queues, `pg_resgroup`=resource groups, `pg_resgroupcapability`=resource group capabilities)                                                                                          |
 | `objname`    | name        |            | The name of the object.                                                                                                                                                                                                                                                                 |
 | `objid`      | oid         |            | The OID of the object.                                                                                                                                                                                                                                                                  |
 | `schemaname` | name        |            | The name of the schema where the object resides.                                                                                                                                                                                                                                        |

docs/whpg/7x/ref_guide/utility_guide/reference/pg_dump.md

@@ -36,6 +36,10 @@ Since `pg_dump` is compatible with regular PostgreSQL, it can be used to migrate
 
 When used with one of the archive file formats and combined with `pg_restore`, `pg_dump` provides a flexible archival and transfer mechanism. `pg_dump` can be used to backup an entire database, then `pg_restore`can be used to examine the archive and/or select which parts of the database are to be restored. The most flexible output file formats are the custom format (`-Fc`) and the directory format (`-Fd`). They allow for selection and reordering of all archived items, support parallel restoration, and are compressed by default. The directory format is the only format that supports parallel dumps.
 
+::: warning 
+Restoring a dump causes the destination to run arbitrary code of the source superuser's choice. Partial dumps and partial restores don't limit that risk. If you don't trust the source database's superusers, inspect the dumped SQL statements before restoring them. You can inspect a non-plain-text dump with `pg_restore`'s `--file` option. The client running the dump and restore doesn't need to trust the source or destination superusers.
+:::
+
 <a id="section4"></a>
 
 ## Options
@@ -233,6 +237,11 @@ Add `ON CONFLICT DO NOTHING` to `INSERT` commands. This option is not valid unle
 \--quote-all-identifiers
 Force quoting of all identifiers. This option is recommended when dumping a database from a server whose WarehousePG major version is different from `pg_dump`'s, or when the output is intended to be loaded into a server of a different major version. By default, `pg_dump` quotes only identifiers that are reserved words in its own major version. This sometimes results in compatibility issues when dealing with servers of other versions that may have slightly different sets of reserved words. Using `--quote-all-identifiers` prevents such issues, at the price of a harder-to-read dump script.
 
+\--restrict-key=\\&lt;restrict_key>
+Use the provided string as the `psql` `\restrict` key in the dump output. You can only specify this for plain-text dumps, meaning when `--format` is set to `plain` or omitted. If you don't specify a restrict key, `pg_dump` generates a random one. Keys can contain only alphanumeric characters.
+
+This option is mainly for testing and other scenarios that need repeatable output, such as comparing dump files. Avoid it for general use, since a malicious server with advance knowledge of the key could inject code that runs on the machine that runs `psql` with the dump output.
+
 \--rows-per-insert=\\&lt;nrows>
 Dump data as `INSERT` commands (rather than `COPY`). Controls the maximum number of rows per `INSERT` command. The value specified must be a number greater than zero. Any error during restoring will cause only rows that are part of the problematic `INSERT` to be lost, rather than the entire table contents.
 

docs/whpg/7x/ref_guide/utility_guide/reference/pg_dumpall.md

@@ -33,6 +33,10 @@ The SQL script will be written to the standard output. Use the `[-f | --file]` o
 
 `pg_dumpall` needs to connect several times to the WarehousePG coordinator server (once per database). If you use password authentication it is likely to ask for a password each time. It is convenient to have a `~/.pgpass` file in such cases.
 
+::: warning 
+Restoring a dump causes the destination to run arbitrary code of the source superuser's choice. Partial dumps and partial restores don't limit that risk. If you don't trust the source database's superusers, inspect the dumped SQL statements before restoring them. The client running the dump and restore doesn't need to trust the source or destination superusers.
+:::
+
 <a id="section4"></a>
 
 ## Options
@@ -146,6 +150,11 @@ Dump resource queue definitions.
 \--resource-groups
 Dump resource group definitions.
 
+\--restrict-key=restrict_key
+Use the provided string as the `psql` `\restrict` key in the dump output. If you don't specify a restrict key, `pg_dumpall` generates a random one. Keys can contain only alphanumeric characters.
+
+This option is mainly for testing and other scenarios that need repeatable output, such as comparing dump files. Avoid it for general use, since a malicious server with advance knowledge of the key could inject code that runs on the machine that runs `psql` with the dump output.
+
 \--rows-per-insert=nrows
 Dump data as `INSERT` commands (rather than `COPY)`. Controls the maximum number of rows per `INSERT` command. The value specified must be a number greater than zero. Any error during restoring will cause only rows that are part of the problematic `INSERT` to be lost, rather than the entire table contents.