Write Parquet NUMERIC with precision up to 76 and name the array column in overflow messages

CHANGELOG.md

@@ -342,6 +342,19 @@ Two things deliberately did **not** move:
   compatibility fallback no longer exists in the Hive 4.x thrift
   bindings.
 
+### Parquet DECIMAL precision up to 76
+
+- PXF can now write a `NUMERIC(<precision>, <scale>)` column with a precision
+  of up to 76 to Parquet; previously any precision above 38 was rejected. The
+  column is written as a Parquet `DECIMAL` on a `FIXED_LEN_BYTE_ARRAY` wide
+  enough for the precision (32 bytes at 76), which Arrow-based readers such as
+  PyArrow read as `decimal256`. Values are no longer limited by `HiveDecimal`
+  for these columns, and `pxf.parquet.write.decimal.overflow` applies to them
+  as before. Precision up to 38, and the `DECIMAL(38,18)` mapping of a
+  `NUMERIC` column without a precision, are unchanged. Spark, Hive and other
+  readers limited to precision 38 cannot read a column written with a greater
+  precision. ORC writes are unchanged.
+
 ### Bug Fixes
 
 - PTT-1850: external tables that are partitions of a partitioned table (or that
@@ -364,6 +377,11 @@ Two things deliberately did **not** move:
   external-table regression suite's `pxf` write test passes on every supported
   WarehousePG major and the suite is no longer order-dependent. Example code
   only; real connectors were not affected.
+- When a `NUMERIC[]` element did not fit the Parquet decimal type on write, the
+  overflow error (or, with `pxf.parquet.write.decimal.overflow=ignore`, the
+  warning) named the table's first column instead of the array column. The
+  value handling itself was correct; only the column name in the message is
+  fixed.
 
 ### Compatibility notes
 

automation/sqlrepo/features/parquet/decimal/numeric_array_scale/expected/query01.ans

@@ -0,0 +1,38 @@
+-- start_ignore
+-- end_ignore
+-- @description query01 for writing numeric arrays to Parquet. The scale comes from the writable table column: NUMERIC[] is written as DECIMAL(38,18) and rounds values with more than 18 decimal places, NUMERIC(38,20)[] keeps 20 decimal places, and NUMERIC(76,61)[] and TEXT[] keep every digit.
+SELECT e.id, e.v[1] AS original_value, r.v[1] AS parquet_value, e.v[1] = r.v[1] AS same_value
+FROM numeric_array_scale e JOIN pxf_parquet_read_numeric_array_undefined_precision r USING (id) ORDER BY id;
+ id |                         original_value                          |    parquet_value     | same_value
+----+-----------------------------------------------------------------+----------------------+------------
+  1 |                                            0.123456789012345678 | 0.123456789012345678 | t
+  2 |                                           0.1234567890123456789 | 0.123456789012345679 | f
+  3 | 0.0000000000000000000000000000000000000000000000000000000000001 | 0.000000000000000000 | f
+(3 rows)
+
+SELECT e.id, e.v[1] AS original_value, r.v[1] AS parquet_value, e.v[1] = r.v[1] AS same_value
+FROM numeric_array_scale e JOIN pxf_parquet_read_numeric_array_38_20 r USING (id) ORDER BY id;
+ id |                         original_value                          |     parquet_value      | same_value
+----+-----------------------------------------------------------------+------------------------+------------
+  1 |                                            0.123456789012345678 | 0.12345678901234567800 | t
+  2 |                                           0.1234567890123456789 | 0.12345678901234567890 | t
+  3 | 0.0000000000000000000000000000000000000000000000000000000000001 | 0.00000000000000000000 | f
+(3 rows)
+
+SELECT e.id, e.v[1] AS original_value, r.v[1] AS parquet_value, e.v[1] = r.v[1] AS same_value
+FROM numeric_array_scale e JOIN pxf_parquet_read_numeric_array_76_61 r USING (id) ORDER BY id;
+ id |                         original_value                          |                          parquet_value                          | same_value
+----+-----------------------------------------------------------------+-----------------------------------------------------------------+------------
+  1 |                                            0.123456789012345678 | 0.1234567890123456780000000000000000000000000000000000000000000 | t
+  2 |                                           0.1234567890123456789 | 0.1234567890123456789000000000000000000000000000000000000000000 | t
+  3 | 0.0000000000000000000000000000000000000000000000000000000000001 | 0.0000000000000000000000000000000000000000000000000000000000001 | t
+(3 rows)
+
+SELECT e.id, r.v[1]::numeric AS parquet_value, e.v[1] = r.v[1]::numeric AS same_value
+FROM numeric_array_scale e JOIN pxf_parquet_read_numeric_array_text r USING (id) ORDER BY id;
+ id |                          parquet_value                          | same_value
+----+-----------------------------------------------------------------+------------
+  1 |                                            0.123456789012345678 | t
+  2 |                                           0.1234567890123456789 | t
+  3 | 0.0000000000000000000000000000000000000000000000000000000000001 | t
+(3 rows)

automation/sqlrepo/features/parquet/decimal/numeric_array_scale/sql/query01.sql

@@ -0,0 +1,12 @@
+-- @description query01 for writing numeric arrays to Parquet. The scale comes from the writable table column: NUMERIC[] is written as DECIMAL(38,18) and rounds values with more than 18 decimal places, NUMERIC(38,20)[] keeps 20 decimal places, and NUMERIC(76,61)[] and TEXT[] keep every digit.
+SELECT e.id, e.v[1] AS original_value, r.v[1] AS parquet_value, e.v[1] = r.v[1] AS same_value
+FROM numeric_array_scale e JOIN pxf_parquet_read_numeric_array_undefined_precision r USING (id) ORDER BY id;
+
+SELECT e.id, e.v[1] AS original_value, r.v[1] AS parquet_value, e.v[1] = r.v[1] AS same_value
+FROM numeric_array_scale e JOIN pxf_parquet_read_numeric_array_38_20 r USING (id) ORDER BY id;
+
+SELECT e.id, e.v[1] AS original_value, r.v[1] AS parquet_value, e.v[1] = r.v[1] AS same_value
+FROM numeric_array_scale e JOIN pxf_parquet_read_numeric_array_76_61 r USING (id) ORDER BY id;
+
+SELECT e.id, r.v[1]::numeric AS parquet_value, e.v[1] = r.v[1]::numeric AS same_value
+FROM numeric_array_scale e JOIN pxf_parquet_read_numeric_array_text r USING (id) ORDER BY id;

automation/sqlrepo/features/parquet/decimal/numeric_max_precision/expected/query01.ans

@@ -0,0 +1,25 @@
+-- start_ignore
+-- end_ignore
+-- @description query01 for writing numeric with precision above 38 to Parquet. NUMERIC(76,61) and NUMERIC(76,61)[] keep every digit, and precision above 76 is rejected.
+SELECT e.id, r.d, e.d = r.d AS same_d, e.a = r.a AS same_a
+FROM numeric_max_precision e JOIN pxf_parquet_read_numeric_max_precision r USING (id) ORDER BY id;
+ id |                                       d                                        | same_d | same_a
+----+--------------------------------------------------------------------------------+--------+--------
+  1 |                0.0000000000000000000000000000000000000000000000000000000000001 | t      | t
+  2 | -123456789012345.9999999999999999999999999999999999999999999999999999999999999 | t      | t
+  3 |                                                                                |        | t
+(3 rows)
+
+SELECT id, unnest(a) AS a FROM pxf_parquet_read_numeric_max_precision ORDER BY id, a;
+ id |                                       a
+----+-------------------------------------------------------------------------------
+  1 |               0.0000000000000000000000000000000000000000000000000000000000001
+  1 |               0.1234567890123456789000000000000000000000000000000000000000000
+  2 |              -1.0000000000000000000000000000000000000000000000000000000000000
+  2 | 999999999999999.9999999999999999999999999999999999999999999999999999999999999
+  3 |               0.0000000000000000000000000000000000000000000000000000000000000
+  3 |
+(6 rows)
+
+INSERT INTO pxf_parquet_write_numeric_precision_77 SELECT id, d FROM numeric_max_precision;
+ERROR:  PXF server error : Column d is defined as NUMERIC with precision 77 which exceeds the maximum supported precision 76.

automation/sqlrepo/features/parquet/decimal/numeric_max_precision/sql/query01.sql

@@ -0,0 +1,7 @@
+-- @description query01 for writing numeric with precision above 38 to Parquet. NUMERIC(76,61) and NUMERIC(76,61)[] keep every digit, and precision above 76 is rejected.
+SELECT e.id, r.d, e.d = r.d AS same_d, e.a = r.a AS same_a
+FROM numeric_max_precision e JOIN pxf_parquet_read_numeric_max_precision r USING (id) ORDER BY id;
+
+SELECT id, unnest(a) AS a FROM pxf_parquet_read_numeric_max_precision ORDER BY id, a;
+
+INSERT INTO pxf_parquet_write_numeric_precision_77 SELECT id, d FROM numeric_max_precision;

automation/sqlrepo/features/parquet/decimal/numeric_with_large_precision/expected/query01.ans

@@ -1,5 +1,5 @@
 -- start_ignore
 -- end_ignore
--- @description query01 for writing defined precision numeric with pxf.parquet.write.decimal.overflow = round. When try to write a numeric with precision > 38, an error will be thrown.
+-- @description query01 for writing defined precision numeric with pxf.parquet.write.decimal.overflow = round. When try to write a numeric with precision > 76, an error will be thrown.
 INSERT INTO parquet_write_defined_large_precision_numeric SELECT * FROM numeric_precision;
-ERROR:  PXF server error : Column a is defined as NUMERIC with precision 90 which exceeds the maximum supported precision 38.
+ERROR:  PXF server error : Column a is defined as NUMERIC with precision 90 which exceeds the maximum supported precision 76.

automation/sqlrepo/features/parquet/decimal/numeric_with_large_precision/sql/query01.sql

@@ -1,2 +1,2 @@
--- @description query01 for writing defined precision numeric with pxf.parquet.write.decimal.overflow = round. When try to write a numeric with precision > 38, an error will be thrown.
+-- @description query01 for writing defined precision numeric with pxf.parquet.write.decimal.overflow = round. When try to write a numeric with precision > 76, an error will be thrown.
 INSERT INTO parquet_write_defined_large_precision_numeric SELECT * FROM numeric_precision;

automation/src/test/java/org/greenplum/pxf/automation/features/parquet/ParquetWriteTest.java

@@ -28,6 +28,8 @@
 public class ParquetWriteTest extends BaseWritableFeature {
     private static final String NUMERIC_TABLE = "numeric_precision";
     private static final String NUMERIC_UNDEFINED_PRECISION_TABLE = "numeric_undefined_precision";
+    private static final String NUMERIC_MAX_PRECISION_TABLE = "numeric_max_precision";
+    private static final String NUMERIC_ARRAY_SCALE_TABLE = "numeric_array_scale";
     private static final String PXF_PARQUET_PRIMITIVE_TABLE = "pxf_parquet_primitive_types";
     private static final String PXF_PARQUET_LIST_TYPES = "pxf_parquet_list_types";
     private static final String PXF_PARQUET_TIMESTAMP_LIST_TYPES = "pxf_parquet_timestamp_list_types";
@@ -70,10 +72,19 @@ public class ParquetWriteTest extends BaseWritableFeature {
             "a             DECIMAL(90, 38)",
             "b             DECIMAL(100, 50)"
     };
+    private static final String[] NUMERIC_MAX_PRECISION_COLUMNS = new String[]{
+            "id   INTEGER",
+            "d    NUMERIC(76,61)",
+            "a    NUMERIC(76,61)[]"
+    };
     private static final String[] UNDEFINED_PRECISION_NUMERIC = new String[]{
             "description   text",
             "value         numeric"
     };
+    private static final String[] NUMERIC_ARRAY_SCALE_COLUMNS = new String[]{
+            "id   INTEGER",
+            "v    NUMERIC[]"
+    };
 
     // HIVE Parquet array vectorization read currently doesn't support TIMESTAMP and INTERVAL_DAY_TIME
     //https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
@@ -272,6 +283,55 @@ public void parquetWriteUndefinedPrecisionNumericWithScaleOverflow() throws Exce
         runSqlTest("features/parquet/decimal/numeric_undefined_precision_large_scale");
     }
 
+    // Numeric precision above 38 up to 76 is written as a wider Parquet DECIMAL without losing digits,
+    // and precision above 76 is rejected
+    @Test(groups = {"features", "gpdb", "security", "hcfs"})
+    public void parquetWriteNumericWithMaxPrecision() throws Exception {
+        Table numericTable = new Table(NUMERIC_MAX_PRECISION_TABLE, NUMERIC_MAX_PRECISION_COLUMNS);
+        numericTable.setDistributionFields(new String[]{"id"});
+        gpdb.createTableAndVerify(numericTable);
+        gpdb.runQuery("INSERT INTO " + NUMERIC_MAX_PRECISION_TABLE + " VALUES " +
+                "(1, ('0.' || repeat('0', 60) || '1')::numeric, ARRAY[0.1234567890123456789, ('0.' || repeat('0', 60) || '1')::numeric]), " +
+                "(2, ('-123456789012345.' || repeat('9', 61))::numeric, ARRAY[('999999999999999.' || repeat('9', 61))::numeric, -1]), " +
+                "(3, NULL, ARRAY[NULL::numeric, 0])");
+
+        String fileName = "parquet_write_numeric_max_precision";
+        prepareWritableExternalTable("pxf_parquet_write_numeric_max_precision",
+                NUMERIC_MAX_PRECISION_COLUMNS, hdfsPath + fileName, null);
+        gpdb.copyData(NUMERIC_MAX_PRECISION_TABLE, writableExTable);
+
+        waitForAsyncWriteToSucceedOnHCFS(fileName);
+
+        prepareReadableExternalTable("pxf_parquet_read_numeric_max_precision",
+                NUMERIC_MAX_PRECISION_COLUMNS, hdfsPath + fileName);
+        prepareWritableExternalTable("pxf_parquet_write_numeric_precision_77",
+                new String[]{"id INTEGER", "d NUMERIC(77,61)"}, hdfsPath + "parquet_write_numeric_precision_77", null);
+
+        runSqlTest("features/parquet/decimal/numeric_max_precision");
+    }
+
+    // Numeric array scale comes from the writable table column: undefined precision falls back to DECIMAL(38,18)
+    // and rounds values with more than 18 decimal places, a defined scale is kept, and values that do not fit in
+    // 38 digits survive the round trip when written as NUMERIC(76,61)[] or as text[].
+    @Test(groups = {"features", "gpdb", "security", "hcfs"})
+    public void parquetWriteNumericArrayScale() throws Exception {
+        Table numericArrayTable = new Table(NUMERIC_ARRAY_SCALE_TABLE, NUMERIC_ARRAY_SCALE_COLUMNS);
+        numericArrayTable.setDistributionFields(new String[]{"id"});
+        gpdb.createTableAndVerify(numericArrayTable);
+        // 18, 19 and 61 decimal places
+        gpdb.runQuery("INSERT INTO " + NUMERIC_ARRAY_SCALE_TABLE + " VALUES " +
+                "(1, ARRAY[0.123456789012345678]), " +
+                "(2, ARRAY[0.1234567890123456789]), " +
+                "(3, ARRAY[('0.' || repeat('0', 60) || '1')::numeric])");
+
+        writeAndReadNumericArray("undefined_precision", "NUMERIC[]", "NUMERIC[]", null);
+        writeAndReadNumericArray("38_20", "NUMERIC(38,20)[]", "NUMERIC[]", null);
+        writeAndReadNumericArray("76_61", "NUMERIC(76,61)[]", "NUMERIC[]", null);
+        writeAndReadNumericArray("text", "TEXT[]", "TEXT[]", new String[]{"id", "v::text[]"});
+
+        runSqlTest("features/parquet/decimal/numeric_array_scale");
+    }
+
     @Test(groups = {"features", "gpdb", "security", "hcfs"})
     public void parquetWriteLists() throws Exception {
         String writeTableName = "pxf_parquet_write_list";
@@ -570,6 +630,18 @@ private boolean checkHiveVersionForDateSupport(Hive hive) throws Exception {
         }
     }
 
+    private void writeAndReadNumericArray(String suffix, String writeType, String readType, String[] sourceColumns) throws Exception {
+        String fileName = "parquet_write_numeric_array_" + suffix;
+        prepareWritableExternalTable("pxf_parquet_write_numeric_array_" + suffix,
+                new String[]{"id INTEGER", "v " + writeType}, hdfsPath + fileName, null);
+        gpdb.copyData(NUMERIC_ARRAY_SCALE_TABLE, writableExTable, sourceColumns);
+
+        waitForAsyncWriteToSucceedOnHCFS(fileName);
+
+        prepareReadableExternalTable("pxf_parquet_read_numeric_array_" + suffix,
+                new String[]{"id INTEGER", "v " + readType}, hdfsPath + fileName);
+    }
+
     private void prepareNumericWritableExtTable(String filePathName, String fileName, String writableExternalTableName, boolean isPrecisionDefined, boolean isLargePrecision) throws Exception {
         Table gpdbNumericTable;
         String[] numericTableColumns;

docs/content/cfg_server.html.md.erb

@@ -118,7 +118,7 @@ You configure properties in the `pxf-site.xml` file for a PXF server when one or
 | pxf.ppd.hive<sup>1</sup> | Specifies whether or not predicate pushdown is enabled for queries on external tables that specify the `hive`, `hive:rc`, or `hive:orc` profiles. | True; predicate pushdown is enabled. |
 | pxf.sasl.connection.retries | Specifies the maximum number of times that PXF retries a SASL connection request after a refused connection returns a `GSS initiate failed` error. | 5 |
 | pxf.orc.write.decimal.overflow | Specifies how PXF handles numeric data that exceeds the maximum precision of 38 and [overflows](hdfs_orc.html#overflow) when writing to an ORC file. Valid values are: round, error, or ignore | round |
-| pxf.parquet.write.decimal.overflow | Specifies how PXF handles numeric data that exceeds the maximum precision of 38 and [overflows](hdfs_parquet.html#overflow) when writing to a Parquet file. Valid values are: round, error, or ignore | round |
+| pxf.parquet.write.decimal.overflow | Specifies how PXF handles numeric data that exceeds the maximum precision of 76 and [overflows](hdfs_parquet.html#overflow) when writing to a Parquet file. Valid values are: round, error, or ignore | round |
 
 </br><sup>1</sup>&nbsp;Should you need to, you can override this setting on a per-table basis by specifying the `&PPD=<boolean>` option in the `LOCATION` clause when you create the external table.
 

docs/content/hdfs_parquet.html.md.erb

@@ -247,14 +247,16 @@ In this example, you create a Parquet-format writable external table that uses t
 
 ## <a id="overflow"></a>Understanding Overflow Conditions When Writing Numeric Data
 
-PXF uses the `HiveDecimal` class to write numeric Parquet data. `HiveDecimal` limits both the precision and the scale of a numeric type to a maximum of 38.
+PXF writes numeric Parquet data as a `DECIMAL` with a precision of up to 76 digits. Both the precision and the scale of a numeric type are limited to a maximum of 76.
 
-When you define a `NUMERIC` column in an external table without specifying a precision or scale, PXF internally maps the column to a `DECIMAL(38, 18)`.
+When you define a `NUMERIC` column in an external table without specifying a precision or scale, PXF internally maps the column to a `DECIMAL(38, 18)`. To keep more digits, define the column with a precision and scale, for example `NUMERIC(76, 61)`.
+
+<div class="note">Many Parquet readers, including Apache Spark and Apache Hive, support a `DECIMAL` precision of at most 38 and cannot read a column that PXF writes with a greater precision. Arrow-based readers such as PyArrow read such a column as <code>decimal256</code>.</div>
 
 PXF handles the following precision overflow conditions:
 
-- You define a `NUMERIC` column in the external table, and the integer digit count of a value exceeds the maximum supported precision of 38. For example, `1234567890123456789012345678901234567890.12345`, which has an integer digit count of 45.
-- You define a `NUMERIC(<precision>)` column with a `<precision>` greater than 38. For example, `NUMERIC(55)`.
+- You define a `NUMERIC` column in the external table, and the integer digit count of a value exceeds the maximum supported precision of 38. For example, `1234567890123456789012345678901234567890.12345`, which has an integer digit count of 40.
+- You define a `NUMERIC(<precision>)` column with a `<precision>` greater than 76. For example, `NUMERIC(90)`.
 - You define a `NUMERIC` column in the external table, and the integer digit count of a value is greater than 20 (38-18). For example, `123456789012345678901234567890.12345`, which has an integer digit count of 30.
 
 If you define a `NUMERIC(<precision>, <scale>)` column and the integer digit count of a value is greater than `<precision> - <scale>`, PXF returns an error. For example, you define a `NUMERIC(20,4)` column and the value is `12345678901234567.12`, which has an integer digit count of 19, which is greater than 20-4=16.