Skip to content

Generate Random Table data for a given query - #1

Open
tanvi2410 wants to merge 8 commits into
wmoustafa:data-generationfrom
tanvi2410:generating-test-data
Open

tanvi2410 wants to merge 8 commits into
wmoustafa:data-generationfrom
tanvi2410:generating-test-data

Conversation

@tanvi2410

Copy link
Copy Markdown

Summary

This PR introduces a new test data generation system that creates realistic test data by analyzing SQL predicates and inferring value constraints for table columns.

Key Features

  • Predicate-Aware Data Generation: Analyzes SQL WHERE clauses to generate data that satisfies query constraints
  • Domain Inference Integration: Uses DomainInferenceProgram to derive input domains from predicates (e.g., SUBSTRING(name, 1, 4) = '2000' generates names starting with "2000")
  • DNF Conversion: Handles complex predicates by converting to Disjunctive Normal Form
  • Random Fallback: Generates type-appropriate random data for unconstrained columns

Implementation Details

  • Supports numeric types (INTEGER, BIGINT, FLOAT, DOUBLE)
  • Supports string types (CHAR, VARCHAR) with regex domain constraints
  • Supports temporal types (DATE, TIME, TIMESTAMP)
  • Handles conjunctive predicates (AND) by intersecting domains
  • Uses ThreadLocalRandom for efficient random value generation

Test

Added unit test with different SQL scenarios.

Test results

=== Test: Generate Test Data with LOWER Predicate ===
SQL: SELECT * FROM test.T WHERE LOWER(name) = 'john'

Generated Test Data:
Row 1: {name=JOHN, age=505, birthdate=2000-08-09}
Row 2: {name=JOHN, age=322, birthdate=2002-05-04}
Row 3: {name=JOHN, age=516, birthdate=2013-12-06}
=== Test Passed ===


=== Test: Generate Test Data with Multiple Predicates ===
SQL: SELECT * FROM test.T WHERE SUBSTRING(name, 1, 4) = '2000' AND age = 25

Generated Test Data:
Row 1: {name=2000, age=25, birthdate=2016-01-09}
Row 2: {name=2000, age=25, birthdate=2011-08-24}
Row 3: {name=2000, age=25, birthdate=2008-04-28}
=== Test Passed ===


=== Test: Generate Test Data with No Predicate ===
SQL: SELECT * FROM test.T

Generated Test Data:
Row 1: {name=yB3p6DY0nt, age=581, birthdate=2012-07-13}
Row 2: {name=PUaVJ0R50z, age=173, birthdate=2009-12-22}
Row 3: {name=1ei9EFv5lc, age=614, birthdate=2010-07-27}
Row 4: {name=zzYqA6nkiQ, age=545, birthdate=2016-06-21}
Row 5: {name=wux0RgM8Aa, age=35, birthdate=2023-07-06}
=== Test Passed ===


=== Test: Generate Test Data with SUBSTRING Predicate ===
SQL: SELECT * FROM test.T WHERE SUBSTRING(name, 1, 4) = '2000'

Generated Test Data:
Row 1: {name=2000, age=769, birthdate=2018-09-14}
Row 2: {name=2000, age=980, birthdate=2002-12-12}
Row 3: {name=2000, age=687, birthdate=2020-10-06}
Row 4: {name=2000, age=928, birthdate=2006-08-21}
Row 5: {name=2000, age=805, birthdate=2010-08-26}
=== Test Passed ===

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants