lake: add UDF docs - #23544
Conversation
ti-chi-bot
Bot
added
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Skipping CI for Draft Pull Request. |
📝 WalkthroughWalkthroughAdded a UDF type-selection guide for TiDB Cloud Lake. Updated external UDF examples to use the TiDB Cloud Lake SDK and HTTPS endpoints. Added and corrected related navigation links. ChangesUDF documentation
Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: 🟡 Moderate · up to The new documentation currently includes external UDF examples that can fail because of invalid syntax and omits required server deployment and tenant allowlist steps; one AI example also lacks a required Python dependency, and one link is broken. These issues can lead users to failed registrations or queries, so fixes are needed before merge. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ti-chi-bot
Bot
added
missing-translation-status
ti-chi-bot
Bot
added
size/XL
ti-chi-bot
Bot
removed
the
missing-translation-status
There was a problem hiding this comment.
Pull request overview
This PR adds and surfaces TiDB Cloud Lake UDF documentation, including a new guide to help users choose among UDF types, and updates the External Function docs and examples to use the tidbcloudlake-udf package and public HTTPS endpoints.
Changes:
- Adds a new “Choose a User-Defined Function Type” guide and links it from the Lake TOC.
- Exposes the “External AI Functions” guide in the Lake TOC and updates Python examples to use
tidbcloudlake-udf. - Updates External Function navigation and examples, including adding the
CREATE FUNCTIONreference entry.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| TOC-tidb-cloud-lake.md | Adds TOC entries for External AI Functions, the UDF type selection guide, and External Function command references. |
| tidb-cloud-lake/sql/external-function.md | Updates the External Function overview table to point at the correct command pages (and should align link text with actual SQL statements). |
| tidb-cloud-lake/sql/create-function.md | Updates the external function end-to-end example to use tidbcloudlake-udf and a public HTTPS ADDRESS. |
| tidb-cloud-lake/sql/alter-function-sql.md | Updates external function ADDRESS examples to a public HTTPS endpoint. |
| tidb-cloud-lake/guides/external-ai-functions.md | Updates the external AI function example to use tidbcloudlake-udf. |
| tidb-cloud-lake/guides/choose-a-udf-type.md | Adds a new guide explaining how to choose between SQL/script/WASM/aggregate/table/external UDF types and operational considerations. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| ```bash | ||
| pip install databend-udf | ||
| pip install tidbcloudlake-udf |
| CREATE FUNCTION gcd AS (INT, INT) | ||
| RETURNS INT | ||
| LANGUAGE python | ||
| HANDLER = 'gcd' | ||
| ADDRESS = 'http://localhost:8815'; | ||
| ADDRESS = 'https://udf.example.com'; |
|
@sdojjy: adding LGTM is restricted to approvers and reviewers in OWNERS files. DetailsIn response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
ti-chi-bot
Bot
removed
the
do-not-merge/work-in-progress
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: addeaf63-3c6c-4a46-a6c3-9da9ffd30949
📒 Files selected for processing (6)
TOC-tidb-cloud-lake.mdtidb-cloud-lake/guides/choose-a-udf-type.mdtidb-cloud-lake/guides/external-ai-functions.mdtidb-cloud-lake/sql/alter-function-sql.mdtidb-cloud-lake/sql/create-function.mdtidb-cloud-lake/sql/external-function.md
Included review availability: Your plan includes up to 4 reviews per rolling hour; 3 remain after this review.
| CREATE FUNCTION normalize_address(value VARCHAR) | ||
| RETURNS VARCHAR | ||
| LANGUAGE python | ||
| HANDLER = 'normalize_address' | ||
| ADDRESS = 'https://udf.example.com'; | ||
| ``` |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- candidate files ---'
git ls-files | rg '(^|/)(choose-a-udf-type\.md|alter-function-sql\.md)$|function|udf'
printf '%s\n' '--- relevant syntax references and definitions ---'
rg -n -i -C 3 'CREATE FUNCTION|ALTER FUNCTION|LANGUAGE python|HANDLER|ADDRESS|AS \(' tidb-cloud-lake --glob '*.md' --glob '*.mdx' --glob '*.sql' | head -n 400Repository: pingcap/docs
Length of output: 36730
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- external function reference ---'
sed -n '1,220p' tidb-cloud-lake/sql/create-function.md
printf '%s\n' '--- alter function references ---'
sed -n '1,180p' tidb-cloud-lake/sql/alter-function.md
sed -n '1,120p' tidb-cloud-lake/sql/alter-function-sql.md
printf '%s\n' '--- guide examples ---'
sed -n '50,170p' tidb-cloud-lake/guides/choose-a-udf-type.md
sed -n '200,300p' tidb-cloud-lake/guides/choose-a-udf-type.md
printf '%s\n' '--- all external UDF grammar examples ---'
rg -n -i -C 5 'CREATE FUNCTION|ALTER FUNCTION' tidb-cloud-lake/sql tidb-cloud-lake/guides --glob '*.md' | rg -i -C 5 'LANGUAGE|HANDLER|ADDRESS|CREATE FUNCTION|ALTER FUNCTION' | head -n 500Repository: pingcap/docs
Length of output: 42351
🌐 Web query:
TiDB Cloud Lake external UDF CREATE FUNCTION AS input_param_types HANDLER ADDRESS syntax
💡 Result:
In TiDB Cloud Lake, the syntax for creating an external user-defined function (UDF) that connects to an external UDF server uses the ADDRESS parameter [1]. The required syntax is as follows: CREATE FUNCTION function_name (input_param_types) RETURNS return_type LANGUAGE language_name HANDLER = 'handler_name' ADDRESS = 'udf_server_address'; Parameters: - function_name: The name of the function to be created [1]. - input_param_types: A comma-separated list of the input parameter data types (e.g., INT, VARCHAR) [1]. - return_type: The data type the function returns [1]. - LANGUAGE: The runtime language used (e.g., python) [1]. - HANDLER: The name of the specific handler function to call within the external code [1]. - ADDRESS: The network address (URL) of the external UDF server (e.g., 'http://0.0.0.0:8815') [1]. Note that while TiDB Cloud Lake supports various UDF types—including embedded Python/JavaScript functions (defined with AS
Citations:
- 1: https://docs.pingcap.com/tidbcloudlake/alter-function-sql/
- 2: https://docs.pingcap.com/tidbcloudlake/create-scalar-function/
- 3: https://docs.pingcap.com/tidbcloudlake/create-aggregate-function/
- 4: https://docs.pingcap.com/tidbcloudlake/create-table-function/
- 5: https://docs.databend.com/sql/sql-commands/ddl/external-function/ddl-create-function
🏁 Script executed:
#!/bin/bash
set -eu
for url in \
'https://docs.pingcap.com/tidbcloudlake/create-function/' \
'https://docs.pingcap.com/tidbcloudlake/alter-function-sql/'; do
printf '\n--- %s ---\n' "$url"
curl -L --fail --silent --show-error "$url" |
sed 's/<[^>]*>/ /g' |
tr -s '[:space:]' ' ' |
grep -o -i -E '.{0,180}(CREATE FUNCTION|ALTER FUNCTION|HANDLER|ADDRESS|AS[[:space:]]*\().{0,260}' |
head -n 20
done
printf '\n--- repository references to the external grammar ---\n'
rg -n -i -C 8 'external function|external UDF|AS \( *<input_param_types>|CREATE FUNCTION .*ADDRESS|ALTER FUNCTION .*ADDRESS' . --glob '!*.json' --glob '!*.yaml' --glob '!*.yml' | head -n 500Repository: pingcap/docs
Length of output: 48938
🌐 Web query:
site:github.com/datafuselabs/databend CREATE FUNCTION ADDRESS HANDLER parser external function AS
💡 Result:
The syntax pattern CREATE FUNCTION ... HANDLER ... ADDRESS is specifically used in Databend, a cloud-native data warehouse, to define external User-Defined Functions (UDFs) that run on a remote server [1][2]. In Databend, this command registers a UDF that communicates with an external UDF server (often written in Python) to execute custom logic [1][2]. The syntax is structured as follows [1]: CREATE FUNCTION <udf_name> (<arg_type>,...) RETURNS <return_type> LANGUAGE HANDLER = '<handler_name>' ADDRESS = '<server_address>'; Key components of this syntax: - HANDLER: Specifies the name of the function or method to be called within the external service [1]. - ADDRESS: Defines the network location (URL/IP) of the external UDF server, which typically communicates with the Databend kernel using a protocol like Arrow Flight [1][2]. This functionality allows users to extend Databend's capabilities by offloading complex computations to dedicated external services [2].
Citations:
- 1: feat: implement udf server in databend databendlabs/databend#12729
- 2: Feature: Implement Databend UDF Server (OSPP) databendlabs/databend#11517
🏁 Script executed:
#!/bin/bash
set -eu
for endpoint in \
'https://api.github.com/repos/datafuselabs/databend/pulls/12729' \
'https://api.github.com/repos/datafuselabs/databend/pulls/12729/files?per_page=100'; do
printf '\n--- %s ---\n' "$endpoint"
curl -L --fail --silent --show-error \
-H 'Accept: application/vnd.github+json' "$endpoint" |
python3 -c '
import json, sys
data=json.load(sys.stdin)
if isinstance(data, list):
for item in data:
print("FILE:", item.get("filename"))
patch=item.get("patch","")
for line in patch.splitlines():
if any(word in line.upper() for word in ("CREATE FUNCTION", "ALTER FUNCTION", "HANDLER", "ADDRESS")):
print(line)
else:
for key in ("title","html_url","state","merged_at","merge_commit_sha"):
print(f"{key}: {data.get(key)}")
'
doneRepository: pingcap/docs
Length of output: 15001
🏁 Script executed:
#!/bin/bash
set -eu
base='https://raw.githubusercontent.com/databendlabs/databend/f06d9b311ec046f6702f42d5a9947f9b6e173b61'
for path in \
'src/query/ast/src/parser/statement.rs' \
'src/query/ast/src/ast/statements/udf.rs' \
'tests/sqllogictests/suites/udf_server/udf_server_test' \
'tests/udf-server/README.md'; do
printf '\n--- %s ---\n' "$path"
curl -L --fail --silent --show-error "$base/$path" |
grep -n -i -C 5 -E 'CREATE FUNCTION|ALTER FUNCTION|RETURNS TABLE|return_type|udf_server|HANDLER|ADDRESS' |
head -n 300
doneRepository: pingcap/docs
Length of output: 11994
🏁 Script executed:
#!/bin/bash
set -eu
url='https://raw.githubusercontent.com/databendlabs/databend/f06d9b311ec046f6702f42d5a9947f9b6e173b61/src/query/ast/src/parser/statement.rs'
content="$(curl -L --fail --silent --show-error "$url")"
printf '%s\n' '--- udf_arg_type definition and nearby parser rules ---'
printf '%s\n' "$content" |
grep -n -i -C 20 -E 'udf_arg_type|udf_server|RETURNS TABLE|table.*return'
printf '%s\n' '--- published references to external table UDFs ---'
rg -n -i -C 5 'external table UDF|RETURNS TABLE.*LANGUAGE python|result_type.*token|table handler' tidb-cloud-lake --glob '*.md'Repository: pingcap/docs
Length of output: 12478
Use positional types for external UDFs.
External UDFs do not use AS. Change the guide examples to (VARCHAR), not (value VARCHAR). The gcd CREATE FUNCTION and ALTER FUNCTION examples already use the correct form. RETURNS TABLE(...) is not valid external-function grammar; use a supported return type or document split_words as a SQL table UDF.
📍 Affects 2 files
tidb-cloud-lake/guides/choose-a-udf-type.md#L261-L266(this comment)tidb-cloud-lake/guides/choose-a-udf-type.md#L284-L289tidb-cloud-lake/sql/alter-function-sql.md#L35-L38
Source: Path instructions
| ## How It Works | ||
|
|
||
| 1. **Create AI Server**: Build your AI/ML server using Python and [databend-udf](https://pypi.org/project/databend-udf) | ||
| 1. **Create AI Server**: Build your AI/ML server using Python and [`tidbcloudlake-udf`](https://pypi.org/project/tidbcloudlake-udf/) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -e
file="tidb-cloud-lake/guides/external-ai-functions.md"
printf '%s\n' '--- target excerpt ---'
sed -n '1,80p' "$file"
printf '%s\n' '--- dependency references ---'
rg -n -i 'sentence[-_ ]transformers|SentenceTransformer|tidbcloudlake-udf|requirements|pip install|dependencies' tidb-cloud-lake
printf '%s\n' '--- tracked nearby files ---'
git ls-files 'tidb-cloud-lake/*' | sed -n '1,120p'Repository: pingcap/docs
Length of output: 21761
🏁 Script executed:
#!/bin/bash
set -e
python3 - <<'PY'
import json
import urllib.request
for name in ("tidbcloudlake-udf", "sentence-transformers"):
url = f"https://pypi.org/pypi/{name}/json"
with urllib.request.urlopen(url, timeout=10) as response:
metadata = json.load(response)["info"]
print(f"{name}: version={metadata['version']}")
print(f"requires_dist={metadata.get('requires_dist') or []}")
PYRepository: pingcap/docs
Length of output: 2436
🏁 Script executed:
#!/bin/bash
set -e
for name in tidbcloudlake-udf sentence-transformers; do
curl --silent --show-error --insecure --fail "https://pypi.org/pypi/${name}/json" |
python3 -c 'import json,sys; d=json.load(sys.stdin)["info"]; print("name=", d["name"]); print("version=", d["version"]); print("requires_dist=", d.get("requires_dist") or [])'
doneRepository: pingcap/docs
Length of output: 1233
Document the model dependency used by the example.
The example imports SentenceTransformer, but tidbcloudlake-udf does not install sentence-transformers. Add it to the setup instruction to prevent ModuleNotFoundError.
Committable replacement
-1. **Create AI Server**: Build your AI/ML server using Python and [`tidbcloudlake-udf`](https://pypi.org/project/tidbcloudlake-udf/)
+1. **Create AI Server**: Install [`tidbcloudlake-udf`](https://pypi.org/project/tidbcloudlake-udf/) and `sentence-transformers`, then build your AI/ML server using Python.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| 1. **Create AI Server**: Build your AI/ML server using Python and [`tidbcloudlake-udf`](https://pypi.org/project/tidbcloudlake-udf/) | |
| 1. **Create AI Server**: Install [`tidbcloudlake-udf`](https://pypi.org/project/tidbcloudlake-udf/) and `sentence-transformers`, then build your AI/ML server using Python. |
Source: Path instructions
| LANGUAGE python | ||
| HANDLER = 'gcd' | ||
| ADDRESS = 'http://localhost:8815'; | ||
| ADDRESS = 'https://udf.example.com'; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- document outline ---'
ast-grep outline tidb-cloud-lake/sql/create-function.md --lang markdown 2>/dev/null || true
printf '%s\n' '--- document size ---'
wc -l tidb-cloud-lake/sql/create-function.md
printf '%s\n' '--- relevant document sections ---'
sed -n '1,150p' tidb-cloud-lake/sql/create-function.md
printf '%s\n' '--- related repository references ---'
rg -n -i --glob '*.md' --glob '*.mdx' \
'UDFServer|udf\.example\.com|allowlist|allow list|hosted endpoint|CREATE FUNCTION|ADDRESS\s*=' . \
| head -250Repository: pingcap/docs
Length of output: 26590
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- external AI functions guide ---'
sed -n '1,90p' tidb-cloud-lake/guides/external-ai-functions.md
printf '%s\n' '--- UDF type guide deployment and registration ---'
sed -n '230,310p' tidb-cloud-lake/guides/choose-a-udf-type.md
printf '%s\n' '--- all UDF deployment/allowlist references ---'
rg -n -i -C 3 --glob 'tidb-cloud-lake/**/*.md' \
'deploying and allowlisting|allowlist.*UDF|UDF.*allowlist|deploy.*server|server.*deploy' \
| head -200Repository: pingcap/docs
Length of output: 5326
Add deployment and allowlist prerequisites before registration.
Before CREATE FUNCTION, instruct readers to deploy the UDF Server at a public HTTPS endpoint and add its hostname to the tenant UDF server allowlist. The local UDFServer("0.0.0.0:8815") process cannot serve Cloud queries at https://udf.example.com.
Source: Path instructions
[LGTM Timeline notifier]Timeline:
|
What is changed, added or deleted? (Required)
This PR publishes TiDB Cloud Lake UDF (user-defined functions) documentation with minimal changes to existing pages:
CREATE FUNCTIONreference in the Lake TOC;tidbcloudlake-udfandtidbcloudlake_udfin external Python examples;The examples were validated against the published
tidbcloudlake-udfv0.2.20 package and a staging Lake warehouse. SQL scalar/table UDFs and Python/JavaScript aggregate UDFs passed. External registration correctly returnedUnallowed UDF server addressfor a hostname not yet added to the tenant allowlist.Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions.
What is the related PR or file link(s)?
AI agent involvement
Do your changes match any of the following descriptions?
Summary by CodeRabbit
ALTERandDROPstatements.