Skip to content

Commit 8f264e8

Browse files
simonwclaude
simonw
and
claude
committed
transform() now works for tables referenced by views
Bracket the ALTER TABLE ... RENAME TO statements emitted by transform_sql() with PRAGMA legacy_alter_table=ON/OFF. Since SQLite 3.25 the rename would otherwise rewrite references in every view definition, which failed with "no such table" when a view referenced the just-dropped table, and with keep_table= silently repointed dependent views at the frozen backup table. View definitions are now left byte-for-byte unchanged by a transform. Closes #831 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent a7b7349 commit 8f264e8

5 files changed

Lines changed: 179 additions & 2 deletions

File tree

‎docs/changelog.rst‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,13 @@
44
Changelog
55
===========
66

7+
.. _unreleased:
8+
9+
Unreleased
10+
----------
11+
12+
- ``table.transform()`` now works for tables that are referenced by views. Previously the ``ALTER TABLE ... RENAME TO`` step raised ``no such table`` if a view referenced the table being transformed. View definitions are left unchanged - see :ref:`python_api_transform_views`. This also fixes a bug where ``transform(keep_table=...)`` silently rewrote dependent views to point at the frozen backup table instead of the live one. (:issue:`831`)
13+
714
.. _v3_39_1:
815

916
3.39.1 (2026-07-25)
@@ -18,6 +25,7 @@
1825

1926
- ``table.transform()`` now raises a ``TransactionError`` if called while a transaction is open with ``PRAGMA foreign_keys`` enabled and the table is referenced by foreign keys with destructive ``ON DELETE`` actions - ``CASCADE``, ``SET NULL`` or ``SET DEFAULT``. The pragma cannot be changed inside a transaction, so previously dropping the old table as part of the transform could fire those actions and silently delete or modify referencing rows. See :ref:`python_api_transform_foreign_keys_transactions` for details and workarounds. (:issue:`794`)
2027
- The :ref:`CLI <cli>` and :ref:`Python API <python_api>` documentation now cross-reference each other: CLI sections link to the equivalent Python API functionality and Python API sections link back to the corresponding CLI command. (:issue:`791`)
28+
2129
.. _v4_1:
2230

2331
4.1 (2026-07-11)

‎docs/cli.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2288,7 +2288,11 @@ If you want to see the SQL that will be executed to make the change without actu
22882288
INSERT INTO "roadside_attractions_new_4033a60276b9" ("longitude", "latitude", "id", "name")
22892289
SELECT "longitude", "latitude", "pk", "name" FROM "roadside_attractions";
22902290
DROP TABLE "roadside_attractions";
2291+
PRAGMA legacy_alter_table=ON;
22912292
ALTER TABLE "roadside_attractions_new_4033a60276b9" RENAME TO "roadside_attractions";
2293+
PRAGMA legacy_alter_table=OFF;
2294+
2295+
Tables that are referenced by views can be transformed - the view definitions are left unchanged, see :ref:`python_api_transform_views` for details.
22922296

22932297
.. note::
22942298
In Python: :ref:`table.transform() <python_api_transform>` CLI reference: :ref:`sqlite-utils transform <cli_ref_transform>`

‎docs/python-api.rst‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1986,6 +1986,17 @@ A bare column name drops any foreign key that column participates in, including
19861986
19871987
Renaming a column with ``rename=`` updates any foreign keys that use it, and dropping a column with ``drop=`` also drops any foreign keys it participates in - for a compound foreign key this removes the whole constraint.
19881988
1989+
.. _python_api_transform_views:
1990+
1991+
Tables referenced by views
1992+
--------------------------
1993+
1994+
Tables that are referenced by views can be safely transformed - the view definitions are left byte-for-byte unchanged, and views continue to read from the live table even when ``keep_table=`` is used to keep a copy of the original around.
1995+
1996+
A view that references a column which the transform renamed or dropped will remain defined but will raise a ``no such column`` error when it is next queried. This is inherent to SQLite views, whose SQL is stored as text - if you rename or drop columns that a view depends on you should update that view definition yourself.
1997+
1998+
To achieve this, the SQL produced by ``transform_sql()`` brackets its ``ALTER TABLE ... RENAME TO`` statements with ``PRAGMA legacy_alter_table=ON`` and ``PRAGMA legacy_alter_table=OFF`` - without this, SQLite would attempt to rewrite references to the renamed table in every view definition, which fails when a view references the table that was just dropped. One consequence is that ``PRAGMA legacy_alter_table`` is reset to ``OFF`` (the SQLite default) after a transform, even if it was previously set to ``ON`` for the connection.
1999+
19892000
.. _python_api_transform_sql:
19902001
19912002
Custom transformations with .transform_sql()

‎sqlite_utils/db.py‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2824,17 +2824,25 @@ def fk_with_renamed_columns(fk: ForeignKey) -> ForeignKey:
28242824
new_cols=", ".join(quote_identifier(col) for col in new_cols),
28252825
)
28262826
sqls.append(copy_sql)
2827-
# Drop (or keep) the old table
2827+
# Drop (or keep) the old table, then rename the new one into place.
2828+
# Since SQLite 3.25 ALTER TABLE ... RENAME TO rewrites references to
2829+
# the renamed table in every view definition, which fails if a view
2830+
# references the table that was just dropped - and with keep_table=
2831+
# would silently repoint views at the backup table. These renames are
2832+
# an implementation detail of transform(), so use legacy_alter_table
2833+
# to leave view definitions untouched.
28282834
if keep_table:
2835+
sqls.append("PRAGMA legacy_alter_table=ON;")
28292836
sqls.append(
28302837
f"ALTER TABLE {quote_identifier(self.name)} RENAME TO {quote_identifier(keep_table)};"
28312838
)
28322839
else:
28332840
sqls.append(f"DROP TABLE {quote_identifier(self.name)};")
2834-
# Rename the new one
2841+
sqls.append("PRAGMA legacy_alter_table=ON;")
28352842
sqls.append(
28362843
f"ALTER TABLE {quote_identifier(new_table_name)} RENAME TO {quote_identifier(self.name)};"
28372844
)
2845+
sqls.append("PRAGMA legacy_alter_table=OFF;")
28382846
# Re-add existing indexes
28392847
for index in self.indexes:
28402848
if index.origin != "pk":

‎tests/test_transform.py‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,9 @@
1616
'CREATE TABLE "dogs_new_suffix" (\n "id" INTEGER PRIMARY KEY,\n "name" TEXT,\n "age" TEXT\n);',
1717
'INSERT INTO "dogs_new_suffix" ("rowid", "id", "name", "age")\n SELECT "rowid", "id", "name", "age" FROM "dogs";',
1818
'DROP TABLE "dogs";',
19+
"PRAGMA legacy_alter_table=ON;",
1920
'ALTER TABLE "dogs_new_suffix" RENAME TO "dogs";',
21+
"PRAGMA legacy_alter_table=OFF;",
2022
],
2123
),
2224
# Change column type
@@ -26,7 +28,9 @@
2628
'CREATE TABLE "dogs_new_suffix" (\n "id" INTEGER PRIMARY KEY,\n "name" TEXT,\n "age" INTEGER\n);',
2729
'INSERT INTO "dogs_new_suffix" ("rowid", "id", "name", "age")\n SELECT "rowid", "id", "name", "age" FROM "dogs";',
2830
'DROP TABLE "dogs";',
31+
"PRAGMA legacy_alter_table=ON;",
2932
'ALTER TABLE "dogs_new_suffix" RENAME TO "dogs";',
33+
"PRAGMA legacy_alter_table=OFF;",
3034
],
3135
),
3236
# Rename a column
@@ -36,7 +40,9 @@
3640
'CREATE TABLE "dogs_new_suffix" (\n "id" INTEGER PRIMARY KEY,\n "name" TEXT,\n "dog_age" TEXT\n);',
3741
'INSERT INTO "dogs_new_suffix" ("rowid", "id", "name", "dog_age")\n SELECT "rowid", "id", "name", "age" FROM "dogs";',
3842
'DROP TABLE "dogs";',
43+
"PRAGMA legacy_alter_table=ON;",
3944
'ALTER TABLE "dogs_new_suffix" RENAME TO "dogs";',
45+
"PRAGMA legacy_alter_table=OFF;",
4046
],
4147
),
4248
# Drop a column
@@ -46,7 +52,9 @@
4652
'CREATE TABLE "dogs_new_suffix" (\n "id" INTEGER PRIMARY KEY,\n "name" TEXT\n);',
4753
'INSERT INTO "dogs_new_suffix" ("rowid", "id", "name")\n SELECT "rowid", "id", "name" FROM "dogs";',
4854
'DROP TABLE "dogs";',
55+
"PRAGMA legacy_alter_table=ON;",
4956
'ALTER TABLE "dogs_new_suffix" RENAME TO "dogs";',
57+
"PRAGMA legacy_alter_table=OFF;",
5058
],
5159
),
5260
# Convert type AND rename column
@@ -56,7 +64,9 @@
5664
'CREATE TABLE "dogs_new_suffix" (\n "id" INTEGER PRIMARY KEY,\n "name" TEXT,\n "dog_age" INTEGER\n);',
5765
'INSERT INTO "dogs_new_suffix" ("rowid", "id", "name", "dog_age")\n SELECT "rowid", "id", "name", "age" FROM "dogs";',
5866
'DROP TABLE "dogs";',
67+
"PRAGMA legacy_alter_table=ON;",
5968
'ALTER TABLE "dogs_new_suffix" RENAME TO "dogs";',
69+
"PRAGMA legacy_alter_table=OFF;",
6070
],
6171
),
6272
# Change primary key
@@ -66,7 +76,9 @@
6676
'CREATE TABLE "dogs_new_suffix" (\n "id" INTEGER,\n "name" TEXT,\n "age" TEXT PRIMARY KEY\n);',
6777
'INSERT INTO "dogs_new_suffix" ("rowid", "id", "name", "age")\n SELECT "rowid", "id", "name", "age" FROM "dogs";',
6878
'DROP TABLE "dogs";',
79+
"PRAGMA legacy_alter_table=ON;",
6980
'ALTER TABLE "dogs_new_suffix" RENAME TO "dogs";',
81+
"PRAGMA legacy_alter_table=OFF;",
7082
],
7183
),
7284
# Change primary key to a compound pk
@@ -76,7 +88,9 @@
7688
'CREATE TABLE "dogs_new_suffix" (\n "id" INTEGER,\n "name" TEXT,\n "age" TEXT,\n PRIMARY KEY ("age", "name")\n);',
7789
'INSERT INTO "dogs_new_suffix" ("rowid", "id", "name", "age")\n SELECT "rowid", "id", "name", "age" FROM "dogs";',
7890
'DROP TABLE "dogs";',
91+
"PRAGMA legacy_alter_table=ON;",
7992
'ALTER TABLE "dogs_new_suffix" RENAME TO "dogs";',
93+
"PRAGMA legacy_alter_table=OFF;",
8094
],
8195
),
8296
# Remove primary key, creating a rowid table
@@ -86,7 +100,9 @@
86100
'CREATE TABLE "dogs_new_suffix" (\n "id" INTEGER,\n "name" TEXT,\n "age" TEXT\n);',
87101
'INSERT INTO "dogs_new_suffix" ("rowid", "id", "name", "age")\n SELECT "rowid", "id", "name", "age" FROM "dogs";',
88102
'DROP TABLE "dogs";',
103+
"PRAGMA legacy_alter_table=ON;",
89104
'ALTER TABLE "dogs_new_suffix" RENAME TO "dogs";',
105+
"PRAGMA legacy_alter_table=OFF;",
90106
],
91107
),
92108
# Keeping the table
@@ -95,8 +111,10 @@
95111
[
96112
'CREATE TABLE "dogs_new_suffix" (\n "id" INTEGER PRIMARY KEY,\n "name" TEXT\n);',
97113
'INSERT INTO "dogs_new_suffix" ("rowid", "id", "name")\n SELECT "rowid", "id", "name" FROM "dogs";',
114+
"PRAGMA legacy_alter_table=ON;",
98115
'ALTER TABLE "dogs" RENAME TO "kept_table";',
99116
'ALTER TABLE "dogs_new_suffix" RENAME TO "dogs";',
117+
"PRAGMA legacy_alter_table=OFF;",
100118
],
101119
),
102120
],
@@ -139,7 +157,9 @@ def tracer(sql, params):
139157
'CREATE TABLE "dogs_new_suffix" (\n "id" INTEGER,\n "name" TEXT,\n "age" TEXT\n);',
140158
'INSERT INTO "dogs_new_suffix" ("rowid", "id", "name", "age")\n SELECT "rowid", "id", "name", "age" FROM "dogs";',
141159
'DROP TABLE "dogs";',
160+
"PRAGMA legacy_alter_table=ON;",
142161
'ALTER TABLE "dogs_new_suffix" RENAME TO "dogs";',
162+
"PRAGMA legacy_alter_table=OFF;",
143163
],
144164
),
145165
# Change column type
@@ -149,7 +169,9 @@ def tracer(sql, params):
149169
'CREATE TABLE "dogs_new_suffix" (\n "id" INTEGER,\n "name" TEXT,\n "age" INTEGER\n);',
150170
'INSERT INTO "dogs_new_suffix" ("rowid", "id", "name", "age")\n SELECT "rowid", "id", "name", "age" FROM "dogs";',
151171
'DROP TABLE "dogs";',
172+
"PRAGMA legacy_alter_table=ON;",
152173
'ALTER TABLE "dogs_new_suffix" RENAME TO "dogs";',
174+
"PRAGMA legacy_alter_table=OFF;",
153175
],
154176
),
155177
# Rename a column
@@ -159,7 +181,9 @@ def tracer(sql, params):
159181
'CREATE TABLE "dogs_new_suffix" (\n "id" INTEGER,\n "name" TEXT,\n "dog_age" TEXT\n);',
160182
'INSERT INTO "dogs_new_suffix" ("rowid", "id", "name", "dog_age")\n SELECT "rowid", "id", "name", "age" FROM "dogs";',
161183
'DROP TABLE "dogs";',
184+
"PRAGMA legacy_alter_table=ON;",
162185
'ALTER TABLE "dogs_new_suffix" RENAME TO "dogs";',
186+
"PRAGMA legacy_alter_table=OFF;",
163187
],
164188
),
165189
# Make ID a primary key
@@ -169,7 +193,9 @@ def tracer(sql, params):
169193
'CREATE TABLE "dogs_new_suffix" (\n "id" INTEGER PRIMARY KEY,\n "name" TEXT,\n "age" TEXT\n);',
170194
'INSERT INTO "dogs_new_suffix" ("rowid", "id", "name", "age")\n SELECT "rowid", "id", "name", "age" FROM "dogs";',
171195
'DROP TABLE "dogs";',
196+
"PRAGMA legacy_alter_table=ON;",
172197
'ALTER TABLE "dogs_new_suffix" RENAME TO "dogs";',
198+
"PRAGMA legacy_alter_table=OFF;",
173199
],
174200
),
175201
],
@@ -903,3 +929,123 @@ def test_transform_with_unique_constraint_implicit_index(fresh_db):
903929
"You must manually drop this index prior to running this transformation and manually recreate the new index after running this transformation."
904930
in str(excinfo.value)
905931
)
932+
933+
934+
def test_transform_preserves_view(fresh_db):
935+
# https://github.com/simonw/sqlite-utils/issues/831
936+
dogs = fresh_db["dogs"]
937+
dogs.insert({"id": 1, "name": "Cleo"}, pk="id")
938+
fresh_db.execute("create view dogs_view as select id, name from dogs")
939+
view_sql_before = fresh_db.execute(
940+
"select sql from sqlite_master where name = 'dogs_view'"
941+
).fetchone()[0]
942+
dogs.transform(rename={"name": "title"})
943+
view_sql_after = fresh_db.execute(
944+
"select sql from sqlite_master where name = 'dogs_view'"
945+
).fetchone()[0]
946+
assert view_sql_before == view_sql_after
947+
948+
949+
@pytest.mark.parametrize(
950+
"transform_params",
951+
[
952+
{"types": {"name": int}},
953+
{"pk": "name"},
954+
{"add_foreign_keys": [("other_id", "other", "id")]},
955+
{"drop_foreign_keys": ["other_id"]},
956+
],
957+
)
958+
def test_transform_variants_preserve_view(fresh_db, transform_params):
959+
# Covers retyping, changing primary key and foreign key modifications,
960+
# with a view whose columns are untouched by the transform
961+
fresh_db["other"].insert({"id": 1}, pk="id")
962+
dogs = fresh_db["dogs"]
963+
dogs.insert({"id": 1, "name": "Cleo", "other_id": 1}, pk="id")
964+
if "drop_foreign_keys" in transform_params:
965+
dogs.transform(add_foreign_keys=[("other_id", "other", "id")])
966+
fresh_db.execute("create view dogs_view as select id, name from dogs")
967+
view_sql_before = fresh_db.execute(
968+
"select sql from sqlite_master where name = 'dogs_view'"
969+
).fetchone()[0]
970+
dogs.transform(**transform_params)
971+
view_sql_after = fresh_db.execute(
972+
"select sql from sqlite_master where name = 'dogs_view'"
973+
).fetchone()[0]
974+
assert view_sql_before == view_sql_after
975+
assert list(fresh_db["dogs_view"].rows) == [{"id": 1, "name": "Cleo"}]
976+
977+
978+
def test_transform_view_referencing_renamed_column(fresh_db):
979+
# The view survives but querying it raises "no such column" - inherent
980+
# to SQLite views, whose SQL is stored as text
981+
dogs = fresh_db["dogs"]
982+
dogs.insert({"id": 1, "name": "Cleo"}, pk="id")
983+
fresh_db.execute("create view dogs_view as select id, name from dogs")
984+
dogs.transform(rename={"name": "title"})
985+
with pytest.raises(OperationalError, match="no such column"):
986+
fresh_db.execute("select * from dogs_view")
987+
988+
989+
def test_transform_view_on_view(fresh_db):
990+
dogs = fresh_db["dogs"]
991+
dogs.insert({"id": 1, "name": "Cleo"}, pk="id")
992+
fresh_db.execute("create view v1 as select id, name from dogs")
993+
fresh_db.execute("create view v2 as select name from v1")
994+
sqls_before = fresh_db.execute(
995+
"select sql from sqlite_master where type = 'view' order by name"
996+
).fetchall()
997+
dogs.transform(types={"id": str})
998+
sqls_after = fresh_db.execute(
999+
"select sql from sqlite_master where type = 'view' order by name"
1000+
).fetchall()
1001+
assert sqls_before == sqls_after
1002+
assert list(fresh_db["v2"].rows) == [{"name": "Cleo"}]
1003+
1004+
1005+
def test_transform_keep_table_does_not_repoint_view(fresh_db):
1006+
# Without legacy_alter_table the ALTER TABLE dogs RENAME TO dogs_backup
1007+
# step would rewrite the view to select from "dogs_backup"
1008+
dogs = fresh_db["dogs"]
1009+
dogs.insert({"id": 1, "name": "Cleo"}, pk="id")
1010+
fresh_db.execute("create view dogs_view as select id, name from dogs")
1011+
dogs.transform(types={"name": str}, keep_table="dogs_backup")
1012+
view_sql = fresh_db.execute(
1013+
"select sql from sqlite_master where name = 'dogs_view'"
1014+
).fetchone()[0]
1015+
assert "dogs_backup" not in view_sql
1016+
# View reads from the live table, not the frozen backup
1017+
dogs.insert({"id": 2, "name": "Pancakes"})
1018+
assert list(fresh_db["dogs_view"].rows) == [
1019+
{"id": 1, "name": "Cleo"},
1020+
{"id": 2, "name": "Pancakes"},
1021+
]
1022+
1023+
1024+
def test_transform_sql_standalone_statements_work_with_view(fresh_db):
1025+
# The documented "run these statements yourself" workflow should be
1026+
# standalone-correct, so the pragmas must come from transform_sql()
1027+
dogs = fresh_db["dogs"]
1028+
dogs.insert({"id": 1, "name": "Cleo"}, pk="id")
1029+
fresh_db.execute("create view dogs_view as select id, name from dogs")
1030+
sqls = dogs.transform_sql(types={"name": str}, tmp_suffix="suffix")
1031+
assert sqls[-3] == "PRAGMA legacy_alter_table=ON;"
1032+
assert sqls[-2] == 'ALTER TABLE "dogs_new_suffix" RENAME TO "dogs";'
1033+
assert sqls[-1] == "PRAGMA legacy_alter_table=OFF;"
1034+
for sql in sqls:
1035+
fresh_db.execute(sql)
1036+
assert list(fresh_db["dogs_view"].rows) == [{"id": 1, "name": "Cleo"}]
1037+
1038+
1039+
def test_transform_with_view_in_open_transaction(fresh_db):
1040+
fresh_db.conn.execute("PRAGMA foreign_keys=ON")
1041+
dogs = fresh_db["dogs"]
1042+
dogs.insert({"id": 1, "name": "Cleo"}, pk="id")
1043+
fresh_db.execute("create view dogs_view as select id, name from dogs")
1044+
with fresh_db.conn:
1045+
fresh_db.execute("insert into dogs (id, name) values (2, 'Pancakes')")
1046+
dogs.transform(rename={"name": "title"})
1047+
assert dogs.columns_dict == {"id": int, "title": str}
1048+
view_sql = fresh_db.execute(
1049+
"select sql from sqlite_master where name = 'dogs_view'"
1050+
).fetchone()[0]
1051+
assert view_sql == "CREATE VIEW dogs_view as select id, name from dogs"

0 commit comments

Comments
 (0)