Обсуждение: CREATE OR REPLACE MATERIALIZED VIEW
I like to add CREATE OR REPLACE MATERIALIZED VIEW with the attached patches. Patch 0001 adds CREATE OR REPLACE MATERIALIZED VIEW similar to CREATE OR REPLACE VIEW. It also includes regression tests and changes to docs. Patch 0002 deprecates CREATE MATERIALIZED VIEW IF NOT EXISTS because it no longer seems necessary with patch 0001. Tom Lane commented[1] about the general dislike of IF NOT EXISTS, to which I agree, but maybe this was meant only in response to adding new commands. Anyway, my idea is to deprecate that usage in PG18 and eventually remove it in PG19, if there's consensus for it. We can drop that clause without violating any standard because matviews are a Postgres extension. I'm not married to the idea, just want to put it on the table for discussion. Motivation ---------- At $JOB we use materialized views for caching a couple of expensive views. But every now and then those views have to be changed, e.g., new logic, new columns, etc. The matviews have to be dropped and re-created to include new columns. (Just changing the underlying view logic without adding new columns is trivial because the matviews are just thin wrappers that just have to be refreshed.) We also have several views that depend on those matviews. The views must also be dropped in order to re-create the matviews. We've already automated this with two procedures that stash and re-create dependent view definitions. Native support for replacing matviews would simplify our setup and it would make CREATE MATERIALIZED VIEW more complete when compared to CREATE VIEW. I searched the lists for previous discussions on this topic but couldn't find any. So, I don't know if this was ever tried, but rejected for some reason. I've found slides[2] from 2013 (when matviews landed in 9.3) which have OR REPLACE on the roadmap: > Materialised Views roadmap > > * CREATE **OR REPLACE** MATERIALIZED VIEW > * Just an oversight that it wasn't added > [...] Replacing Matviews ------------------ With patch 0001, a matview can be replaced without having to drop it and its dependent objects. In our use case it is no longer necessary to define the actual query in a separate view. Replacing a matview works analogous to CREATE OR REPLACE VIEW: * the new query may change SELECT list expressions of existing columns * new columns can be added to the end of the SELECT list * existing columns cannot be renamed * the data type of existing columns cannot be changed In addition to that, CREATE OR REPLACE MATERIALIZED VIEW also replaces access method, tablespace, and storage parameters if specified. The clause WITH [NO] DATA works as expected: it either populates the matview or leaves it in an unscannable state. It is an error to specify both OR REPLACE and IF NOT EXISTS. Example ------- postgres=# CREATE MATERIALIZED VIEW test AS SELECT 1 AS a; SELECT 1 postgres=# SELECT * FROM test; a --- 1 (1 row) postgres=# CREATE OR REPLACE MATERIALIZED VIEW test AS SELECT 2 AS a, 3 AS b; CREATE MATERIALIZED VIEW postgres=# SELECT * FROM test; a | b ---+--- 2 | 3 (1 row) Implementation Details ---------------------- Patch 0001 extends create_ctas_internal in order to adapt an existing matview to the new tuple descriptor, access method, tablespace, and storage parameters. This logic is mostly based on DefineViewRelation. This also reuses checkViewColumns, but adds argument is_matview in order to tell if we want error messages for a matview (true) or view (false). I'm not sure if that flag is the correct way to do that, or if I should just create a separate function just for matviews with the same logic. Do we even need to distinguish between view and matview in those error messages? The patch also adds tab completion in psql for CREATE OR REPLACE MATERIALIZED VIEW. [1] https://www.postgresql.org/message-id/226806.1693430777%40sss.pgh.pa.us [2] https://wiki.postgresql.org/images/a/ad/Materialised_views_now_and_the_future-pgconfeu_2013.pdf#page=23 -- Erik
Вложения
Hi, > Patch 0002 deprecates CREATE MATERIALIZED VIEW IF NOT EXISTS because it > no longer seems necessary with patch 0001. Tom Lane commented[1] about > the general dislike of IF NOT EXISTS, to which I agree, but maybe this > was meant only in response to adding new commands. Anyway, my idea is > to deprecate that usage in PG18 and eventually remove it in PG19, if > there's consensus for it. We can drop that clause without violating any > standard because matviews are a Postgres extension. I'm not married to > the idea, just want to put it on the table for discussion. I can imagine how this may impact many applications and upset many software developers worldwide. Was there even a precedent (in the recent decade or so) when PostgreSQL broke the SQL syntax? To clarify, I'm not opposed to this idea. If we are fine with breaking backward compatibility on the SQL level, this would allow dropping the support of inherited tables some day, a feature that in my humble opinion shouldn't exist (I realize this is another and very debatable question though). I just don't think this is something we ever do in this project. But I admit that this information may be incorrect and/or outdated. -- Best regards, Aleksander Alekseev
> On 2 Jul 2024, at 03:22, Erik Wienhold <ewie@ewie.name> wrote: > Patch 0002 deprecates CREATE MATERIALIZED VIEW IF NOT EXISTS because it > no longer seems necessary with patch 0001. Tom Lane commented[1] about > the general dislike of IF NOT EXISTS, to which I agree, but maybe this > was meant only in response to adding new commands. Anyway, my idea is > to deprecate that usage in PG18 and eventually remove it in PG19, if > there's consensus for it. Considering the runway we typically give for deprecations, that seems like a fairly short timeframe for a SQL level command which isn't unlikely to exist in application code. -- Daniel Gustafsson
I wrote: > Patch 0002 deprecates CREATE MATERIALIZED VIEW IF NOT EXISTS because it > no longer seems necessary with patch 0001. Tom Lane commented[1] about > the general dislike of IF NOT EXISTS, to which I agree, but maybe this > was meant only in response to adding new commands. One could also argue that since matviews are a hybrid of tables and views, that CREATE MATERIALIZED VIEW should accept both OR REPLACE (as in CREATE VIEW) and IF NOT EXISTS (as in CREATE TABLE). But not in the same invocation of course. On 2024-07-02 12:46 +0200, Aleksander Alekseev wrote: > > Anyway, my idea is to deprecate that usage in PG18 and eventually > > remove it in PG19, if there's consensus for it. We can drop that > > clause without violating any standard because matviews are a > > Postgres extension. I'm not married to the idea, just want to put > > it on the table for discussion. > > I can imagine how this may impact many applications and upset many > software developers worldwide. Was there even a precedent (in the > recent decade or so) when PostgreSQL broke the SQL syntax? A quick spelunking through the changelog with git log --grep deprecat -i --since '10 years ago' turned up two commits: 578b229718 "Remove WITH OIDS support, change oid catalog column visibility." e8d016d819 "Remove deprecated COMMENT ON RULE syntax" Both were committed more than 10 years after deprecating the respective feature. My proposed one-year window seems a bit harsh in comparison. On 2024-07-02 14:27 +0200, Daniel Gustafsson wrote: > Considering the runway we typically give for deprecations, that seems like a > fairly short timeframe for a SQL level command which isn't unlikely to exist > in application code. Is there some general agreed upon timeframe, or is decided on a case-by-case basis? I can imagine waiting at least until the last release without the deprecation reaches EOL. That would be 5 years with the current versioning policy. -- Erik
> On 2 Jul 2024, at 15:58, Erik Wienhold <ewie@ewie.name> wrote: > On 2024-07-02 14:27 +0200, Daniel Gustafsson wrote: >> Considering the runway we typically give for deprecations, that seems like a >> fairly short timeframe for a SQL level command which isn't unlikely to exist >> in application code. > > Is there some general agreed upon timeframe, or is decided on a > case-by-case basis? I can imagine waiting at least until the last > release without the deprecation reaches EOL. That would be 5 years with > the current versioning policy. AFAIK it's all decided on a case-by-case basis depending on impact. There are for example the removals you listed, and there are functions in libpq which were deprecated in the postgres 6.x days which are still around to avoid breaking ABI. -- Daniel Gustafsson
Hi,
+1 for this feature.
Replacing Matviews ------------------ With patch 0001, a matview can be replaced without having to drop it and its dependent objects. In our use case it is no longer necessary to define the actual query in a separate view. Replacing a matview works analogous to CREATE OR REPLACE VIEW: * the new query may change SELECT list expressions of existing columns * new columns can be added to the end of the SELECT list * existing columns cannot be renamed * the data type of existing columns cannot be changed In addition to that, CREATE OR REPLACE MATERIALIZED VIEW also replaces access method, tablespace, and storage parameters if specified. The clause WITH [NO] DATA works as expected: it either populates the matview or leaves it in an unscannable state. It is an error to specify both OR REPLACE and IF NOT EXISTS.
I noticed replacing the materialized view is blocking all reads. Is that expected ? Even if there is a unique index ?
Best,
Saïd
On 2024-07-04 22:18 +0200, Said Assemlal wrote: > +1 for this feature. Thanks! > I noticed replacing the materialized view is blocking all reads. Is that > expected ? Even if there is a unique index ? That is expected because AccessExclusiveLock is acquired on the existing matview. This is also the case for CREATE OR REPLACE VIEW. My initial idea, while writing the patch, was that one could replace the matview without populating it and then run the concurrent refresh, like this: CREATE OR REPLACE MATERIALIZED VIEW foo AS ... WITH NO DATA; REFRESH MATERIALIZED VIEW CONCURRENTLY foo; But that won't work because concurrent refresh requires an already populated matview. Right now the patch either populates the replaced matview or leaves it in an unscannable state. Technically, it's also possible to skip the refresh and leave the old data in place, perhaps by specifying WITH *OLD* DATA. New columns would just be null. Of course you can't tell if you got stale data without knowing how the matview was replaced. Thoughts? -- Erik
> That is expected because AccessExclusiveLock is acquired on the existing > matview. This is also the case for CREATE OR REPLACE VIEW. Right, had this case many times. > > My initial idea, while writing the patch, was that one could replace the > matview without populating it and then run the concurrent refresh, like > this: > > CREATE OR REPLACE MATERIALIZED VIEW foo AS ... WITH NO DATA; > REFRESH MATERIALIZED VIEW CONCURRENTLY foo; > > But that won't work because concurrent refresh requires an already > populated matview. > > Right now the patch either populates the replaced matview or leaves it > in an unscannable state. Technically, it's also possible to skip the > refresh and leave the old data in place, perhaps by specifying > WITH *OLD* DATA. New columns would just be null. Of course you can't > tell if you got stale data without knowing how the matview was replaced. > Thoughts? I believe the expectation is to get materialized views updated whenever it gets replaced so likely to confuse users ?
On 2024-07-12 16:49 +0200, Said Assemlal wrote: > > My initial idea, while writing the patch, was that one could replace the > > matview without populating it and then run the concurrent refresh, like > > this: > > > > CREATE OR REPLACE MATERIALIZED VIEW foo AS ... WITH NO DATA; > > REFRESH MATERIALIZED VIEW CONCURRENTLY foo; > > > > But that won't work because concurrent refresh requires an already > > populated matview. > > > > Right now the patch either populates the replaced matview or leaves it > > in an unscannable state. Technically, it's also possible to skip the > > refresh and leave the old data in place, perhaps by specifying > > WITH *OLD* DATA. New columns would just be null. Of course you can't > > tell if you got stale data without knowing how the matview was replaced. > > Thoughts? > > I believe the expectation is to get materialized views updated whenever it > gets replaced so likely to confuse users ? I agree, that could be confusing -- unless it's well documented. The attached 0003 implements WITH OLD DATA and states in the docs that this is intended to be used before a concurrent refresh. Patch 0001 now covers all matview cases in psql's tab completion. I missed some of them with v1. -- Erik