In Flink 1.12, Flink introduced a new concept which is called versioned table, It is very similar with temporal table function,but I am kind of confused between these two concepts.
temporal table function only supports append-only table(Please correct me if I am wrong),such as:
(changelog kind) update_time currency rate
================ ============= ========= ====
+(INSERT) 09:00:00 Yen 102
+(INSERT) 09:00:00 Euro 114
+(INSERT) 09:00:00 USD 1
+(INSERT) 11:15:00 Euro 119
+(INSERT) 11:49:00 Pounds 108
Can we call the above append-only table also a versioned table, From the official documentation it looks the answer is NO? . But I think it can be called a versioned table since it has kept the change history, and I can get the version(snapshot) of the table for the given time point (this is the confusing part to me)
In the following link, it is converting the append-only table to a versioned table by defining a versioned-table-views, with this additional conversion, it looks that append-only table is not versioned table?
Could you please explain about what is versioned table and the difference between versioned table and temporal table function. The official documentation doesn't explain very clearly.
-- Define a versioned view
CREATE VIEW versioned_rates AS
SELECT currency, rate, update_time -- (1) `update_time` keeps the event time
FROM (
SELECT *,
ROW_NUMBER() OVER (PARTITION BY currency -- (2) the inferred unique key `currency` can be a primary key
ORDER BY update_time DESC) AS rownum
FROM currency_rates)
WHERE rownum = 1;
-- the view `versioned_rates` will produce a changelog as the following.
(changelog kind) update_time currency rate
================ ============= ========= ====
+(INSERT) 09:00:00 Yen 102
+(INSERT) 09:00:00 Euro 114
+(INSERT) 09:00:00 USD 1
+(UPDATE_AFTER) 10:45:00 Euro 116
+(UPDATE_AFTER) 11:15:00 Euro 119
+(INSERT) 11:49:00 Pounds 108
You can view Temporal Table Function as a previous version of some sorts of versioned tables (This is also stated in the legacy features part of Flink).
In your first example, you create an append only table which will hold multiple keys of the same currency (i.e. EURO), which makes it ineligible to be a primary key.
Can you call an append only table a versioned table? The definition on Versioned Tables in doc says:
Flink SQL can define versioned tables over any dynamic table with a PRIMARY KEY constraint and time attribute.
This implies that an append only table cannot serve as a versioned table since it can't, as we said, hold true the primary key constraint.
But, since the append only table in your example does hold the relevant information to become a versioned table, containing inserts as well as updates and deletes, we can turn it to one with a deduplication query, as you posted above.