Hello,
I'm not sure if I should have posted this to pgsql-advocacy, but this is being developed so I posted here.
Does anyone know if this development come to open source Postgres, or only to the cloud services of Microsoft and
Google?
(I wonder this will become another reason that Postgres won't incorporate optimizer hint feature.)
Data systems that learn to be better
http://news.mit.edu/2020/mit-data-systems-learn-be-better-tsunami-bao-0810
[Quote]
--------------------------------------------------
As a first step toward this vision, Kraska and colleagues developed Tsunami and Bao. Tsunami uses machine learning to
automaticallyre-organize a dataset’s storage layout based on the types of queries that its users make. Tests show that
itcan run queries up to 10 times faster than state-of-the-art systems. What’s more, its datasets can be organized via a
seriesof "learned indexes" that are up to 100 times smaller than the indexes used in traditional systems.
Bao, meanwhile, focuses on improving the efficiency of query optimization through machine learning.
...
Traditional query optimizers take years to build, are very hard to maintain, and, most importantly, do not learn from
theirmistakes. Bao is the first learning-based approach to query optimization that has been fully integrated into the
populardatabase management system PostgreSQL. Lead author Ryan Marcus, a postdoc in Kraska’s group, says that Bao
producesquery plans that run up to 50 percent faster than those created by the PostgreSQL optimizer, meaning that it
couldhelp to significantly reduce the cost of cloud services, like Amazon’s Redshift, that are based on PostgreSQL.
Kraska says that in contrast to other learning-based approaches to query optimization, Bao learns much faster and can
outperformopen-source and commercial optimizers with as little as one hour of training time.In the future, his team
aimsto integrate Bao into cloud systems to improve resource utilization in environments where disk, RAM, and CPU time
arescarce resources.
...
The work was done as part of the Data System and AI Lab (DSAIL@CSAIL), which is sponsored by Intel, Google, Microsoft,
andthe U.S. National Science Foundation.
--------------------------------------------------
Regards
Takayuki Tsunakawa