Asynchronous execution on FDW

Поиск
Список
Период
Сортировка
От Kyotaro HORIGUCHI
Тема Asynchronous execution on FDW
Дата
Msg-id 20150702.144824.266400086.horiguchi.kyotaro@lab.ntt.co.jp
обсуждение исходный текст
Ответы Re: Asynchronous execution on FDW  (Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>)
Re: Asynchronous execution on FDW  (Heikki Linnakangas <hlinnaka@iki.fi>)
Список pgsql-hackers
Hello. This is the new version of FDW async exection feature.

The status of this feature is as follows, as of the last commitfest.

- Async execution is valuable to have.
- But do the first kick in ExecInit phase is wrong.

So the design outline of this version is as following,

- The patch set consists of three parts. The fist is the infrastracture in core-side, second is the code to enable
asynchronousexecution of Postgres-FDW. The third part is the alternative set of three methods to adapt fetch size,
whichmakes asynchronous execution more effective.
 

- It was a problem when to give the first kick for async exec. It is not in ExecInit phase, and ExecProc phase does not
fit,too. An extra phase ExecPreProc or something is too invasive. So I tried "pre-exec callback".
 
 Any init-node can register callbacks on their turn, then the registerd callbacks are called just before ExecProc phase
inexecutor. The first patch adds functions and structs to enable this.
 

- The second part is not changed from the previous version. Add PgFdwConn as a extended PgConn which have some members
tosupport asynchronous execution.
 
 The asynchronous execution is kicked only for the first ForeignScan node on the same foreign server. And the state
lastsuntil the next scan comes. This behavior is mainly controlled in fetch_more_data(). The behavior limits the number
ofsimultaneous exection for one foreign server to 1. This behavior is decided from the reason that no reasonable method
tolimit multiplicity of execution on *single peer* was found so far.
 

- The third part is three kind of trials of adaptive fetch size feature.
  The first one is duration-based adaptation. The patch  increases the fetch size by every FETCH execution but try to
keepthe duration of every FETCH below 500 ms. But it is not  promising because it looks very unstable, or the behavior
is nearly unforeseeable..
 
  The second one is based on byte-based FETCH feature. This  patch adds to FETCH command an argument to limit the
numberof  bytes (octets) to send. But this might be a over-exposure of  the internals. The size is counted based on
internal representation of a tuple and the client is needed to send the  overhead of its internal tuple representation
inbytes. This  is effective but quite ugly..
 
  The third is the most simple and straight-forward way, that  is, adds a foreign table option to specify the
fetch_size.The  effect of this is also in doubt since the size of tuples for  one foreign table would vary according to
thereturn-columns  list. But it is foreseeable for users and is a necessary knob  for those who want to tune it.
Foreignserver also could have  the same option as the default for that for foreign tables but  this patch have not
addedit.
 


The attached patches are the following,

- 0001-Add-infrastructure-of-pre-execution-callbacks.patch Infrastructure of pre-execution callback

- 0002-Allow-asynchronous-remote-query-of-postgres_fdw.patch FDW asynchronous execution feature

- 0003a-Add-experimental-POC-adaptive-fetch-size-feature.patch Adaptive fetch size alternative 1: duration based
control

- 0003b-POC-Experimental-fetch_by_size-feature.patch Adaptive fetch size alternative 2: FETCH by size

- 0003c-Add-foreign-table-option-to-set-fetch-size.patch Adaptive fetch size alternative 3: Foreign table option.

regards,

В списке pgsql-hackers по дате отправления:

Предыдущее
От: Guillaume Lelarge
Дата:
Сообщение: Re: [PROPOSAL] VACUUM Progress Checker.
Следующее
От: Kyotaro HORIGUCHI
Дата:
Сообщение: Re: Asynchronous execution on FDW