On Tue, Nov 17, 2020 at 6:56 PM Etsuro Fujita <etsuro.fujita@gmail.com> wrote:
> * I haven't yet done anything about the issue on postgres_fdw's
> handling of concurrent data fetches by multiple ForeignScan nodes
> (below *different* Append nodes in the query) using the same
> connection discussed in [2]. I modified the patch to just disable
> applying this feature to problematic test cases in the postgres_fdw
> regression tests, by a new GUC enable_async_append.
A solution for the issue would be a scheduler designed to handle such
data fetches more efficiently, but I don’t think it’s easy to create
such a scheduler. Rather than doing so, I'd like to propose to allow
FDWs to disable async execution of them in problematic cases by
themselves during executor startup in the first cut. What I have in
mind for that is:
1) For an FDW that has async-capable ForeignScan(s), we allow the FDW
to record, for each of the async-capable and non-async-capable
ForeignScan(s), the information on a connection to be used for the
ForeignScan into EState during BeginForeignScan().
2) After doing ExecProcNode() to each SubPlan and the main query tree
in InitPlan(), we give the FDW a chance to a) reconsider, for each of
the async-capable ForeignScan(s), whether the ForeignScan can be
executed asynchronously as planned, based on the information stored
into EState in #1, and then b) disable async execution of the
ForeignScan if not.
#1 and #2 would be done after initial partition pruning, so more
async-capable ForeignScans would be executed asynchronously, if other
async-capable ForeignScans conflicting with them are removed by that
pruning.
This wouldn’t prevent us from adding a feature like what was proposed
by Horiguchi-san later.
BTW: while considering this, I noticed some bugs with
ExecAppendAsyncBegin() in the previous patch. Attached is a new
version of the patch fixing them. I also tweaked some comments a
little bit.
Best regards,
Etsuro Fujita