Re: group locking: incomplete patch, just for discussion

On 31 October 2014 04:42, Amit Kapila <amit.kapila16@gmail.com> wrote:

>> In fact it would be more sensible to lock the toast table earlier.
>>
>
> It might make some sense to lock the toast table earlier for this
> particular case, but I don't think in general it will be feasible to lock
> all the tables (including catalog tables) which might get used in the
> overall operation before starting parallel operation.  I believe it will
> make doing parallel stuff difficult if we try to go via this route, as
> we need to always do this for any operation we want to make parallel.
> It will always have the risk that we might miss something and another
> thing is it might not be feasible in all kind of cases.
>
> If I understand correctly, you are suggesting that to get the first version
> of parallel operation out earlier, we should try to avoid all the stuff
> (like group locking, ...),

I advocate trying to avoid it, to see. The problems cited so far are
not so extreme they cannot be easily got around, if you have a will to
do so.

I've just posted about an idea to reduce catalog locking.

> have some restrictions on which kind of cases
> Create Index can work, do some hackery in Create Index path to avoid
> any kind of locking problems and do the parallel operation for External
> Sort (which might avoid need for shared memory allocator) and then
> call it a day and then do the things which we need for other parallel
> operations as and when they are required.

> I think this might be a good
> approach in itself if somebody wants to target something to release
> early, however in medium to short term when we want to provide
> non-trivial parallel operations we have to eventually solve those problems
> and delaying will only make the things worse.

My (by now) long experience on Postgres has shown me that developing
something now is a good route to take. By moving towards a solution at
a reasonable pace you learn things which may affect the longer term
viability of the project. New functionality in each release is useful.
Big bang developments that don't deliver until the end have a habit of
not delivering at the end either. "MVP" is a term frequently used in
VCs for a reason.

Lack of urgency in delivering a useful outcome from the project looks
strange. If all you develop in this release is core infrastructure for
parallelism and a custom scan API, it begins to look like there may
not be an open source version delivered at all. How people make their
income is up to them, but its a concern that's been raised my way
before, so it seems reasonable to do so for others also. That is why
we have spent much time explaining clearly the goals and licencing of
BDR, for example. No need for big arguments; these are not allegations
or implications etc..

> In short, I agree that there is a merit in your idea w.r.t getting the first
> version of parallel operation out early, however if we see from medium
> to long term, I feel solving group locking problem (or some other general
> infrastructure what Robert mentioned upthread) can yield better results,
> unless you feel that is not at all required for parallel operations.

Is it genuinely required for most parallel operations? I think it's
clear that none of us knows the answer. Sure, the general case needs
it, but is the general case the same thing as the reasonably common
case?

Certainly, having a working prototype for something useful would help
build the case for why these things are needed. It would certainly
help the cause to have something working in this release.

-- Simon Riggs                   http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services