Обсуждение: Proposal: Job Scheduler

Поиск
Список
Период
Сортировка

Proposal: Job Scheduler

От
"Wang Cheng"
Дата:
Hackers,

We are the PostgreSQL team in Tencent. We have recently developed a job scheduler that runs inside the database to schedules and manages jobs similar to Oracle DBMS_JOB package, and we would like to contribute this feature to the community.

Similar to autovacuum, the job scheduler consists of 2 parts: the job launcher and the job worker. The job launcher periodically scans a metadata table and signals the postmaster to start new workers if needed.

As far as we know, there are currently two open-sourced job scheduling extensions for PostgreSQL: pg_cron (https://github.com/citusdata/pg_cron/) and pg_dbms_job (https://github.com/MigOpsRepos/pg_dbms_job/tree/main). However, the cron-based syntax is not easy to use and suffers some limitations like one-off commands. The pg_dbms_job extension is difficult to manage and operate because it runs as a standalone process .

That's why we have developed the job scheduler that runs as a process inside the database just like autovacuum.

We can start to send the patch if this idea makes sense to the you. Thanks for your time.


Regards,
Cheng
 

Re: Proposal: Job Scheduler

От
Laurenz Albe
Дата:
On Thu, 2024-06-06 at 16:27 +0800, Wang Cheng wrote:
> We are the PostgreSQL team in Tencent. We have recently developed a job scheduler
> that runs inside the database to schedules and manages jobs similar to Oracle
> DBMS_JOB package, and we would like to contribute this feature to the community.
>
> As far as we know, there are currently two open-sourced job scheduling extensions
> for PostgreSQL: pg_cron (https://github.com/citusdata/pg_cron/) and pg_dbms_job
> (https://github.com/MigOpsRepos/pg_dbms_job/tree/main). However, the cron-based
> syntax is not easy to use and suffers some limitations like one-off commands.
> The pg_dbms_job extension is difficult to manage and operate because it runs as
> a standalone process .

There is also pg_timetable:
https://github.com/cybertec-postgresql/pg_timetable

> That's why we have developed the job scheduler that runs as a process inside the
> database just like autovacuum.
>
> We can start to send the patch if this idea makes sense to the you.

Perhaps your job scheduler is much better than all the existing ones.
But what would be a compelling reason to keep it in the PostgreSQL source tree?
With PostgreSQL's extensibility features, it should be possible to write your
job scheduler as an extension and maintain it outside the PostgreSQL source.

I am sure that the PostgreSQL community will be happy to use the extension
if it is any good.

Yours,
Laurenz Albe



Re: Proposal: Job Scheduler

От
Dave Page
Дата:


On Thu, 6 Jun 2024 at 09:47, Laurenz Albe <laurenz.albe@cybertec.at> wrote:
On Thu, 2024-06-06 at 16:27 +0800, Wang Cheng wrote:
> We are the PostgreSQL team in Tencent. We have recently developed a job scheduler
> that runs inside the database to schedules and manages jobs similar to Oracle
> DBMS_JOB package, and we would like to contribute this feature to the community.
>
> As far as we know, there are currently two open-sourced job scheduling extensions
> for PostgreSQL: pg_cron (https://github.com/citusdata/pg_cron/) and pg_dbms_job
> (https://github.com/MigOpsRepos/pg_dbms_job/tree/main). However, the cron-based
> syntax is not easy to use and suffers some limitations like one-off commands.
> The pg_dbms_job extension is difficult to manage and operate because it runs as
> a standalone process .

There is also pg_timetable:
https://github.com/cybertec-postgresql/pg_timetable

And probably the oldest of them all, pgAgent: https://www.pgadmin.org/docs/pgadmin4/8.7/pgagent.html
 


> That's why we have developed the job scheduler that runs as a process inside the
> database just like autovacuum.
>
> We can start to send the patch if this idea makes sense to the you.

Perhaps your job scheduler is much better than all the existing ones.
But what would be a compelling reason to keep it in the PostgreSQL source tree?
With PostgreSQL's extensibility features, it should be possible to write your
job scheduler as an extension and maintain it outside the PostgreSQL source.

I am sure that the PostgreSQL community will be happy to use the extension
if it is any good.

I agree. This is an area in which there are lots of options at the moment, with compelling reasons to choose from various of them depending on your needs.

It's this kind of choice that means it's unlikely we'd include any one option in PostgreSQL, much like various other tools such as failover managers or poolers. 

--

Re: Proposal: Job Scheduler

От
"Wang Cheng"
Дата:
Noted. Thanks for suggestions. We will open-source it as an extension.


Regards,
Cheng
 


------------------ Original ------------------
From: "Dave Page" <dpage@pgadmin.org>;
Date: Thu, Jun 6, 2024 04:59 PM
To: "Laurenz Albe"<laurenz.albe@cybertec.at>;
Cc: "Wang Cheng"<348448708@qq.com>;"pgsql-hackers"<pgsql-hackers@lists.postgresql.org>;
Subject: Re: Proposal: Job Scheduler



On Thu, 6 Jun 2024 at 09:47, Laurenz Albe <laurenz.albe@cybertec.at> wrote:
On Thu, 2024-06-06 at 16:27 +0800, Wang Cheng wrote:
> We are the PostgreSQL team in Tencent. We have recently developed a job scheduler
> that runs inside the database to schedules and manages jobs similar to Oracle
> DBMS_JOB package, and we would like to contribute this feature to the community.
>
> As far as we know, there are currently two open-sourced job scheduling extensions
> for PostgreSQL: pg_cron (https://github.com/citusdata/pg_cron/) and pg_dbms_job
> (https://github.com/MigOpsRepos/pg_dbms_job/tree/main). However, the cron-based
> syntax is not easy to use and suffers some limitations like one-off commands.
> The pg_dbms_job extension is difficult to manage and operate because it runs as
> a standalone process .

There is also pg_timetable:
https://github.com/cybertec-postgresql/pg_timetable

And probably the oldest of them all, pgAgent: https://www.pgadmin.org/docs/pgadmin4/8.7/pgagent.html
 


> That's why we have developed the job scheduler that runs as a process inside the
> database just like autovacuum.
>
> We can start to send the patch if this idea makes sense to the you.

Perhaps your job scheduler is much better than all the existing ones.
But what would be a compelling reason to keep it in the PostgreSQL source tree?
With PostgreSQL's extensibility features, it should be possible to write your
job scheduler as an extension and maintain it outside the PostgreSQL source.

I am sure that the PostgreSQL community will be happy to use the extension
if it is any good.

I agree. This is an area in which there are lots of options at the moment, with compelling reasons to choose from various of them depending on your needs.

It's this kind of choice that means it's unlikely we'd include any one option in PostgreSQL, much like various other tools such as failover managers or poolers. 

--

Re: Proposal: Job Scheduler

От
Andrei Lepikhov
Дата:
On 6/6/2024 16:04, Wang Cheng wrote:
> Noted. Thanks for suggestions. We will open-source it as an extension.
It would be nice! `For me doesn't matter where to contribute: to 
PostgreSQL core or to its extension if it is published under BSD license.

-- 
regards, Andrei Lepikhov




Re: Proposal: Job Scheduler

От
Alvaro Herrera
Дата:
On 2024-Jun-06, Dave Page wrote:

> It's this kind of choice that means it's unlikely we'd include any one
> option in PostgreSQL, much like various other tools such as failover
> managers or poolers.

TBH I see that more as a bug than as a feature, and I see the fact that
there are so many schedulers as a process failure.  If we could have
_one_ scheduler in core that encompassed all the important features of
all the independent ones we have, with hooks or whatever to allow the
user to add any fringe features they need, that would probably lead to
less duplicative code and divergent UIs, and would be better for users
overall.

That's, of course, just my personal opinion.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/



Re: Proposal: Job Scheduler

От
Dmitry Dolgov
Дата:
> On Thu, Jun 06, 2024 at 12:53:38PM GMT, Alvaro Herrera wrote:
> On 2024-Jun-06, Dave Page wrote:
>
> > It's this kind of choice that means it's unlikely we'd include any one
> > option in PostgreSQL, much like various other tools such as failover
> > managers or poolers.
>
> TBH I see that more as a bug than as a feature, and I see the fact that
> there are so many schedulers as a process failure.  If we could have
> _one_ scheduler in core that encompassed all the important features of
> all the independent ones we have, with hooks or whatever to allow the
> user to add any fringe features they need, that would probably lead to
> less duplicative code and divergent UIs, and would be better for users
> overall.
>
> That's, of course, just my personal opinion.

+1. The PostgreSQL ecosystem is surprisingly fragmented, when it comes
to quite essential components that happen to be outside of the core. But
of course it doesn't mean that there should be _one_ component of every
kind in core, more like it makes sense to have _one_ component available
out of the box (where the box is whatever form of PostgreSQL that gets
delivered to users, e.g. a distro package, container, etc.).