Обсуждение: Database viewpoint of subject - Sending e-mails from database table with Cronjob

> I generate e-mail messages to a database table and then with a CronJob I
> sent the e-mails.
>
> My doubt is... The CronJob runs every 10 minutes, but If I have 100.000
> e-mails to send the script will not be able to send all the 100.000 e-mails
> in 10 minutes.
>
> How can I deal with this problem? There is no problem to have multiple
> CronJobs runing in background?

If N is the number of mails you can send per cronjob in 10 minutes,
then run 100 000/N cron jobs.
You will be very happy if you also add som kind of "sleep" (for a
growing number of seconds) to the cronjobs when there are currently no
more mails to send. Otherwise you will end up with a very annoying CPU
load.

The other way is to let the cron job spawn new processes (up to a
limited number of child proceses) as long as there are mails to send.
These child processes runs as long as there are mails to send, then
they die. The cron job will then mostly do process controll/start new
processes.

That ought to do the trick.

On Wed, Apr 21, 2010 at 3:41 PM, Andre Lopes <lopes80andre@gmail.com> wrote:
> Hi,
>
> Thanks for the reply.
>
> [quote]
> The other way is to let the cron job spawn new processes (up to a
> limited number of child proceses) as long as there are mails to send.
> These child processes runs as long as there are mails to send, then
> they die. The cron job will then mostly do process controll/start new
> processes.
> [/quote]
>
> How can I do this. I'am thinking to use a PHP Script?
>
> Best Regards,
>
>
> On Wed, Apr 21, 2010 at 12:35 PM, A B <gentosaker@gmail.com> wrote:
>>
>> > I generate e-mail messages to a database table and then with a CronJob I
>> > sent the e-mails.
>> >
>> > My doubt is... The CronJob runs every 10 minutes, but If I have 100.000
>> > e-mails to send the script will not be able to send all the 100.000
>> > e-mails
>> > in 10 minutes.
>> >
>> > How can I deal with this problem? There is no problem to have multiple
>> > CronJobs runing in background?
>>
>> If N is the number of mails you can send per cronjob in 10 minutes,
>> then run 100 000/N cron jobs.
>> You will be very happy if you also add som kind of "sleep" (for a
>> growing number of seconds) to the cronjobs when there are currently no
>> more mails to send. Otherwise you will end up with a very annoying CPU
>> load.
>>
>> The other way is to let the cron job spawn new processes (up to a
>> limited number of child proceses) as long as there are mails to send.
>> These child processes runs as long as there are mails to send, then
>> they die. The cron job will then mostly do process controll/start new
>> processes.
>>
>> That ought to do the trick.
>
>


I really hope this is not for SPAM mail.

Allan.

> [quote]
> The other way is to let the cron job spawn new processes (up to a
> limited number of child proceses) as long as there are mails to send.
> These child processes runs as long as there are mails to send, then
> they die. The cron job will then mostly do process controll/start new
> processes.
> [/quote]
>
> How can I do this. I'am thinking to use a PHP Script?

This is getting a little of topic, but I guess something like
exec('php some_script.php') will work?

In response to A B <gentosaker@gmail.com>:

> > [quote]
> > The other way is to let the cron job spawn new processes (up to a
> > limited number of child proceses) as long as there are mails to send.
> > These child processes runs as long as there are mails to send, then
> > they die. The cron job will then mostly do process controll/start new
> > processes.
> > [/quote]
> >
> > How can I do this. I'am thinking to use a PHP Script?
>
> This is getting a little of topic, but I guess something like
> exec('php some_script.php') will work?

There's a php/pgsql list available on this listserve that will get
you a lot more help in this area.

However, I would suggest PHP's pcntl_fork() and friends for creating
multiple processes.  Of course, that's only needed if you want to
run them in parallel, which was unclear to me.

--
Bill Moran
http://www.potentialtech.com
http://people.collaborativefusion.com/~wmoran/

Andre Lopes <lopes80andre@gmail.com> wrote:

> My doubt is... The CronJob runs every 10 minutes, but If I have 100.000
> e-mails to send the script will not be able to send all the 100.000 e-mails
> in 10 minutes.

More generally, your question is how to deal with a cron job that is
intended to run every ten minutes, but which you don't want to run more
than one copy of (i.e. this has little or nothing to do with PostgreSQL,
really).

My usual solution is twofold:

1. have the cron job take a lock file as the first thing it does: this
   ensures only one job can run at a time

   Something like FreeBSD's lockf(1) is good for this, and isn't hard to
   port or write a similar utility for any OS I've had to use:

   http://www.freebsd.org/cgi/man.cgi?query=lockf&apropos=0&sektion=1&manpath=FreeBSD+8.0-RELEASE&format=html

2. include in the job a 'dead man switch' that terminates the job
   if it runs for "too long", whatever "too long" is in your context.

   Obviously, 10 minutes is possible; 10 hours is probably not.

   Making sure client applications terminate (be they cron jobs, CGI
   scripts, or anything else) stops all your database connections
   being tied up by hung scripts, to bring the discussion slightly
   back to PostgreSQL).

> How can I deal with this problem? There is no problem to have multiple
> CronJobs runing in background?

Multiple cron jobs in general, no.  Multiple instances of a single
cron job depends on the cron job.

As another poster wrote, I do hope this isn't for spam, as you say it
isn't, but the basics I'd follow are #1 and #2 above, and let the mail
system handle both the queue and the allowable amount of parallel
outbound SMTP connections.  That's what mail software is for.

Of course, I'd _definitely_ not use any such thing for spamming....

Regards,

Giles