#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2011
    Posts
    198
    Rep Power
    3

    Emails - Cron or no Cron


    Hi, i have built a custom site using Codeigniter.

    My site allows users to post a story to the site... then other users can comment on the story.

    There is an option under the users Settings to choose whether you want to be notified by email if:

    A - MY STORY
    [ ] a user comments on my story

    B - MY COMMENTS ON OTHER USER'S STORIES
    [ ] another user comments on a story that i have commented on


    I don't think i need any special script or service to handle email A, as it is just one email to one user, however Email B is a different story as 1000 users might have commented on a story and then 1000 users might need to receive an email each time a new user comments on the same story they have commented on, so:

    1 - is there a 3rd party service that i should use in conjunction with codeigniter to handle these bulk emails

    2 - should i process all of these batch emails using a cron... if so can someone recommend the best way to run this process

    Thanks in advance for your help...
  2. #2
  3. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2011
    Posts
    105
    Rep Power
    51
    You can run a script in the background using proc_open or shell_exec.

    This person has the exact same situation as you do.
    Post to Facebook using a Nokia 3310
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2011
    Posts
    198
    Rep Power
    3
    Cool, some people mentioned using SendGrid
  6. #4
  7. Mad Scientist
    Devshed Expert (3500 - 3999 posts)

    Join Date
    Oct 2007
    Location
    North Yorkshire, UK
    Posts
    3,661
    Rep Power
    4123
    I go for the mail queue option. A cron runs looking at the mail queue.

    The main app logic just populates the mail queue.

    this way the expensive task of sending out the emails runs in a separate process the the user visiting the site and commenting
    I said I didn't like ORM!!! <?php $this->model->update($this->request->resources[0])->set($this->request->getData())->getData('count'); ?>

    PDO vs mysql_* functions: Find a Migration Guide Here

    [ Xeneco - T'interweb Development ] - [ Are you a Help Vampire? ] - [ Read The manual! ] - [ W3 methods - GET, POST, etc ] - [ Web Design Hell ]
  8. #5
  9. No Profile Picture
    Contributing User
    Devshed Intermediate (1500 - 1999 posts)

    Join Date
    Sep 2006
    Posts
    1,917
    Rep Power
    533
    Originally Posted by Northie
    I go for the mail queue option. A cron runs looking at the mail queue.

    The main app logic just populates the mail queue.

    this way the expensive task of sending out the emails runs in a separate process the the user visiting the site and commenting
    Rhytz's solution will also make it a separate process. I would probably go this route since you won't need extra tables to hold the queue, however, either will work well.
  10. #6
  11. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2011
    Posts
    198
    Rep Power
    3
    Ok, thanks... so are you saying there is no need to go with a 3rd party email service provider
  12. #7
  13. No Profile Picture
    Contributing User
    Devshed Intermediate (1500 - 1999 posts)

    Join Date
    Sep 2006
    Posts
    1,917
    Rep Power
    533
    Originally Posted by oo7ml
    Ok, thanks... so are you saying there is no need to go with a 3rd party email service provider
    No need, but you might want to consider using PHPMailer just for sending your normal emails.
  14. #8
  15. Mad Scientist
    Devshed Expert (3500 - 3999 posts)

    Join Date
    Oct 2007
    Location
    North Yorkshire, UK
    Posts
    3,661
    Rep Power
    4123
    There's no need to...it depends on your requirements.

    Part of what we do where we work is to design, build and broadcast email marketing campaigns for our clients. Some of our clients have lists with only a few thousand email addresses on it, others have hundreds of thousands in their lists.

    This started to take a toll on our dedicated box which is was primarily being used to host client's websites and transactional mail

    when we got to a stage where our server was sending out too many bulk emails that it interfered with transactional mail we started to look for an alternative.

    We considered sendgrid, but then figured out we could just do what sendgrid do - but a lot cheaper, and just for us - so we set up a series of VPSs with postfix installed to run as a relay server, locked it down to our dedicated server's IP (where our email marketing web app runs from) and there we had it - our own private network of relay servers, each one costing about 10 month for hosting and bandwidth. I recken each server is capable of sending out 5 million emails per week at full capacity (add a few for extra bandwidth) and each server can be resized (currently 0.01/hour for 256Mb server; could be 0.02 for 512Mb, doubling each time up to 32 GB ram)

    If a new client comes on board with huge lists then we create a new VPS just for them, so IP reputation can be contained / limited and as the general volume grows we can resize the VPS as necessary.

    We use PHPMailer's SMTP mode to connect to our remote VPSs

    Sending out 1000 emails takes a few minutes using a mysql based mail queue (looping over the list and sending our through PHPMailer). Our cron sends out 300 / minute. However, these are full HTML marketing emails. For notication emails like devshed send out you could probably increase this limit quite a bit.

    The main software we use is actually 3rd party and partly closed source but when I rebuild it for the new version of our software I will not be limiting the number of emails sent out on the cron. Instead the script will have a time limit on it, and each time the script runs it will grap a few records, send these out, time the sending and keep an eye on the remaining time, then die gracefully before the script time limit - meaning that email delivery is not halted part way through delivery and also be "multi-threaded" by having the time limit greater than the cron interval
    Last edited by Northie; January 11th, 2013 at 10:18 AM.
    I said I didn't like ORM!!! <?php $this->model->update($this->request->resources[0])->set($this->request->getData())->getData('count'); ?>

    PDO vs mysql_* functions: Find a Migration Guide Here

    [ Xeneco - T'interweb Development ] - [ Are you a Help Vampire? ] - [ Read The manual! ] - [ W3 methods - GET, POST, etc ] - [ Web Design Hell ]
  16. #9
  17. No Profile Picture
    Lost in code
    Devshed Supreme Being (6500+ posts)

    Join Date
    Dec 2004
    Posts
    8,317
    Rep Power
    7170
    I recommend using a queue+cron over a background process because it allows you to throttle outgoing mail more easily and is more robust. Sending mail is an expensive task, so if you're sending out more than one about email in a request you need to use one method or the other.

    There are two primary problems with using a background task:

    1) If you spawn a new background task every time situation B occurs, you might end up with many backgrounds tasks all trying to send thousands of emails at once. You could bog down the server easily.

    2) If your background task has sent 500 of 1000 emails and your server crashes, you lose the last 500 emails. With a queue+cron, you don't.


    If you're going to be sending thousands of emails I recommend that you do go with a mail server provider like SendGrid. What Northie said in his last post is true, but most companies are not big enough that it is economical for them to do what Northie has done. Correctly sysadmin'ing a mail server is a lot of work.
    PHP FAQ

    Originally Posted by Spad
    Ah USB, the only rectangular connector where you have to make 3 attempts before you get it the right way around
  18. #10
  19. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2011
    Posts
    198
    Rep Power
    3
    Thanks guys, i think SendGrid is my best option, thanks...

IMN logo majestic logo threadwatch logo seochat tools logo