#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2005
    Posts
    125
    Rep Power
    10

    Help with Popular and Trending pages.


    Hey all,

    Just wondering if anyone could explain to me the best option for calculating Popular and Tredning pages based on visits and the most efficient way to store and calculate visits.

    I have a site where users create pages. What I want to do is have a list of popular pages and a list of trending pages.

    Popular seems easy enough. But its collecting accurate page view information that seems the problem. I cant think of a better way then logging to a db every time the page is viewed. I just thought that seems like a waste of a db call.

    Also I just wanted to know are these normally calculated by unique views?

    I saw someone suggest storing the views value in memcache and then update the db roughly every x page views. But I wasn't sure how to take unique views into account in this matter.

    Then there is the issue of calculating trending pages. I guess it would be over a 24hr period.

    I was just wondering what is the best way to achieve these. With fairly accurate view numbers.

    Thanks in advance
  2. #2
  3. Did you steal it?
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    13,988
    Rep Power
    9397
    You use the term "popular" and "trending" without really knowing what they mean to you. How would you define them? By number of views? Average number of views over a period? That's your first problem.

    As for recording the views, the easiest method is most certainly storing those counts in the database with the pages. Querying that is trivial.
    If you wanted the memcached route, which only really helps if you have very high traffic, you'd have to periodically import those values into the database. Like with a cronjob.
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2005
    Posts
    125
    Rep Power
    10
    Thanks for the reply. The best example I could find to illustrate what I am trying to achieve is digg.com. How they calculate their popular category is what I consider trending. The most-shared stories on Digg in the last 18 hours. I'm not sure how to best calculate the most popular pages over a given amount of time.

    Any ideas?
  6. #4
  7. Did you steal it?
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    13,988
    Rep Power
    9397
    I forgot to mention: you'd need to track the individual views. So you know when they were; an overall total wouldn't be enough.

    Once you get those into a database it's as simple as
    Code:
    SELECT page, COUNT(1)
    FROM page table
    JOIN views table ON page ID = view's page ID
    WHERE view is in the time range you want to consider
  8. #5
  9. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2005
    Posts
    125
    Rep Power
    10
    Originally Posted by requinix
    I forgot to mention: you'd need to track the individual views. So you know when they were; an overall total wouldn't be enough.

    Once you get those into a database it's as simple as
    Code:
    SELECT page, COUNT(1)
    FROM page table
    JOIN views table ON page ID = view's page ID
    WHERE view is in the time range you want to consider
    Ahh cool that should work for the popular scenario. But is the concept for trending pages different? I figured it would be based on hits per minute or something larger on my scale? Is that easy calculate based on logging all visits to pages?
  10. #6
  11. Sarcky
    Devshed Supreme Being (6500+ posts)

    Join Date
    Oct 2006
    Location
    Pennsylvania, USA
    Posts
    10,850
    Rep Power
    6351
    You need to define "trending" on your own. It's not a mathematical term like "average" or "most." What's "trending?" What makes sense in your environment and for your userbase? Twitter defines trending as terms which have had a very sharp increase in overall usage over the last day. Imgur defines trending as images which have been hit directly (referrals outside of imgur) the most in the last week, with more-recent views being weighted more heavily. How do YOU define "trending"? Do that in the code.
    HEY! YOU! Read the New User Guide and Forum Rules

    "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin

    "The greatest tragedy of this changing society is that people who never knew what it was like before will simply assume that this is the way things are supposed to be." -2600 Magazine, Fall 2002

    Think we're being rude? Maybe you asked a bad question or you're a Help Vampire. Trying to argue intelligently? Please read this.
  12. #7
  13. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2005
    Posts
    125
    Rep Power
    10
    Originally Posted by ManiacDan
    You need to define "trending" on your own. It's not a mathematical term like "average" or "most." What's "trending?" What makes sense in your environment and for your userbase? Twitter defines trending as terms which have had a very sharp increase in overall usage over the last day. Imgur defines trending as images which have been hit directly (referrals outside of imgur) the most in the last week, with more-recent views being weighted more heavily. How do YOU define "trending"? Do that in the code.
    Yeah that's understood. I think I have an idea now of how I plan on doing it. I think I want to base it on the largest spike in traffic in the last day. Do you think its best to calculate that by number of hits that day or the frequency of hits. eg i could have 1 post that has 3 hits for the day evenly spaced or one post that has 3 hits for the day but 1 minute apart and more recently. Im just looking for opinions.
  14. #8
  15. Sarcky
    Devshed Supreme Being (6500+ posts)

    Join Date
    Oct 2006
    Location
    Pennsylvania, USA
    Posts
    10,850
    Rep Power
    6351
    3 hits? With such a small delta size this whole argument doesn't really matter. I would just do the most-hit in the last 24 hours.
    HEY! YOU! Read the New User Guide and Forum Rules

    "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin

    "The greatest tragedy of this changing society is that people who never knew what it was like before will simply assume that this is the way things are supposed to be." -2600 Magazine, Fall 2002

    Think we're being rude? Maybe you asked a bad question or you're a Help Vampire. Trying to argue intelligently? Please read this.
  16. #9
  17. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2005
    Posts
    125
    Rep Power
    10
    Originally Posted by ManiacDan
    3 hits? With such a small delta size this whole argument doesn't really matter. I would just do the most-hit in the last 24 hours.
    I was just using smaller data as an example. It has much more traffic then that. But the algorithm has to work regardless of amount of hits. One page could have 50k hits a day another could have 3.

    Just think about that example with 50k hits. Both achieve it in one day but one was in an hour and the other which took 24 hrs.

    At one particular point in the day one was trending much faster then the other.

    I want to list the fastest trending pages at the time the user is viewing the page.
  18. #10
  19. Sarcky
    Devshed Supreme Being (6500+ posts)

    Join Date
    Oct 2006
    Location
    Pennsylvania, USA
    Posts
    10,850
    Rep Power
    6351
    Then make your algorithm a weighted one. Views in the last hour times 3, plus views in the last day times 2, plus views in the last week. Or something.
    HEY! YOU! Read the New User Guide and Forum Rules

    "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin

    "The greatest tragedy of this changing society is that people who never knew what it was like before will simply assume that this is the way things are supposed to be." -2600 Magazine, Fall 2002

    Think we're being rude? Maybe you asked a bad question or you're a Help Vampire. Trying to argue intelligently? Please read this.
  20. #11
  21. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jul 2005
    Posts
    125
    Rep Power
    10
    That makes perfect sense. Thanks.

IMN logo majestic logo threadwatch logo seochat tools logo