Next, I wanted to understand how popular each of the URLs was by evaluating the number of times it has been shared on Facebook. I've spent a great deal of time in the Facebook developer area, looking through the Facebook Atlas, Graph, and Marketing APIs, and can't find any source for evaluating trending URLs, allowing me to discover things by keyword, and number of shares--maybe I'm missing something, and I"ll keep looking, but nothing stands out as a solution to me currently.
I can only pass a URL to the Facebook Graph API and get back the number of shares, and I can only pull a total of 4800 calls per daily, per 24 hours period per IP address. The pulling of URLs for the 70+ domains can be scaled vertically--meaning I can just purchase a larger Amazon EC2 instance, and scale up the number of calls I make using a single server. When pulling Facebook share details for each url, I am only processing between 3-5 domains, pulling every 5 minutes, per a single Amazon server--which has its own IP address.
Currently, I have 15 separate server instances pulling the Facebook Graph information for each URL, across the 70 domains. As I can afford it I will scale this up or down. Unfortunately defining the sharing graph for each domain, across all its URLs is costly from a compute standpoint. I'm also having to balance the pulling new URLs vs. the pulling of an update for each URL in an attempt to keep track of which URLs are trending upwards or downwards.
I'm pulling the sharing statistics for URLs as fast as I can, with the Facebook API rate limits and resources to pay for Amazon EC2 instances my only bottlenecks. I'll keep pulling and prioritizing this portion of the process in real-time, 24 hours a day. For now it is running at capacity.