|
Wednesday, April 08, 2009
Yet again, Twitter is burning about funkiness with FeedBurner subscriber counts cratering. The situation has gone on for too long and I now feel I have to write something about this, especially as (a) nobody from Google / FeedBurner is stepping up to the plate, and (b) enough, already.
Full disclosure: Since we compete with FeedBurner, feel free to use whatever size pinch of salt you deem necessary as you read.
How Subscriber Counts are Calculated
First a backgrounder. When you read your feed, it's been fetched from the source (such as FeedBurner or FeedBlitz's new FeedBurner alternative) and delivered to you by a piece of software: your browser; your online RSS service (such as Google Reader or Bloglines or FeedBlitz's email service); your desktop browser (such as FeedDemon) or a search engine or something else. Think of these different pieces of software as vehicles loading up on newspapers at your newspaper's printing plant. Some, like Google Reader, are big trucks and take lots of newspapers for delivery to lots of subscribers; some are just individual subscribers (e.g. individual FeedDemon users) in their virtual cars taking just their own copy.
When these aggregators visit FeedBurner or Feedblitz, we have to decide a few things when figuring out your subscriber count. Firstly, is it an aggregator / feed reader at all, or simply a search engine? This matters because search engines aren't directly contributing to circulation, so they don't affect subscriber counts just because they're scanning your feed for others to find later. The hit is noted but that's about it.
If it's an aggregator reading the feed, however, then bulk aggregators will (usually) report the number of subscribers they're working on behalf of using what's called the HTTP user agent header. So, in a sense, what that tells us is how big the truck is that just loaded up at our virtual newspaper press and who runs it. If on any one day that aggregator truck backs up several times, we'll take the maximum reported number of subscribers that day as the number reported for your feed.
For other desktop aggregators it's a little more involved, but at the end of the day FeedBurner and FeedBlitz total up the numbers and ta-daa: your daily subscriber count.
Problem #1 - Sampling
The thing is, both services report a daily subscriber count. A day is midnight to midnight in the relevant time zone (for FeedBlitz it's US eastern time and for FeedBurner I believe it's still US central time). So, for example, say an aggregator checks your feed just once every 24 hours (ish). Say that one day this aggregator checks your feed at 11:59pm (i.e. just before midnight) on day 1, not at all on day 2, and then at 12:01 am on day 3. That aggregator has, more or less, checked your feed once every 24 hours. But because it was a minute "early" one day and minute "late" the next there's a whole "day" where the truck didn't show up at all for the purposes of calculating your circulation. Result: zero subscribers reported from that aggregator on day 2.
This is a sampling error, in that the aggregator in question wasn't present in the sample used to produce your subscriber counts for day 2.
What you see (at least, in FeedBurner - we do things a little differently at FeedBlitz and I'll get to that later) is a cratering subscriber count and a good business day for coronary emergency departments around the globe. (OK, just kidding about that part).
In practice, most aggregators check feeds once every 30-60 minutes, so this kind of thing actually doesn't happen very often in the real world.
Problem #2 - Broken User Agents
Sometimes aggregators don't report their subscriber count at all (this has been a problem recently with Google's FeedFetcher agent, which pulls feeds on behalf of Google Reader and iGoogle). Sometimes user agents misreport the subscriber count. Our very own FeedBlitz's email service was guilty of this (*cough*) for some publishers a couple of days early last year.
The numbers reported by the user agents are what RSS services use, for better and for worse, to derive your circulation. GIGO (garbage in, garbage out) holds true here just as it does anywhere else.
Why I believe this does NOT explain FeedBurner's recent issues
[If you want to hold your nose because we compete with FeedBurner then here's where you want to do it]
Belatedly FeedBurner reported an issue with FeedFetcher. Basically they're saying "it's not us (FeedBurner), it's them (Google Reader's FeedFetcher)" - or, in other words, this is a case of problem #2 above. Nothing to see here, folks, move along, it'll be fixed soon.
I don't buy it.
Don't get me wrong. The Google Reader agent (FeedFetcher) was hosed a couple of times recently. Here are the Google FeedFetcher numbers for http://feed.feedblitz.com/feedblitz from the last 10 days as reported by the FeedBlitz logs:
So what you see is that Google failed to report numbers to us on two days - March 30th and April 2nd. (The large jump on 4/5 is that we at FeedBlitz discovered that, because of yet another quirk in Google FeedFetcher, that FeedBlitz was actually under-reporting Google's numbers, which we fixed for reports from 4/5 forward).
Blaming FeedFetcher only cuts the mustard as far as I'm concerned for problems with subscriber counts reported for 3/30 and 4/2. Since numbers are reported the day after, FeedBurner feed subscriber counts should have burped on 3/31 and 4/3 and then bounced back on 4/1 and 4/4. All should now be well again.
But it isn't. They're apparently off - again (or should that be still?)- today. Which begs the question that Google has yet to answer: WTF is going on over there?
Beats me.
I suppose there could be a reasonable explanation for this. Here are a few I thought of.
Perhaps FeedFetcher is using some sneaky internal-to-Google route to get to feedburner feeds now they (IMHO completely unnecessarily) moved everyone to Google domains, are using different code than they are for external feed fetching and so the FeedBurner is missing subscriber counts that everyone else is actually seeing.
Feels unlikely, doesn't it?
Perhaps it's sampling error combined with a broken agent then? That 1 hour difference between the FeedBurner and FeedBlitz "days." Perhaps FeedFetcher happened to stop reporting numbers after 1am eastern on all the FeedBurner problem days, so FeedBlitz saw them OK (between 12am and 1am) and FeedBurner didn't (because their day starts at 1am eastern), so there were more "days" when the count was zero for FeedBurner than for FeedBlitz?
I suppose it's at least possible. So I did a quick check. Looking at our logs, Google FeedFetcher reported susbcribers correctly on March 29th, obviously didn't on March 30th, but DID report numbers OK between midnight and 1am on the morning of March 31. So based on FeedBlitz's logs from the first incident I can't see the evidence for a sampling error problem either. (I didn't check the April date).
So perhaps it's just that FeedBlitz is new, and because we aren't hosting nearly as many feeds as FeedBurner (yet...). Perhaps the problem is not global to all servers running FeedFetcher but just to a few of the machines, and so we've merely been lucky to avoid seeing the problem. Well, OK, this might have some legs. But, OTOH if this is the case, it raises serious questions about FeedFetcher's QA and Google's roll out of new code into production. It's not as if the Google crew is short of money to fund these things properly. So, hmmm.
Perhaps someone from Google would care to comment here. For anti-spam reasons I moderate comments to this blog, but I commit to publish anything from Google here if someone shows up. I'll edit this post at noon eastern tomorrow (4/9) with any updates, errors or corrections based on these interactions, should they occur. If I'm wrong I'm more than happy to admit it and make the appropriate correction.
[Update: Noon April 9th - My update is in the comments to this post. Nobody from Google or FeedBurner has commented so far, and counts are broken again.]
Managing Sampling Errors and Bad Agent Strings Better
There's a more subtle question here, though. Knowing that sh*t happens, how can an RSS service avoid the yo-yo effect when subscribers come and go and user agent strings can get messed up? The goal here after all is to give you, the RSS feed publisher, the best insight into your readership.
I can't speak for FeedBurner but what we at FeedBlitz do is this:
1) In the reports we give you the unadjusted data for any given day. We report the readerships and so, if one of the "trucks" doesn't back up in that 24 hours timespan, it won't show up in that day's metrics.
2) But ... In the circulation chicklet we report a moving 3-day average. This smooths out any sampling errors, minimizes the impact of broken user-agent strings, and gives you a much clearer insight as to the true trend of your circulation. The dailies are always visible to you (see (1) above), but we believe that a moving average gives comparable accuracy with much better stability and, therefore, better results for you.
RSS Circulation, not Subscribers
Since we're on metrics, here's something else I believe that FeedBlitz does differently from FeedBurner. FeedBurner is an RSS company; they care about RSS susbcribers and that's what they report in the chicklets and on your dashboard. From what I've read it is my understanding (and, again, I'm prepared to be corrected on this) that FeedBurner does not count browsers in the RSS subscriber count.
In other words, if someone views your feed in their browser but does not subscribe (using either the browser's internal subscription mechanism or some other feed reader) then I believe that they aren't counted in your feed's total subscriber count from FeedBurner.
But, hang on a minute. These visitors are reading your feed content. So, if my assumption is correct, FeedBurner isn't counting them in your RSS subscriber count because they're readers, not subscribers. But because they're not on your web site (they're on FeedBurner's), your web analytics package isn't counting them either.
Yikes. There's a potentially significant population of "ghost" feed readers that nobody's accurately tracking. Continuing with my newspaper analogy, a subscriber who views your feed in their browser but doesn't subscribe is equivalent to someone who buys their copy of the newspaper at the news-stand, gas station, or the convenience store. They're not a subscriber to that newspaper, but they are part of the newspaper's circulation.
FeedBlitz tracks and counts these readers, because we're a social media marketing company. We care about all your readers no matter how they consume your content. Non-subscribing visitors are still viewing your content, getting your message, thinking about what you're saying. Such readers are therefore part of your feed's circulation too.
FeedBlitz therefore adds these readers to build a circulation (not a subscriber) metric. It turns out to be statistically significant. For the Feedblitz News feed yesterday (4/7/09) some 2% of FeedBlitz News' total circulation were in this non-subscribing category. If you exclude the FeedBlitz News subscribers managed by our email delivery service, this casual count is in fact the same order of magnitude as RSS subscribers from other (not FeedBlitz email) aggregators. That's an important number to understand and not to miss.
The $1.49 Question
So. If you're reading this far you probably care a great deal about your RSS subscribers and how they're reported. Perhaps you're really hacked off with FeedBurner's persistent volatility and unresponsiveness. Perhaps you're really, really annoyed with yo-yo metrics and, like me, don't understand why this should be so.
OK. How pissed off are you, exactly? If you thought about it in monetary terms, is your continuing annoyance and frustration greater than our minimum $1.49 /month fee? Or, more positively, is it worth at least $1.49 / month for you to have reliable, trustworthy metrics and a supported service?
Really? It is? Then start our 30-day trial at http://www.feedblitz.com/ (click RSS - New) so you can focus on what you do best and stop worrying about your metrics. I think we have a better service, but if metrics are what really get your blood boiling then, like I said at the beginning of this post:
Enough already.
Do something about it.
Switch to FeedBlitz.
Full disclosure: Since we compete with FeedBurner, feel free to use whatever size pinch of salt you deem necessary as you read.
How Subscriber Counts are Calculated
First a backgrounder. When you read your feed, it's been fetched from the source (such as FeedBurner or FeedBlitz's new FeedBurner alternative) and delivered to you by a piece of software: your browser; your online RSS service (such as Google Reader or Bloglines or FeedBlitz's email service); your desktop browser (such as FeedDemon) or a search engine or something else. Think of these different pieces of software as vehicles loading up on newspapers at your newspaper's printing plant. Some, like Google Reader, are big trucks and take lots of newspapers for delivery to lots of subscribers; some are just individual subscribers (e.g. individual FeedDemon users) in their virtual cars taking just their own copy.
When these aggregators visit FeedBurner or Feedblitz, we have to decide a few things when figuring out your subscriber count. Firstly, is it an aggregator / feed reader at all, or simply a search engine? This matters because search engines aren't directly contributing to circulation, so they don't affect subscriber counts just because they're scanning your feed for others to find later. The hit is noted but that's about it.
If it's an aggregator reading the feed, however, then bulk aggregators will (usually) report the number of subscribers they're working on behalf of using what's called the HTTP user agent header. So, in a sense, what that tells us is how big the truck is that just loaded up at our virtual newspaper press and who runs it. If on any one day that aggregator truck backs up several times, we'll take the maximum reported number of subscribers that day as the number reported for your feed.
For other desktop aggregators it's a little more involved, but at the end of the day FeedBurner and FeedBlitz total up the numbers and ta-daa: your daily subscriber count.
Problem #1 - Sampling
The thing is, both services report a daily subscriber count. A day is midnight to midnight in the relevant time zone (for FeedBlitz it's US eastern time and for FeedBurner I believe it's still US central time). So, for example, say an aggregator checks your feed just once every 24 hours (ish). Say that one day this aggregator checks your feed at 11:59pm (i.e. just before midnight) on day 1, not at all on day 2, and then at 12:01 am on day 3. That aggregator has, more or less, checked your feed once every 24 hours. But because it was a minute "early" one day and minute "late" the next there's a whole "day" where the truck didn't show up at all for the purposes of calculating your circulation. Result: zero subscribers reported from that aggregator on day 2.
This is a sampling error, in that the aggregator in question wasn't present in the sample used to produce your subscriber counts for day 2.
What you see (at least, in FeedBurner - we do things a little differently at FeedBlitz and I'll get to that later) is a cratering subscriber count and a good business day for coronary emergency departments around the globe. (OK, just kidding about that part).
In practice, most aggregators check feeds once every 30-60 minutes, so this kind of thing actually doesn't happen very often in the real world.
Problem #2 - Broken User Agents
Sometimes aggregators don't report their subscriber count at all (this has been a problem recently with Google's FeedFetcher agent, which pulls feeds on behalf of Google Reader and iGoogle). Sometimes user agents misreport the subscriber count. Our very own FeedBlitz's email service was guilty of this (*cough*) for some publishers a couple of days early last year.
The numbers reported by the user agents are what RSS services use, for better and for worse, to derive your circulation. GIGO (garbage in, garbage out) holds true here just as it does anywhere else.
Why I believe this does NOT explain FeedBurner's recent issues
[If you want to hold your nose because we compete with FeedBurner then here's where you want to do it]
Belatedly FeedBurner reported an issue with FeedFetcher. Basically they're saying "it's not us (FeedBurner), it's them (Google Reader's FeedFetcher)" - or, in other words, this is a case of problem #2 above. Nothing to see here, folks, move along, it'll be fixed soon.
I don't buy it.
Don't get me wrong. The Google Reader agent (FeedFetcher) was hosed a couple of times recently. Here are the Google FeedFetcher numbers for http://feed.feedblitz.com/feedblitz from the last 10 days as reported by the FeedBlitz logs:
| Date | FeedFetcher Count |
| 3/28/2009 | 429 |
| 3/29/2009 | 429 |
| 3/30/2009 | 0 |
| 3/31/2009 | 428 |
| 4/1/2009 | 428 |
| 4/2/2009 | 0 |
| 4/3/2009 | 424 |
| 4/4/2009 | 424 |
| 4/5/2009 | 834 |
| 4/6/2009 | 834 |
| 4/7/2009 | 834 |
So what you see is that Google failed to report numbers to us on two days - March 30th and April 2nd. (The large jump on 4/5 is that we at FeedBlitz discovered that, because of yet another quirk in Google FeedFetcher, that FeedBlitz was actually under-reporting Google's numbers, which we fixed for reports from 4/5 forward).
Blaming FeedFetcher only cuts the mustard as far as I'm concerned for problems with subscriber counts reported for 3/30 and 4/2. Since numbers are reported the day after, FeedBurner feed subscriber counts should have burped on 3/31 and 4/3 and then bounced back on 4/1 and 4/4. All should now be well again.
But it isn't. They're apparently off - again (or should that be still?)- today. Which begs the question that Google has yet to answer: WTF is going on over there?
Beats me.
I suppose there could be a reasonable explanation for this. Here are a few I thought of.
Perhaps FeedFetcher is using some sneaky internal-to-Google route to get to feedburner feeds now they (IMHO completely unnecessarily) moved everyone to Google domains, are using different code than they are for external feed fetching and so the FeedBurner is missing subscriber counts that everyone else is actually seeing.
Feels unlikely, doesn't it?
Perhaps it's sampling error combined with a broken agent then? That 1 hour difference between the FeedBurner and FeedBlitz "days." Perhaps FeedFetcher happened to stop reporting numbers after 1am eastern on all the FeedBurner problem days, so FeedBlitz saw them OK (between 12am and 1am) and FeedBurner didn't (because their day starts at 1am eastern), so there were more "days" when the count was zero for FeedBurner than for FeedBlitz?
I suppose it's at least possible. So I did a quick check. Looking at our logs, Google FeedFetcher reported susbcribers correctly on March 29th, obviously didn't on March 30th, but DID report numbers OK between midnight and 1am on the morning of March 31. So based on FeedBlitz's logs from the first incident I can't see the evidence for a sampling error problem either. (I didn't check the April date).
So perhaps it's just that FeedBlitz is new, and because we aren't hosting nearly as many feeds as FeedBurner (yet...). Perhaps the problem is not global to all servers running FeedFetcher but just to a few of the machines, and so we've merely been lucky to avoid seeing the problem. Well, OK, this might have some legs. But, OTOH if this is the case, it raises serious questions about FeedFetcher's QA and Google's roll out of new code into production. It's not as if the Google crew is short of money to fund these things properly. So, hmmm.
Perhaps someone from Google would care to comment here. For anti-spam reasons I moderate comments to this blog, but I commit to publish anything from Google here if someone shows up. I'll edit this post at noon eastern tomorrow (4/9) with any updates, errors or corrections based on these interactions, should they occur. If I'm wrong I'm more than happy to admit it and make the appropriate correction.
[Update: Noon April 9th - My update is in the comments to this post. Nobody from Google or FeedBurner has commented so far, and counts are broken again.]
Managing Sampling Errors and Bad Agent Strings Better
There's a more subtle question here, though. Knowing that sh*t happens, how can an RSS service avoid the yo-yo effect when subscribers come and go and user agent strings can get messed up? The goal here after all is to give you, the RSS feed publisher, the best insight into your readership.
I can't speak for FeedBurner but what we at FeedBlitz do is this:
1) In the reports we give you the unadjusted data for any given day. We report the readerships and so, if one of the "trucks" doesn't back up in that 24 hours timespan, it won't show up in that day's metrics.
2) But ... In the circulation chicklet we report a moving 3-day average. This smooths out any sampling errors, minimizes the impact of broken user-agent strings, and gives you a much clearer insight as to the true trend of your circulation. The dailies are always visible to you (see (1) above), but we believe that a moving average gives comparable accuracy with much better stability and, therefore, better results for you.
RSS Circulation, not Subscribers
Since we're on metrics, here's something else I believe that FeedBlitz does differently from FeedBurner. FeedBurner is an RSS company; they care about RSS susbcribers and that's what they report in the chicklets and on your dashboard. From what I've read it is my understanding (and, again, I'm prepared to be corrected on this) that FeedBurner does not count browsers in the RSS subscriber count.
In other words, if someone views your feed in their browser but does not subscribe (using either the browser's internal subscription mechanism or some other feed reader) then I believe that they aren't counted in your feed's total subscriber count from FeedBurner.
But, hang on a minute. These visitors are reading your feed content. So, if my assumption is correct, FeedBurner isn't counting them in your RSS subscriber count because they're readers, not subscribers. But because they're not on your web site (they're on FeedBurner's), your web analytics package isn't counting them either.
Yikes. There's a potentially significant population of "ghost" feed readers that nobody's accurately tracking. Continuing with my newspaper analogy, a subscriber who views your feed in their browser but doesn't subscribe is equivalent to someone who buys their copy of the newspaper at the news-stand, gas station, or the convenience store. They're not a subscriber to that newspaper, but they are part of the newspaper's circulation.
FeedBlitz tracks and counts these readers, because we're a social media marketing company. We care about all your readers no matter how they consume your content. Non-subscribing visitors are still viewing your content, getting your message, thinking about what you're saying. Such readers are therefore part of your feed's circulation too.
FeedBlitz therefore adds these readers to build a circulation (not a subscriber) metric. It turns out to be statistically significant. For the Feedblitz News feed yesterday (4/7/09) some 2% of FeedBlitz News' total circulation were in this non-subscribing category. If you exclude the FeedBlitz News subscribers managed by our email delivery service, this casual count is in fact the same order of magnitude as RSS subscribers from other (not FeedBlitz email) aggregators. That's an important number to understand and not to miss.
The $1.49 Question
So. If you're reading this far you probably care a great deal about your RSS subscribers and how they're reported. Perhaps you're really hacked off with FeedBurner's persistent volatility and unresponsiveness. Perhaps you're really, really annoyed with yo-yo metrics and, like me, don't understand why this should be so.
OK. How pissed off are you, exactly? If you thought about it in monetary terms, is your continuing annoyance and frustration greater than our minimum $1.49 /month fee? Or, more positively, is it worth at least $1.49 / month for you to have reliable, trustworthy metrics and a supported service?
Really? It is? Then start our 30-day trial at http://www.feedblitz.com/ (click RSS - New) so you can focus on what you do best and stop worrying about your metrics. I think we have a better service, but if metrics are what really get your blood boiling then, like I said at the beginning of this post:
Enough already.
Do something about it.
Switch to FeedBlitz.
Labels: FeedBlitz, feedburner, metrics, RSS, subscriber counts
|
|
posted at 9:44 AM
9 comments
