FeedBlitz maintenance completed, Friday 30th, 3pm eastern
Thursday, June 29, 2006
Feedblitz's service management application will be offline from about 3pm US eastern for a minor tune up. We'll balance the tires, change the oil, that kind of thing. We'll be back, all things being equal, in less than half an hour. Progress updates wil be noted in this post.Update
- maintenance completed at abut 3:20pm
I suppose it was inevitable...
Wednesday, June 28, 2006
... but today is the first day we have had to block complete domains (and the network that hosts the users responsible) for abuse of the FeedBlitz service and the import facility. A pity.
New email server - Please read If you block by IP
Mail admins, please note. 220.127.116.11 is a new mail server we'll be bringing online this week for better scalability and system performance; please update your records to add this host if you block / permit by IP.
All the brew that's fit to print
Tuesday, June 27, 2006
The Miller Brewing Co.'s Brew Blog
got a brief mention in last week's Advertising Age
(registration required). An industry insider's look at the beer and spirits industry, Brew's email subscription services are powered by FeedBlitz. Emails sent from "Brew" are customized, and the blog also takes advantage of FeedBlitz's new verification question feature
Miller's blog has so far got them a nice piece of PR in an industry publication, plus a lot of mentions in beer-related blogs (that's reaching enthusiasts and their readers
, so those who presumably buy a fair amount of beer, so the blog has also reached the industry's best customers). It's also risen to be the most popular beer blog on FeedBlitz
Congrats to the team making Brew work so well so quickly. Cheers!
FeedBlitz confirmation emails now customized using Pro templates
Monday, June 26, 2006
The one last place where a Pro custom template wasn't being used was the email sent to subscribers confirming their registration.
So we changed that.
As of now, the body of emails that are sent to users confirm subscription requests, or to inform them after an import that their email service has changed, are now formatted using your "Pro" branding.
FeedBlitz adds "newsflash" email broadcast capability
Sometimes you have breaking news that just can't wait to be blogged and picked up in the next scheduled FeedBlitz run. Sometimes you just have to get the news out now, now, now.
And, now, you can, if you're a FeedBlitz premium publisher.
Premium publishers - Pro and Turbo - can now send emails immediately to their subscriber base. If you have a Pro template enabled, the mail is formatted using it. You can send "newsflashes" by clicking the alert icon
that now appears at www.feedblitz.com/f?Lists
The editor is the same one used for Pro template creation, so you can style your message, insert images and more - it's very easy to use. Once you create your text, you preview it in its formatted style and then kick off the email (you can't stop it once it starts, so be really
sure it's ready before you hit that final button!). Once done, you are sent a quick report (by mail of course) of how many subscribers the broadcast was sent to.
Uses for this "out of band" (non-blog derived) communication include:
- Breaking news (e.g. is your boss going to be on TV in an hour? Send a newsflash!).
- Urgent needs or opportunities (e.g. a seat for tomorrow's training class just opened up).
- A way to email your email subscribers only without going via the blog (e.g. communicating a special offer).
Unsubscribe usability improvements
Friday, June 23, 2006
Unsubscribing from an email used to be a single click process. Turns out that it's really just a little too easy to hit the "Stop Receving..." links accidentally, so we've inserted an extra confirmation page for users to validate their removal request. So now it's two clicks, but the risk of accidentally removing oneself is dramatically lower, and a lot more usable - especially for the "Stop all link" which deletes all subscriptions in one fell swoop.
Upgrade and performance update 2
Thursday, June 22, 2006
Earlier this week
I'd said we'd need at least three straight runs to complete well before we thought about declaring victory. Well, Thursday morning's run was the third to go OK, and so all is looking good, as we'd hoped for.
Meanwhile, performance is continuing to exceed my expectations. Not only is www.feedblitz.com
more responsive, but there are small but noticeable improvements in the overnight runs. Bulk update runs are about 15% faster than they were before. They're still too slow, but next week we'll make a major step forward to address that.
So onwards and upwards. Meanwhile, if you have emails that you absolutely want to get out that were written on Sunday the 18th or Wednesday the 14th, please write to support and we'll get them out for you provided they're still in the feed.
Signup verification question added for publications in regulated industries
Tuesday, June 20, 2006
Last week we added a new feature, important to publishers associated with alcohol and other regulated or adult-oriented industries. You can specify a question to ask the subscriber at signup time, at the same time as the image verification test. If a question is specified (it must be a yes / no question, such as "Are you over 21?", or a statement to affirm, such as "I am over 21") the user must also click the checkbox on the subscription screen for their subscription to be accepted. You may also specify the text display if the user attempts to subscribe without validating the question.
It is highly recommended that publishers in the liquor, tobacco, adult entertainment, other regulated spaces or feeds that regularly feature profanity update their subscription options. To use the new option, click your publication's details icon
Migration and performance update
OK, here’s the scoop on the database migration:
- After the new database went live on the 14th it quickly settled down happily.
Batch updates on the night of the 14th went out OK.
- The daily update for the 14th (starting at 1:20 am on the 15th) started fine.
- This run failed, and despite several fixes being deployed was ultimately not sent. Apologies to all.
- Friday morning’s run (for posts written on the 15th) went out successfully, although very late.
- Saturday’s run was delayed as the scheduler didn’t start it while the prior run was under way. It did complete successfully, however, although again late.
- Sunday morning’s run (for posts written on the 17h) went out correctly and on time.
- Fetch me Murphy’s Law! Monday’s run hit a completely unrelated performance flaw that caused it to run extremely slowly. FeedBlitz’s safety protocols terminated the run automatically after 23 hours.
- This morning’s run (for posts written on the 19th) started on time and successfully completed.
Decision day was last Friday, because I wasn’t prepared to allow two consecutive failures. If we hadn’t got the Friday morning run out successfully that day we were going to roll it all back and start over. The fixes worked, however, and the fact that we have had no database issues since – and performance is better - vindicates that decision.
So the bottom line is, simply, unacceptable performance. Ugh. For what it’s worth, the good news is that no database-related issues have surfaced since Friday. Not that that matters to you, of course. Crappy performance is still crappy performance, and not what we’re all used to from FeedBlitz.
So, what’s the plan? What did we learn?
Well, although the upgrade performed great in testing and even better than expected in early production on Wednesday, our testing clearly did not adequately discover the issues we’d face in production. Based on the testing we had done and the system’s performance on Wednesday up to and including the start of the overnight run we had no basis to expect anything as bad as actually happened.
Lesson learned, changes under way.
For future changes of this magnitude we must - and will - do better. Although this has been an ugly few days the net result is a more efficient, more scalable, more flexible FeedBlitz that is now positioned for the next stage of its growth. On which more later, but for now we need to get your confidence back. I’m confident in the changes we’ve made to improve reliability, but won’t declare victory until at least three consecutive overnight runs complete successfully. Any failure will reset the count, regardless of cause.
Meanwhile, a couple of upgrade related fixed to tell you about:
- A defect in the dashboard that caused the grand totals to be incorrect has been fixed.
- A defect that caused some feeds to be added without their titles and descriptions has been corrected.
Good News, Bad News - Updated
Thursday, June 15, 2006
OK, so which would you like first? The good news is that the database upgarde went well and web interactions are fast and scalable. Cool.
The bad news is that under load overnight it suffered teething problems and only the first few thousand mails went out. So we'll be retstarting shortly once we can keep the connection going for the time required.UPDATE
After researching this we've discovered the likely problem (it relates to how database connections are reused by the server software and FeedBlitz's highly parallel architecture). We've put in a series of changes to mitigate these issues. Today's smaller updates have gone out fine, so we'll see how well tonight's run (containing this post) goes.
Tuesday, June 13, 2006
Ah, yes. Database upgrade
. The dread phrase guaranteed to send a chill into the hearts of the most experienced user. And for FeedBlitz, D-day is Wednesday, where we expect to be unavailable several hours as we go through this process.
So, what's going on? Why now? Why in the middle of the day?
Here's why. We have stretched the limit of the current system to breaking point. Performance and reliability issues have been cropping up over the last 10 days or so, and while they've been manifesting themselves in different ways, the core problem is the database FeedBlitz is using. The DB takes a great deal of care and feeding now, and limits our ability to meet our continued growth.
So we're changing it. Simple, really.
A little background. FeedBlitz has been running, despite its growth, on the very same single server and database software it started out on when it was first made public in August last year. That makes for high performance and low latency - no network communications issues, for example - but poor scalability. The current database - well suited for fast development but not for high volume, multiple client applications - neither readily nor efficiently enables multiple remote clients, and is size limited. It has other issues too. So, in short, it gives us no room for maneuver any more. Time for it to go.
This upgrade is, in fact, two significant changes rolled into one. We're adding a dedicated high performance database server, and we're upgrading the software to an industrial strength solution. These two items are linked; we need to upgrade the server in order to move the database onto a separate machine. And by doing that, we are able to then add more polling and application and database servers at will, all of which can be done (almost) seamlessly, without having to take the service down again. Runs will go faster as we deploy more hardware, and we'll be prepped to handle publishers with subscriber counts measured in the tens and hundreds of thousands.
So to the size of the maintenance window and its timing. FeedBlitz does most of its work overnight, the time when most significant software upgrades take place. Our "off hours" are actually the middle of the working day, between nightly runs. Which makes things a little more inconvenient, to be sure, but we must have the update secured and settled before the next nightly run. The size of the maintenance window will be long enough to migrate the very latest data and test it before we take the new database live.
Once this upgrade completes, we will then start to roll in some additional polling resources to spread the work (and so shorten) the main nightly update. Other than that, any unplanned maintenance windows should last no more than a few minutes as the system shakes down. If you see any major issues after the service is restored please let us know at the support address.
Extended Maintenance - Wed, 14 June - Completed
UPDATE 5:43 pm:
Back online - we're now running through the missed pollsUPDATE 4:06 pm:
Data migration completed; final testing under way.UPDATE 2:00 pm:
Maintenance is startingUPDATE
: Maintenance will start at 2pm eastern
FeedBlitz will enter extended maintenance
approximately one hour after the nightly update completes on Wednesday (which, on recent form, will mean around 1pm US eastern). The maintenance is to upgrade the core database used by FeedBlitz, and is expected to take several hours to complete
. We apologize for the inconvenience that this will cause; please plan accordingly.
The work is being performed during the working day in order to assure that it is fully completed in time for the overnight run on Wednesday night / Thursday morning. Once completed, the site will be made available to uses and a catch up program implemented to send any scheduled emails that may have been skipped while the service was unavailable.
For progress updates please track this blog post.
New FeedBlitz Blogs
Thursday, June 01, 2006
© FeedBlitz - Blog and RSS Email Solutions | www.feedblitz.com | firstname.lastname@example.org | Privacy | Terms of Service