Wednesday, April 23, 2008
Well, "the next nice problem to have" (see the end of my prior post
) didn't take too long to show up, did it? I'm sure there's a lesson here about Fate, Temptation, and not mixing the two together.
So, having made the web site a whole bunch spiffier it took, what, a little over a couple of days for the back end poller to get stuck in the weeds. The root cause was the queries we use to build the jobs on each server that divvy up the bulk mailings, slowing down due to the ever growing number of subscribers and subscriptions we're managing. In the end the slow down was enough to create a kind of "traffic jam" in the DB which, in turn, pretty much slowed everything else down too. A positive feedback effect which only served to amplify the problem. So, not a failure of the improvements we made last week; rather a timely reminder that there are many components in an application stack and that a problem in any one of them affects performance as a whole.
Briefly, then, we've modified our data architecture to slim down one of the three main queries used in each batch. So far so good, in that FeedBlitz caught up with the backlog relatively quickly this evening (Wednesday); we'll see how things perform under the stress of an overnight run. There's still more to be done on this front, but this ought to go a long way to bringing bulk mail performance (and all the other interactions that depend on a sprightly database) back to where they used to be. Thanks for your patience on this one...
Faster, FeedBlitz, Faster!
Friday, April 18, 2008
You ever heard the phrase "that's a nice problem to have"? It's the kind of problem like having too much money (I've yet to experience this one :-) ), or - for an Internet service - so much traffic that your site is brought to its knees. Congratulations on being successful, and by the way your site is down.
A nice problem is still, well, a problem.
So last week we weren't crushed, but FeedBlitz has been much busier than usual, and performance has been, at times (let's be charitable), sluggish. Why? We had the "nice problem" of seeing usage ramp up significantly. On April 15th we sent nearly 3.1 million messages, easily our busiest day ever. Our next busiest day was the day before, at over 2.4 million messages. I haven't written a monthly update for a while, but March's output was 20% greater than February's, at nearly 48 million messages sent. So far this month we've sent 32 million already, so we're on track to blow away March's record as well, probably by a similar amount. That's the "nice" part. But the cost of all this goodness was performance started to deteriorate on the web site. That would be the "problem" side of the equation.
The good news is that as this hit, we were already working on performance improvements knowing that Gawker was about to go live
. Gawker being a very busy site, we didn't want them to go live and then have us, their shiny new email service, go and disappear under all the traffic that they might send our way. As it turns out, the timing was coincidental; we just hit a critical mass of publishers around the same time that pushed our servers too close to the edge for comfort. But we were ready - and I'm glad we were.
We added an extra web server and an extra mail server to handle the load better, but (more importantly) we have made other changes that have now made the site much snappier to use and the mailings much less stressful to our infrastructure. We're supporting more subscribers, publishers and visitors with less effort than 10 days ago. FeedBlitz is now much faster and much more responsive than before (possibly ever, in fact).
So far, so good. If you're no nerdling then please disembark this post now as we have reached your destination. Otherwise, read on to find out what we did beyond enabling the extra iron.Amazon S3
One of the biggest challenges we were facing last week, as it turns out, was not that our servers running out of steam per se
- they all had CPU, bandwidth, memory and disk to spare. What we were starting to run out of was sockets
for Internet connectivity. Between the WWW access, mail deliveries, TCP timeouts and HTTP keep-alives (see, I told you this would be techie) some of the servers were occasionally exhausting their socket pools, effectively taking them offline for a few seconds during busy times of the day, apparently randomly. Adding extra servers helped, to be sure, but doing so only defers the problem, and is also a relatively expensive, high-maintenance fix. We needed a more long-term solution to managing our day to day WWW usage.
And we found it. A little late to the party, perhaps, but over the last 7 days we have offloaded all the work that doesn't reflect our core value-add but is important to the web site, such as image and script file serving, to Amazon's S3
service. Our static images and scripts are now served from the cloud, and none from our servers. This is the proverbial win-win situation, as Amazon serves the images faster than we ever could, which in turn makes feedblitz.com much more responsive as well as improving our connectivity. So we solved our immediate need and
got snappier site performance in the bargain.
Most of our pages now need exactly one socket access per page, to pull down the core HTML; all the other baggage comes from S3, saving about 3 connections per web site visitor, or saving us 75%. And right now it's costing us a whopping ~$3 a day for what is, effectively, an infinitely large image / asset server
. It's serving our images more cheaply, more quickly, more reliably and much more manageably than we ever could or can using a self-managed dedicated server.
Obviously, I'm a fan. If you're running a site or service that is going to get big, I'm now of the opinion that you're nuts not to outsource to S3 or a similar service to store and serve objects that aren't core to your value add. It's faster, better and cheaper and whole lot less hassle. Do it!
That said, those in the know will no doubt be asking, what about the complementary Amazon services, EC2 and Simple DB? Well, we're not such a good fit there. Our code doesn't work on the EC2 infrastructure, so that's a non-starter. There may in fact be a role for SDB in future service offerings, but we've no plans to move our current database over to it right now. I will
say this, though: If we were to be creating FeedBlitz now, there's no doubt in my mind that I'd use SDB (or a similar service) and S3 as the back end from the get go.FastCGI
I also mentioned that we're not a good fit for EC2 because of the way our application is built. Won't bore you with the details. Up until this week it delivered the goods to the WWW via an old back-end protocol called CGI. Which worked nicely, but isn't known for its performance. This week we deployed new versions of the core HTML newsletter application and our email ad server using FastCGI, which (as the name implies) is like CGI, only faster. Since we're a custom app, it wasn't trivial to add FastCGI to what we do, but now we have and it's working a treat. It's reduced the load on our systems all along the stack and increased both throughput and responsiveness.
So there you have it: some re-engineering (FastCGI), selective outsourcing (S3) and extra servers have made the pain go away. Until the next nice problem to have, anyway ...
Labels: Amazon S3, FastCGI, FeedBlitz
Wednesday, April 16, 2008
" to Stefano Cazzani
for our new Italian subscriber pages! Dutch will be next.
If you'd like to volunteer to translate our subscriber pages (about 280 strings) into your language, please write to us at the support address.
Labels: FeedBlitz, Translation
Branding your Messages: New Template Tags
Monday, April 14, 2008
In yesterday's post
I remarked that Gawker's sites were using our standard template capabilities to customize their emails and subscriber landing pages. I also mentioned that we'd added a few new tags to the template; as promised here's a description.What's a template again?
The template controls the layout (styles, fonts, logos etc.) of your content. It controls how
your articles are formatted. This is contrast to the newsletter settings links in the newsletter center, which control what
goes into the article in the first place. So the newsletter settings control things like tracking, truncation, comments and forward to a friend settings. The template takes the results and makes it all pretty for you.
The template editor is found in the graphic design area of your newsletter center. The template controls the layout by using custom tags inside HTML; the custom tags act as variables, substituting content such as your blog's title, content or time stamp into the HTML that becomes your newsletter or subscription form.New Tags
There are several new tag options, now available to everyone, that we built for Gawker and a couple of other premium clients. The three main new tags are:
- <$Date format="formatstring"$>
OK, so what do these do?<$BlogFloatImagesLeft$>
The "FloatImage" tags take any image they find in the content and do their level best to force it to the left or right margins, wrapping your article text around it. They make the text more pleasing to the eye, and save space as well.
Look at Lifehacker's (one of the Gawker sites) newsletter preview
, and now look at the RSS feed the newsletter is built from
. In the RSS, the image floats all alone, above the text. It wastes a lot of space. In the newsletter, without the <$BlogFloatImagesLeft$> tag, it would behave exactly the same way, which is not how Gawker wanted things to look. With the tag in place, you get what you see now - the image on the left, and the text all cozied up to it, nice and friendly. If you use this tag, know that it works on all
images that emerge from the content engine. So it works well if you only have one image, or their well spaced apart. If you use lots of pictures in each article, things can get bunched up and ugly. So preview it before you take it live!<$Date format="formatstring"$>
This ought to be a popular one! You can now control the date formatting FeedBlitz puts in (still English and still US eastern, one thing at a time, people...). Programmers will know what to do with this, and you can see the results in the Lifehacker newsletter
. The date in the mast head, under the logo, is the time the newsletter was generated (so that's now in the online preview). 3 letter day name, 3 letter month, date, year. Now, if you look at each article, you'll see the time it was posted (FeedBlitz pulls this information from the RSS). Hours and minutes are added. (Both are upper case because of the styles that are applied to the dates once generated).
Both dates use the new <$Date$> syntax, but with different format strings. Here are some format strings and what they do for the time February 9, 2008, 2:45 PM
|<$Date format="%a, %b %d %Y, %I:%M %p"$>||Sat, Feb 09 2008, 2:45 P.M.|
|<$Date format="%B %d %Y, %H:%M"$>||February 09 2008, 14:45|
|<$Date format="%A - %H:%M"$>||Saturday - 14:45|
There are a couple more new tags but I suspect they'll be rarely used; you can find them all listed under the template editor itself. Access your template editor using the "graphic design" link in your Newsletter Center.
Gawker Media Weekly News Updates
Sunday, April 13, 2008
Media, the company that runs some of the best-known blogs ever
, last week started to offer weekly updates of the top 5 stories via FeedBlitz
. You can subscribe to Gawker's
weekly newsletters from any page on a Gawker
site using the form at the end of the contacts section in the left hand column. Now they're up & running, newsletter management and delivery is completely automated
, giving each site's fans another way to increase their engagement with their favorite online media resource.
site extensively of customizes the newsletter and subscriber pages, to parallel their parent web sites. All this is achieved using FeedBlitz
standard templates (although, to be fair, we added some features to accommodate Gawker's
needs, but these capabilities are now available to everyone; more in a later post).
Click on the preview links below to see the current online version, and click the "subscribe" links to see how the subscription forms are laid out with each sites branding. Might as well subscribe, while you're there :-) You can click through to the main site from the logo at the top of any of these pages. (N.B. If you're not familiar with Gawker
sites, be aware that they are well written but often irreverent
, somewhat profane, sometimes politically incorrect and frequently deal with topics you might not want to discuss with your mother. Which is why they're so popular, of course, but don't say you weren't warned...).
My main point is this. Look at the art of the possible that these newsletters show can be built with FeedBlitz, and consider: Is your brand - personal, professional, corporate - being carried through to your email conversations like these? And, seeing these newsletters, do you think you could do more to reinforce your identity?
All is achieved with the FeedBlitz template capabilities, accessed from the "Graphic Design" section of your Newsletter Center. If you are currently using the FeedBlitz default layout (friendly orange logo in the upper left), take a look. You can customize subject lines, from names and addresses, and (as these Gawker titles show) almost everything about the look and feel of your HTML newsletters and subscriber landing pages. All publishers can customize their template; you don't have to be a premium (paying) publisher to do this.
I hope that we've inspired you to revisit your emails look and feel. Tomorrow I'll write in more detail about how the templates were constructed and the new HTML newsletter template capbilities we now offer.
Need more convincing? Here are some more designs that have been built and used in FeedBlitz mailings.
Labels: consumerist, deadspin, defamer, FeedBlitz, gawker, gizmodo, idolator, io9, jalopnik, jezebel, kotaku, lifehacker, valleywag, wonkette
Spanish Subscriber Pages Added
Thursday, April 10, 2008
We now have Spanish subscriber pages, thanks to David Readman
. These complement our French
translations. Italian will be live shortly, and Dutch is en route. Meanwhile, we're still looking for a volunteer to translate into German. Come on, Herren und Damen
, don't be shy!
FeedBlitz en Français!
Monday, April 07, 2008
Following last week's release of our Portuguese subscriber pages
, I'm delighted to be able to thank Pierre Carnicelli
(visit his site at http://www.carnicelli.eu/
) for our newly released French version. We have the Spanish spreadsheet already out to a volunteer, but that still leaves many languages out there. Want to have a go?
We can also handle non-latin
character sets, so we can support translations into asian
and other non-European scripts. So if you want to volunteer, email me at phil
.com (but wait for our new inbound email server
to be up and running first!).
Labels: FeedBlitz, Translation
April 7 2008 - Sending Mail to FeedBlitz (resolved)
Update 3:25pm EDT
- Inbound email is back on a shiny new server. We're working through the backlog now, so please be patient while we get around to you. If your mail to us bounced please re-send it; it should now work ok
inbound mail server is currently experiencing problems, so mail sent TO FeedBlitz
may be delayed or even bounce. A new server is being set up and we expect the situation to return to normal later today. Please check back for updates so you can find out when to resend your email if necessary. We apologize for any inconvenience.
Newsletter deliveries, publisher features and subscriber management functions are NOT affected and continuing as normal.
FeedBlitz Subscriber Pages Translated for Brazilian Readers
Tuesday, April 01, 2008
A brief note to remark that our subscriber facing pages, such as the pages used in the subscription and unsubscribe processes, are now available in Brazilian Portuguese. We have some very large lists from Brazil, and with this change we'll make FeedBlitz much more accessible to Brazilian subscribers.
FeedBlitz automatically detects Portuguese subscribers, but visitors can also override the language selection using the links at the foot of the relevant pages. My thanks to Marcelo Vitorino over at http://www.insightpublicidade.com.br/
for his help actually doing the translating.Want your language?
If you would like to volunteer to translate the FeedBlitz subscriber pages into your language, drop me a note at phil @ you know where. There are about 250 strings to translate, and we will reward you with publicity in the FeedBlitz blog like this and in the footer of the translated pages for as long as we use your text. It's entirely voluntary, but if you'd like to help for your language write in. The first one to send back a completed spreadsheet wins :-)
Labels: FeedBlitz, Translation
© FeedBlitz - Blog and RSS Email Solutions | www.feedblitz.com | firstname.lastname@example.org | Privacy | Terms of Service