Meh.

Well, "the next nice problem to have" (see the end of my prior post) didn't take too long to show up, did it? I'm sure there's a lesson here about Fate, Temptation, and not mixing the two together.

So, having made the web site a whole bunch spiffier it took, what, a little over a couple of days for the back end poller to get stuck in the weeds. The root cause was the queries we use to build the jobs on each server that divvy up the bulk mailings, slowing down due to the ever growing number of subscribers and subscriptions we're managing. In the end the slow down was enough to create a kind of "traffic jam" in the DB which, in turn, pretty much slowed everything else down too. A positive feedback effect which only served to amplify the problem. So, not a failure of the improvements we made last week; rather a timely reminder that there are many components in an application stack and that a problem in any one of them affects performance as a whole.

Briefly, then, we've modified our data architecture to slim down one of the three main queries used in each batch. So far so good, in that FeedBlitz caught up with the backlog relatively quickly this evening (Wednesday); we'll see how things perform under the stress of an overnight run. There's still more to be done on this front, but this ought to go a long way to bringing bulk mail performance (and all the other interactions that depend on a sprightly database) back to where they used to be. Thanks for your patience on this one...