Apple now posting near-daily MobileMe outage updates

[AppleInsider]

Taking the matter head-on, Apple is now fully acknowledging the problems it's had with MobileMe in the past two weeks as well as the attempts to fix it -- to the point where company chief Steve Jobs is involved.

An anonymous Apple worker has been tasked by Jobs with providing multiple updates per week on a dedicated status page that explain at least some of why the company's new push data service has failed and what has been fixed.

"In the 14 days since we launched, it’s been a rocky road and we know the pain some people have been suffering," says the employee. "Be assured people here are working 24-7 to improve matters, and we’re going to favor getting you new info hot off the presses even if we have to post corrections or further updates later."

For its first update, the company maintains that a "serious problem" with its email servers shut out access to about one percent of MobileMe accounts but now says that it has given some of those accounts web access to their messages and that a complete restoration should be completed in one week, or the start of August. The worker does, however, warn that as much as 10 percent of the messages delivered between July 16th and July 18th will have been lost.

While not providing full details, the staffer does also add that many of the company's problems with the service were triggered by a surge of data traffic on the launch of iPhone 3G and the iPhone 2.0 firmware that was far in excess of what was anticipated, knocking out web access to all of MobileMe's features. According to Apple, the company has not only bolstered its capacity but taken preventative steps with its software to guard against these kinds of outages as well as smooth out the overall experience.

"The team has also fixed over 70 bugs including one that was preventing MobileMe IMAP mail folders from syncing correctly between the web app and Mac OS X Mail or Outlook," the employee notes, "plus others correcting display issues in Calendar and in general enhancing the performance of our web apps."

A second update is expected this weekend that should provide further detail. In the meantime, the company has updated its MobileMe email support article to reflect the new partial solution for those affected by the most severe component of the outage.

[ View this article at AppleInsider.com ]

[kenaustus]

An surge in traffic at an unexpected level? With the world waiting for months what did they expect.

It gets back to my favorite hardware lesson from the 70's. IBM rep goes into grate detail preparing a hardware proposal for an oil company in Texas. Lots of work with IT and his own tech resources to ensure everything is spot on. Makes the presentation to the CEO who then says, "fine, now double that".

In an era where people focus on how cheap PCs and servers are it seems that we've lost the reality that we sometimes need to double our precise calculations in order to meet customers' responses.

Relative cost to Apple to do that? Trivial. Cost to Apple not to do it? Rather high, as we've seen.

[ascii]

Cloud (server) stuff is clearly not Apple's strength. It's not about intuitive user interfaces or industrial design but handling thousands of users at once without the computer falling down. This type of situation will rarely be encountered by a desktop or iPhone app coder.

[Virgil-TB2]

It's certainly an improvement that they are talking about what's happening now, but for me this result is also one of the things I feared would happen in that they are still only talking about that "1%" that is without any service because of the server failure.

I am not part of that 1%, nor is anyone I know, but I am still having outrageous problems with MobileMe as are most of the people I talk to.

IMO this "communication" still amounts to completely ignoring the majority of the problems that people are actually having. Apple seems to still be pretending that if not for this 1% the service is fine, when it's pretty much the opposite of that.

In another four days, they will announce the server is fixed, the "1% having problems" will be lifted from the trouble page, and it will appear fixed, but it won't be.

This was one of my big grievances with the old .Mac service. It often did not work at all, but the main page always had that silly little ticker saying that "100% of everything is super-fine" (or whatever) when it simply wasn't.

[Robin Huber]

Originally Posted by ascii

It's not about intuitive user interfaces or industrial design but handling thousands of users at once without the computer falling down. This type of situation will rarely be encountered by a desktop or iPhone app coder.

Kind of like the iTunes Music Store? It being an utter failure and all. ;-)

[the cool gut]

Originally Posted by ascii

Cloud (server) stuff is clearly not Apple's strength. It's not about intuitive user interfaces or industrial design but handling thousands of users at once without the computer falling down. This type of situation will rarely be encountered by a desktop or iPhone app coder.

Who's strength is it really? If you remember, Gmail avoided the problem, by only having an "invitation only" signup when it launched - severely limiting the chances of this happening.

[Shogun]

70 bugs fixed, and still problems? Sounds like more attention was needed before launch.

[columbus]

Who was doing the quality control if they shipped with 70 bugs?

[paxman]

Originally Posted by Shogun

70 bugs fixed, and still problems? Sounds like more attention was needed before launch.

Indeed. I really don't get this whole thing. It is very unlike Apple to be so ill prepared. I mean, if they couldn't predict and prepare for the launch, how will they fare, say, during the Olympics, or when another major world event happens? I don't buy that they were overwhelmed by traffic. What did they think? People where going to hold back and stagger their upgrading of their own accord to prevent a surge of activity that might bring Apple's servers down? We may be loyal fans but we are also highly impatient. I think this whole debacle is bad in a very major way. Not that people won't use mobileMe, but it is very hard to trust the service now. After two weeks it is still erratic, at times unusable slow. I have used .mac for quite some time but it has always been very slow. I have never used the email because it always felt like a project still under development. I had high hopes for mobileMe, and in spite of everything I still do, but I am going to take a lot of convincing to start using the service as my one stop email, calendar etc service. I reckon this major screw up has put the whole thing back at least a year in terms of full usage. How can anyone, let alone a business rely on the service before it has proven itself over time, through busy seasons, and world events. I certainly will not use it with out some kind of back up policy. Can you imagine losing 10% or more of your email? For good? Holy Crap!

[neiltony]

I now, after one week have web mail access.....my bad for not properly setting up backup mail....it would seem I've lost over 7 thousand dollars in sales because of this outage! Thanks Apple!!

[solipsism]

Originally Posted by the cool gut

Who's strength is it really? If you remember, Gmail avoided the problem, by only having an "invitation only" signup when it launched - severely limiting the chances of this happening.

Gmail is a prime example of how it should be rolled out. A controlled flow. Apple wanted v2.0, iPhone 3G and MobileMe to come out as a trifecta of interoperbility but instead it was a perfect storm.

They could have offered parts of the MM service to current .Mac subscribers and/or regulated the free 60 day trial until they had every well tested and under control. Apple should have planned it better to account for the torrent of new users wanting to try out the Push service with their v2.0 firmware.

[Dorotea]

Originally Posted by solipsism

Gmail is a prime example of how it should be rolled out. A controlled flow. Apple wanted v2.0, iPhone 3G and MobileMe to come out as a trifecta of interoperbility but instead it was a perfect storm.

Agree that apple went for the BIG BANG effect and a big bang it was. Way too much rolled out at one time.

I was involved in a total rollover from one web app to a different one and it was a disaster. You can never test enough for load problems. You think you have it nailed and then the hurricane comes.

A controlled rollout would have been MUCH better.

[mark2005]

Originally Posted by paxman

Can you imagine losing 10% or more of your email? For good? Holy Crap!

Yes, I can. It would give me one day of my life back. Ah...

[bpg131313]

Originally Posted by ascii

Cloud (server) stuff is clearly not Apple's strength. It's not about intuitive user interfaces or industrial design but handling thousands of users at once without the computer falling down. This type of situation will rarely be encountered by a desktop or iPhone app coder.

Somehow I think they are getting that experience now, the hard way. Perhaps this hard lesson is what it'll take for the key players in this to receive some special attention from Steve that will motivate them to correct the problem promptly.

Didn't Apple buy up a bunch of new land and make server farms out of it or something a while back? Is this what has become of that?

[bpg131313]

Originally Posted by Virgil-TB2

It's certainly an improvement that they are talking about what's happening now, but for me this result is also one of the things I feared would happen in that they are still only talking about that "1%" that is without any service because of the server failure.

I am not part of that 1%, nor is anyone I know, but I am still having outrageous problems with MobileMe as are most of the people I talk to.

IMO this "communication" still amounts to completely ignoring the majority of the problems that people are actually having. Apple seems to still be pretending that if not for this 1% the service is fine, when it's pretty much the opposite of that.

In another four days, they will announce the server is fixed, the "1% having problems" will be lifted from the trouble page, and it will appear fixed, but it won't be.

This was one of my big grievances with the old .Mac service. It often did not work at all, but the main page always had that silly little ticker saying that "100% of everything is super-fine" (or whatever) when it simply wasn't.

What I don't understand is that, at least in my experience thus far, I've been having periodic trouble all over the place. Sometimes it's iCal calendars not syncing. Sometimes it's the inability to post a blog entry on my website. Sometimes it's Address Book not syncing at all. And then other times, it's the inability for people, including myself, to be able to post comments on my website blog. It's all over the place. Sometimes it works, sometimes it doesn't.

I must say that I'm not happy with what I've seen so far. I find it a cop out that they didn't anticipate this volume of traffic. They knew how many people already had accounts. They knew how many first version iPhones they had sold, and they knew that with the upgrade all of those phones, not to mention the 3G phones, we all would start knocking on their door for the new content. This couldn't have been a surprise at all to them.

[Ireland]

All my Mobile Me shit is now working, including contacts. Thank God.

[Messiah]

Amateur Hour.

[PG4G]

Originally Posted by neiltony

I now, after one week have web mail access.....my bad for not properly setting up backup mail....it would seem I've lost over 7 thousand dollars in sales because of this outage! Thanks Apple!!

Hey man, you should have read the service conditions. MobileMe is not designed for businesses at all and cannot and will not be held responsible for any issue like this.

Honestly... what are you thinking trusting a new service like this? Send your email temporarily elsewhere. This is a new service. The only people who have a reason to gripe some is the .Mac holdouts, and it was in the agreement there: PERSONAL USE ONLY!

If you want a business email, you set up your own domain and route from there. Don't trust Apple after such hell with MobileMe and .Mac

Originally Posted by bpg131313

What I don't understand is that, at least in my experience thus far, I've been having periodic trouble all over the place. Sometimes it's iCal calendars not syncing. Sometimes it's the inability to post a blog entry on my website. Sometimes it's Address Book not syncing at all. And then other times, it's the inability for people, including myself, to be able to post comments on my website blog. It's all over the place. Sometimes it works, sometimes it doesn't.

I must say that I'm not happy with what I've seen so far. I find it a cop out that they didn't anticipate this volume of traffic. They knew how many people already had accounts. They knew how many first version iPhones they had sold, and they knew that with the upgrade all of those phones, not to mention the 3G phones, we all would start knocking on their door for the new content. This couldn't have been a surprise at all to them.

I agree with you on both points: Traffic should have been better handled, and MobileMe is failing left, right and centre.

[Messiah]

Well I think everyone can agree with you there – Apple can't be trusted for business related services.

[spish]

Agreed. This shouldn't have been rolled out as a usable service until it had been properly put through its paces. Hate on MS all you want but at least they did open betas on their live services. You can bet by the time live mesh is out of beta they'll have sorted out most of the issues apple is dealing with now.

[steviet02]

Originally Posted by columbus

Who was doing the quality control if they shipped with 70 bugs?

Steve Jobs... Remember him?

[Akac]

Originally Posted by columbus

Who was doing the quality control if they shipped with 70 bugs?

70 bugs is not a lot. Most shipping software has more than 70 known bugs. Most are relatively obscure or have work arounds.

Something as complex as MobileMe probably has thousands of bugs in the database with only a handful being major and many still being found.

[smokeonit]

who @ apple came up with an anonymous blog style entry????

ridiculous...!!!

[ncee]

Originally Posted by columbus

Who was doing the quality control if they shipped with 70 bugs?

The people you (we) asked to hurry up and release it.

We have no one to blame but ourselves. Apple WOULD NOT release it, if we said "Hey take you time, just make sure it's perfect when you do release"!

Yeah, like that might happen.

Like the rock bank once said "I want my iPhone and I want it now" - well sort of. I believe they said "I want my MTV, and I want it now" but it's the same - almost

Skip

[Ireland]

Originally Posted by ncee

The people you (we) asked to hurry up and release it.

We have no one to blame but ourselves. Apple WOULD NOT release it, if we said "Hey take you time, just make sure it's perfect when you do release"!

That's a wacky dream you had there.

[Virgil-TB2]

Originally Posted by smokeonit

who @ apple came up with an anonymous blog style entry????

ridiculous...!!!

One of the interesting things about the debacle is that blog. The way it's phrased it sounds like the author was directly appointed by Steve, and that this person had nothing to do with the MobileMe launch before that. It's like he can't even trust the MobileMe staff to correctly report on it.

Overheard conversation at Cupertino:

Steve: "I don't even want to hear from you until it's fixed. Just fix it. Fix it now, and report daily to Jane Doe here so she can post it on the blog. *After* it's fixed, I want you in to my office... "

[melgross]

Originally Posted by Virgil-TB2

It's certainly an improvement that they are talking about what's happening now, but for me this result is also one of the things I feared would happen in that they are still only talking about that "1%" that is without any service because of the server failure.

I am not part of that 1%, nor is anyone I know, but I am still having outrageous problems with MobileMe as are most of the people I talk to.

IMO this "communication" still amounts to completely ignoring the majority of the problems that people are actually having. Apple seems to still be pretending that if not for this 1% the service is fine, when it's pretty much the opposite of that.

In another four days, they will announce the server is fixed, the "1% having problems" will be lifted from the trouble page, and it will appear fixed, but it won't be.

This was one of my big grievances with the old .Mac service. It often did not work at all, but the main page always had that silly little ticker saying that "100% of everything is super-fine" (or whatever) when it simply wasn't.

It doesn't read like that at all. They are mentioning that about 1% has had complete failure, and they are trying their best to fix that, but they are working on the other problems as well.

It's surprising that they've had so many problems. That's the problem when you have to have a service out with partners, on a specific date. Some much of others' money is riding on that date, that it can't easily be pushed off if some last minute problems arise.

We do know that with at least the ver. 2.0 software, the final version came out just days before the introduction. Possibly the same was true for MobileMe. That would explain the problems.

[melgross]

Originally Posted by columbus

Who was doing the quality control if they shipped with 70 bugs?

We've gotten into these problems before. I'm one to want products delivered only after all known bugs are gone, and the testing is rigorous enough to find all others that would be a problem. But many others disagree with this view, and think that as long as most known bugs are gone, it's enough—they want their software, and they want it NOW!

We don't know just how many bugs are present, but 70 isn't very many for a new software platform.

I remember that just after XP was released, The guy in charge of it, who has since retired from MS after the Vista release (somehow, I've forgotten his well known name), was asked how many known bugs were left in XP. His answer, and a straightforward one with no sense of jest, was that there were 68,000 known bugs.

just to give you some sense of comparison, and proportion.

[melgross]

Originally Posted by neiltony

I now, after one week have web mail access.....my bad for not properly setting up backup mail....it would seem I've lost over 7 thousand dollars in sales because of this outage! Thanks Apple!!

Quite honestly, it's your fault for relying on a home service for your professional use.

You should have had a backup plan for the switch.

[paxman]

Originally Posted by smokeonit

who @ apple came up with an anonymous blog style entry????

ridiculous...!!!

Yeah, superlame. If you are trying to restore our confidence, Apple, then stand up and say it like it is and say it to our faces. Who the hell are you? What's this hiding behind Steve Jobs' back. The touting of Steve Jobs' name in the body of the message coupled with a lack of signature at the bottom is completely weird. You are making this far more complicated than it needs to be. Just be honest. It works every time. Is this is how you deal with internal crisis resolution at Apple? Do you bring all affected staff in to the cafeteria and from behind a screen someone tells everyone: "We want to be totally open with you guys. Steve Jobs told me to tell you so. We are working really hard to deal with all your complaints. Thanks for attending."

[melgross]

Originally Posted by solipsism

Gmail is a prime example of how it should be rolled out. A controlled flow. Apple wanted v2.0, iPhone 3G and MobileMe to come out as a trifecta of interoperbility but instead it was a perfect storm.

They could have offered parts of the MM service to current .Mac subscribers and/or regulated the free 60 day trial until they had every well tested and under control. Apple should have planned it better to account for the torrent of new users wanting to try out the Push service with their v2.0 firmware.

The truth is that Gmail is a mess, has always been a mess, and likely will always be a mess. It certainly isn't something I would use as an example. If MobileMe is as unreliable, and insecure, as Gmail is still, after all this time, Apple would do well to abandon it!

It's true that this is a bad beginning, but I'd bet that Apple will fix it, and this will be forgotten by most.

[nvidia2008]

Originally Posted by Virgil-TB2

One of the interesting things about the debacle is that blog. The way it's phrased it sounds like the author was directly appointed by Steve, and that this person had nothing to do with the MobileMe launch before that. It's like he can't even trust the MobileMe staff to correctly report on it.

Overheard conversation at Cupertino:

Steve: "I don't even want to hear from you until it's fixed. Just fix it. Fix it now, and report daily to Jane Doe here so she can post it on the blog. *After* it's fixed, I want you in to my office... "

I know. That's what's so awesome about it (with sympathies to those suffering from Mobile Me, including myself)...

It's like the Hand Of God itself came down and touched this peasant.

And overnight. He is the Mobile Me Messiah. His life now has a new destiny. Once an Average Jane/Joe in the cubicle next to you, now... an emissary of Steve himself.

I hope the Mobile Me team doesn't start slashing his tyres and stuff.

[melgross]

Originally Posted by steviet02

Steve Jobs... Remember him?

That's amusing, but as you should know, Jobs doesn't do quality control. It's those unknown engineers and programmers who do that.

No doubt Jobs does use it first, but it's not likely that with just a few testers, these problems manifested themselves.

[melgross]

Originally Posted by nvidia2008

I know. That's what's so awesome about it (with sympathies to those suffering from Mobile Me, including myself)...

It's like the Hand Of God itself came down and touched this peasant.

And overnight. He is the Mobile Me Messiah. His life now has a new destiny. Once an Average Jane/Joe in the cubicle next to you, now... an emissary of Steve himself.

I hope the Mobile Me team doesn't start slashing his tyres and stuff.

You guys are overthinking this for sure.

[guinness]

Originally Posted by melgross

The truth is that Gmail is a mess, has always been a mess, and likely will always be a mess. It certainly isn't something I would use as an example. If MobileMe is as unreliable, and insecure, as Gmail is still, after all this time, Apple would do well to abandon it!

It's true that this is a bad beginning, but I'd bet that Apple will fix it, and this will be forgotten by most.

Except that Gmail is free, like most of Google's other services, while Apple gets paid while they beta test MobileMe, and then find bugs.

Google and Microsoft both beta tested their online services before releasing them, and then slowly opened them up to more people as the problems were resolved.

It's Apple own fault they are in this situation. Not Google, not Microsoft. It's time they put the big boy pants on if they want to tackle the enterprise and deal with the successes, as well as the problems. See RIM when the Blackberry servers go down and how badly it affects more users, and those that really need them to get business done.

[coffeetime]

The blog states that "Steve Jobs has asked me to write a posting every other day or so to let everyone know what’s happening with MobileMe, and I’m working directly with the MobileMe group to ensure that we keep you really up to date." Yet, there is no name of the person creating the blog entry. Attention to detail - not!

I once worked for a big telecommunication's firm in IT. As part of our data center, we had a company-wide test environment to put the "latest and greatest" code from Bellcore through its paces before releasing it to production. We would bring in actual production data to feed the new code on the test environment while also keeping it running on the production environment until our clients felt that they could safely make the transition, and even then, we could easily roll back to our old tried-and-true production code should the new stuff take a dive. Then came divestiture, and our new company of several smaller companies merged into one. With that merger came consolidation, and our careful methods for rolling out new releases was replaced with a new slash-cut-and-cross-your-fingers method. Needless to say, we experienced many MobileMe-type transitions going forward. Customers, as in trust, are the hardest things to get, and the easiest things to lose. I hope that Apple will really learn from this debacle in terms of planning, rollout, disaster recovery, performance monitoring, and above all, communication with their customers, because if they don't, they will have a very rocky road ahead.

[melgross]

Originally Posted by guinness

Except that Gmail is free, like most of Google's other services, while Apple gets paid while they beta test MobileMe, and then find bugs.

Google and Microsoft both beta tested their online services before releasing them, and then slowly opened them up to more people as the problems were resolved.

It's Apple own fault they are in this situation. Not Google, not Microsoft. It's time they put the big boy pants on if they want to tackle the enterprise and deal with the successes, as well as the problems. See RIM when the Blackberry servers go down and how badly it affects more users, and those that really need them to get business done.

Gmail has been released quite a while ago. To keep calling it a beta is just a marketing tool. It's a way of avoiding admitting the serious problems it has always had.

Apple will clear this up before too long. They always do.

MSN is in bad shape as well. And look at AOL.

Yahoo mail has its own problems.

When RIM's servers go down, as they do every so often, it's a major problem.

No one is immune.

This is a bad startup problem, that's for sure. But we all know it will be fixed. Will there ways be minor problems cropping up? Yes. Nothing is perfect.

Apple miscalculated. They admitted that already.

[The Perfessor]

Originally Posted by Akac

70 bugs is not a lot. Most shipping software has more than 70 known bugs. Most are relatively obscure or have work arounds.

Something as complex as MobileMe probably has thousands of bugs in the database with only a handful being major and many still being found.

When I worked for the darkside it was not uncommon to have several hundred or more known and documented bugs in a new product release. Some of those were very obscure, some not so obscure but minor, and some functionally significant. Those would typically be scheduled to be fixed the next release of the product in a few years. Only seventy bugs in a product as complex and diverse as MobileMe strikes me as pretty remarkable performance.

That said, it would be better if MobileMe would fess up to all the problems. AFIK since the MobileMe rollout no one has been able to update their credit card information and that has resulted in some users losing their .Mac MobileMe accounts altogether. The same glitch prevents anyone from upgrading their MobilMe account to either add more disk space or convert to a family account. The rationale I was given for failing to acknowledge these problems are, "we hope to have them fixed very soon." So how many weeks is "very soon?"

[Felix01]

Originally Posted by Messiah

...Apple can't be trusted for business related services.

Unfortunately that's what many, many IT managers believe as well and this disaster isn't going to do anything but continue to perpetuate that perception.

I really had high hopes Apple was climbing out of that hole with the iPhone but this experience has set them back even further.

So un-Apple like.

P^6 -- Prior Planning Prevents Piss Poor Performance

[melgross]

Originally Posted by Felix01

Unfortunately that's what many, many IT managers believe as well and this disaster isn't going to do anything but continue to perpetuate that perception.

I really had high hopes Apple was climbing out of that hole with the iPhone but this experience has set them back even further.

So un-Apple like.

P^6 -- Prior Planning Prevents Piss Poor Performance

Understand though that MS has major problems as well, but that doesn't stop business from relying on them.

It's a matter of perspective. MS already has the business market, so problems don't easily dislodge them from that. Apple has little business market, so problems prevent others from leaving MS and moving to Apple, which is a recursive problem for Apple.