Ballmer's tenure was a profitable but tumultuous period in which Microsoft pushed into new markets beyond the PC, but also saw its stronghold in smartphones give way to Apple's iPhone and Google's Android. Microsoft has also struggled in recent years against Apple's iPad, which has been eating away at traditional PC sales.
"There is never a perfect time for this type of transition, but now is the right time," Ballmer said in a statement. "We have embarked on a new strategy with a new organization and we have an amazing Senior Leadership Team.
"My original thoughts on timing would have had my retirement happen in the middle of our company?s transformation to a devices and services company. We need a CEO who will be here longer term for this new direction."
Microsoft's Board of Directors have set up a special committee that will find Ballmer's successor. That committee includes former CEO and company founder Bill Gates.
"As a member of the succession planning committee, I?ll work closely with the other members of the board to identify a great new CEO," Gates said. "We?re fortunate to have Steve in his role until the new CEO assumes these duties."
Microsoft's struggles under Ballmer in smartphones and more recently in PC sales have been a point of contention, with the company's board going as far as to criticize the CEO in late 2011. Since then, sales of Apple's iPhone and iPad have continued to grow, while Microsoft's Windows Phone platform and Surface tablets have floundered.
Microsoft began building its own tablets with the launch of Surface last fall, but the company was forced to take a $900 million charge on unsold inventory this year. And the Windows Phone platform accounted for just 3.3 percent of smartphones sold in the second quarter of calendar 2013, according to the latest data from research firm Gartner.
The Ballmer era at Microsoft did usher in the company's Xbox gaming console lineup, which became a major success with its second-generation model, the Xbox 360. The company hopes to gain an even stronger foothold in the living room with the launch of the next-generation Xbox One console this fall.