Here me out. Think of a 50" HD television with the modified internals of an iMac. The television would have a Intel i5 or i7 chipset in it and a dedicated graphics card running a special version of Mac OS X built for the television (think FrontRow on steroids). It would also have a very fancy remote and offer some additional I/O ports such as a few HDMI ports, optical audio, etc....
In addition to being able to display a TV show, it would allow numerous streams to be simutaneously recorded in the background, it would integrate with your iPhone call ID, your email, your calendar, and your texts. Imagine having iOS 5 notification screen accessible via your television? I am thinking that the A5 (or even the A6) chips simply don't have enough power to do all of this.
This device would also connect to the Mac App store and run Mac games. It would run iTunes (of course not in its current form) and relieve your desktop or notebook as the central wireless hub of the home. Of course all your content is synced between it and the through iCloud. Perhaps this is the reason why we haven't seen any AppleTV iOS apps yet??
Best of all the overall package would be very difficult to copy for other TV vendors. They couldn't match the entire package. Something Apple prides itself in.
Call me crazy, but I am thinking that an Apple branded HD television will be much more than an HD set with AppleTV built-in and with a fancy remote.