The AppleTV YouTube experience really sucks with just the Apple IR remote. Granted its a lot simpler using the Apple Remote App on my iPhone or iPad to search and type for YouTube clips.
However if I have to think about it... do I really have a need to spend any time watching YouTube on my big HDTV? Is it a more satisfying experience? Are video clips generally high quality video with 5.1 surround sound? No, not really. Do I don't gather up the family in the living room or sit in bed with my spouse before bedtime watching YouTube videos? Maybe some people do, I don't. That sounds really weird to me. Why not?
Well, I think its been really well said before. The amount of time people are willing to spend in front of a screen is proportional to the screen size. A movie, a tv show, yeah.. ok.. I'll watch that on a big TV screen. Short video clips of YouTubes on the big screen? No, not really worth it to me. Hard enough finding a movie or TV show we all agree on, a bunch of YouTube videos that I want to subject everyone to? Good luck with that. Seems like YouTube clips are short enough that its not really worth it to break them out on the the big screen.
So with the size of screen proportional to the runtime of what you want to watch; Short clips, smaller screen is good enough. If I really want to subject friends and family to a clip, I'll send them a link or walk over with my device and show them the clip.
Now for Netflix, iTunes Movies and TV Show, I really dont seem to have any complaints using the IR remote; other than finding it. Its really small. For those times, I use my iPhone to control the AppleTV. They really need a "Find my AppleTV remote app". If and when I find my IR remote, I prefer it over the Apple Remote App because I keep a screen lock on my phone. Its a real pain to simply pause the program, or rewind without the IR remote. Unlocking the iPhone screen is not a good long term solution, I don't want to drain my iPhone battery and takes to long to wake the phone out of sleep in order to hit pause. When my Apple IR remote broke, I was happy to find out I could use an old no longer used TiVo remote and have the Apple TV learn the IR codes it sends. That was nice. Now its much easier to find.
I guess in summary, I prefer IR remote and big screen for watching TV shows and Movies on the AppleTV. For short video clips, I prefer the iPad and iPhone.
So while using smart phone or tablet to control sounds good on paper, in practice its not desirable to me. Your milage may vary.