I would agree that iTunes never should have been a video app, and should only handle music.
If Apple released an "iTheatre" or whatever app that I could drop all my .mkv and .avi files into and have it deal with those files as well as iTunes deals with .mp3 files, I would be all over it. It would also mean that my 'Music' folder doesn't end up being full of videos and iPhone apps.
I think if Apple had split iTunes at an earlier stage, it could have worked out. At the moment iPhoto is a separate app. Mail is a separate app. iCal and Address Book are separate apps. Yet all of these are synced via iTunes. Hell, remember when iSync used to do the calendar and address synching?
But possibly the biggest barrier to breaking down iTunes now is Windows. Because it has to be a cross-platform app it is easier for Apple, and PC users, just to have one app that does everything they need to work with their iPod/iPhone. I can't imagine PC users would be happy downloading two or three apps just to use their new iPod.
The other major difficulty is the iTunes (music) Store which is so tightly integrated with iTunes. It sells music, videos and apps now. iTunes ends up handling everything, and in a way that is kind of convenient, even if it means it has become horribly bloated.
Eventually I do think Apple will have to overhaul iTunes, if only to make it a 64-bit cocoa app without all those wacky custom widgets.