The problem is that too many consumers, attracted to a lower price, would buy the smaller device instead of the current form factor and then be less happy with using it on a regular basis. This would have a negative long-term impact.
Apple makes a device that fits perfectly with your basic requirement that you would actually have the device with you. Two such devices in fact. The IPod Touch and the iPhone. The difference is that those devices are clearly differentiated from the iPad. There is no danger that someone would buy a Touch on the understanding that it be an alternative to the iPad.
The size of the screen has a lot to do with the sort of experience the iPad delivers. Change it and the iPad comes across as a different device. Attach the iPad name to the smaller device and the larger one suffers from guilt by association. Jobs understood this and I imagine the folks running Apple today do as well.