I think you are missing something. I tried on the sport band at the Apple Store and it felt very high quality.
The material felt silky smooth, unlike any kind of plastic or rubber I've ever come across. To me, it felt almost like a soft cloth. There were no rough parts or seams that would chafe, and it was very comfortable and I could imagine wearing it happily all day.
Also, it's incredibly thoughtfully designed. Take a look at your wrist. Have you ever noticed that if you take a cross-section of it, it's not a symmetrical oval? The side closer to your thumb has a peak like a mountain, and the bottom side closer to your pinky has a big bump and is more square. The sport band has been designed to follow these unique contours on your wrist. This detail to me is evidence of the "care" that Jony Ive always talks about. They didn't just throw a plastic loop on the watch and call it a day.
Whether the sport band is worth what you pay for it, that's up to each person to decide. But as for me, I think it's a reasonable price to pay for a high quality, well-designed product.