When my moonroof on my GS decided to not close anymore, I took it in to Newport Lexus and ended up getting an ES loaner, called by my service representative a "granny car."
Turns out, this thing has a decent sized engine for it's kind, but it is as loose as my old Sienna and certainly drives just as poorly as my dad's Avalon. Despite its power, it's really hard to put it all to good use without the entire car shaking and feeling very wobbly (such technical car terms in this post).
Moderate braking also results in a severe shaking of the steering wheel. Of course, that could be due to the fact that it is a loaner and people treat it as such.
The sound system is also nothing to write home about - sounds very muddy. I clearly have a better system on my GS, despite this being a newer ES.
I know my car and this car are clearly not in the same league, despite both being midsized sedans, but if I knew someone considering buying an ES, I would recommend they go with the Toyota Avalon. I don't see anything special about this car that would make me want to pay extra for the Lexus badge, unless it's solely for the free washes at Newport Lexus, in which case I would completely understand.