Backhanded Compliments?

Eerily accurate for first pic in a Google Search...

So, if you've been reading this blog regularly or know me for reals, you'll know that I used to be...fat. I've lost weight and regained some like Oprah, but at least I'm in good company. Anyway, though I used to love it, I've come to kinda hate when someone asks me if I've lost weight.
The only thing it does, is make me wrack my brains to remember the last time I saw the person, and then try to remember what I looked like, causing me to think like - how fat WAS I? Is the last image they have of me in their memory of some Chubbs McGee? It gets to be disturbing, especially when sometimes I don't remember even being 'fat' when I saw some of these people.
I guess I shouldn't think about it so much or put so much stock in what these people say, but...I'm a woman in the year 2012. If I didn't, there'd be something wrong with me :)