When I was kid I sometimes had the experience of starting a conversation (usually after walking into a store or shop) with someone facing away from me. When they turned to see me (a blond-headed gaijin kid), they were sometimes startled, but almost always the next thing out of their mouth, following the obligatory sucking of air sound, would be “Nihongo ga jōzu desu ne!”

I learned to deflect the statement and thus avoid lengthy explanations with Q&A at the end. :-)

Later in my life, as an adult, I had people tell me that they were fearful of my ability to speak Japanese. At the time it was still fairly rare for most westerners to be able to carry on a competent dialog in the language. When I probed, it turns out they didn't like it when westerners understood them too well. They were afraid of losing the advantage of being Japanese. You can read what you want into that, but I thought it was really interesting.