I had the exact same experience with my daughter over 20 years ago. Back then, we didn't have smartphones (just PHS, pokeberu, and call-only cell phones), so nobody in the waiting room, on train cars, or in elevators could easily look away and pretend to not watch me -- a Japanese-looking dad talk with his Japanese-looking child in English. One time, my then-little girl asked me in a confined space why people were staring at us as we spoke in English. I replied, "They envy you, sweetheart, because you're 5 years old and only took 5 years to be fluent in English. They studied for over 10 years but panic when you ask them How are you?

Would you like to teach them how to reply?

-- No.

Me neither.