Do you ask before taking fruit in public?

I live in a neighborhood with tons of ornamental fruit trees and most of the fruit ends up on the ground. I've always been quick to grab a few pieces when I walk the dog in the morning... especially one prolific Meyer lemon tree that is 100 percent on the street side of a neighbor's fence. Friends seem to think I'm weird and rude for doing this and not asking first. Do you ask? Do you ring the bell and ask a random neighbor for lemons or whatever other fruit they've got?