Negative frequency dependent selection (NFDS) is supposed to be the main force controlling allele evolution at the gametophytic self-incompatibility locus (S-locus) in strictly outcrossing species. Genetic drift also influences S-allele evolution. In perennial sessile organisms, evolution of allelic frequencies over two generations is mainly shaped by individual fecundities and spatial processes. Using wild cherry populations between two successive generations, we tested whether S-alleles evolved following NFDS qualitative and quantitative predictions. We showed that allelic variation was negatively correlated with parental allelic frequency as expected under NFDS. However, NFDS predictions in finite population failed to predict more than half S-allele quantitative evolution. We developed a spatially explicit mating model that included the S-locus. We studied the effects of self-incompatibility and local drift within populations due to pollen dispersal in spatially distributed individuals, and variation in male fecundity on male mating success and allelic frequency evolution. Male mating success was negatively related to male allelic frequency as expected under NFDS. Spatial genetic structure combined with self-incompatibility resulted in higher effective pollen dispersal. Limited pollen dispersal in structured distributions of individuals and genotypes and unequal pollen production significantly contributed to S-allele frequency evolution by creating local drift effects strong enough to counteract the NFDS effect on some alleles.