Genetic variation in brain size may provide the basis for the evolution of the brain and complex behaviours. The genetic substrate and the selective pressures acting on brain size are poorly understood. Here we use the Drosophila Genetic Reference Panel to map polymorphic variants affecting natural variation in mushroom body morphology. We identify 139 genes and 39 transcription factors and confirm effects on development and adult plasticity. We show correlations between morphology and aggression, sleep and lifespan. We propose that natural variation in adult brain size is controlled by interaction of the environment with gene networks controlling development and plasticity.