There is increasing experimental evidence that cells can utilize biochemical noise to switch probabilistically between distinct gene-expression states. In this paper, we demonstrate that such noise-driven switching is dominated by tails of probability distributions and is therefore exponentially sensitive to changes in physiological parameters such as transcription and translation rates. Exponential sensitivity limits the robustness of noise-driven switching, suggesting cells may use other mechanisms in order to switch reliably. We discuss our results in the context of competence in the bacterium Bacillus subtilis.