Vaccines save millions of lives every year. They are recommended by experts, trusted by the majority of people, and promoted by expensive health campaigns. Even so, people with neutral attitudes are more persuaded by people holding anti-vaccine than pro-vaccine attitudes. Our analysis of vaccine-related attitudes in more than 140 countries makes sense of this paradox by including approaches from social influence. Specifically, we show that neutral people are positioned closer to anti- than to pro-vaccine people in the opinion space, and therefore more persuadable by them. We use dynamic social simulations seeded with vaccine survey data, to show how this effect results in a drift towards anti-vaccine opinions. Linking this analysis to data from two other multi-country datasets, we found that countries in which the pro-vaccine people are less associated to the neutrals (and so less able to influence them) exhibit lower vaccination rates and stronger increase in distrust. We conclude our paper by showing how taking social influence into account in vaccine-related policy-making can possibly reduce waves of distrust towards vaccination.