How did the Finno-Ugric language family spread itself so erratically?

How come languages like Finnish, Hungarian, and Estonian are in the same language family, yet they are so distant from each other with many countries speaking other language families in between? How did that happen?