Voter fraud is rare? How do they know that?

One of the constant refrains from those who oppose voter id is that voter fraud is very rare. The trouble with that argument is: how do they know that? It is circular logic: Since without checking voter id it is ridiculously easy to vote in someone else's name, it is impossible to know how often it happens.