If the same peer observed the same address twice, it would be
double counted as different observations. This change adds a map
to make sure we're counting each observer once.

This is easily extended to require more than two observations,
but i have not yet encountered NATs for whom this is relevant.