This should handle early breakages, where a failing connection
would take out the listener entirely.

There are probably other errors we should be handling here,
like secure connection failures.