Otherwise, we could abort while fetching the graph and stay in a state where the
direct pin is removed.

fixes #4650