Otherwise, we could abort while fetching the graph and stay in a state where the direct pin is removed. fixes #4650