Found an interesting case, where I couldn’t remove a [label] by hovering over it, seeing it change to [label x] and pressing the x.
The reason I couldn’t remove it is that when the label changed from [label] to [label x], it did not fit on the single line it was anymore, and word-wrapped to the next line. Trying to move my mouse there to click the x would be seen as moving away from the label, and thus [label x] changed back to [label], and the label word-wrapped back onto the single line.