How can I delete a label from a ref?

Never mind, I found it.
There is a tiny little X when your cursor is above the label.
Click on that.

It’s very subtle, but we did not want to clutter the interface with hundreds of "x"s.

Alternatively you can also use the “Label” toolbar button and uncheck the label there: