How do I remove a column

I feel like I'm missing something really obvious. I added 4 columns to a new page in Page Builder but now I would like to remove one of them and I can't figure out how to do it.

There should be a red "x" in the top-right corner of the column when editing with the page builder.  Click it.