Altering column type from text to bytea

when trying to change a text column to bytea I am getting the following error:

ERROR:  column "comment" cannot be cast to type bytea

I found that others had the same issue but I found no solution for it. I made a test-table and though it was empty meaning the column of concern had only NULL values I got the same error when trying to change that text column to type bytea.

Is it possible to change column types later on to bytea?

Thanks

Alex