Unable To Change Data Type

For some reason I'm unable to change a column's TYPE from VARCHAR(20)
to INTERGER or SMALLINT. I'm required to note the manufactures color
code (value = 198) in the table data but keep getting this error and I
don't understand why:

The error I'm recieving is:

ERROR:  column "color" cannot be cast to type integer

The table is defined as such:

pearl=# \d reference
                                Table "public.reference"
 Column |         Type          |                       Modifiers
--------+-----------------------+--------------------------------------------------------
 id     | integer               | not null default
nextval('reference_seq_id'::regclass)
 type   | character varying(20) | not null
 size   | smallint              | not null
 color  | character varying(20) | not null
 serial | integer               |
Indexes:
    "reference_pkey" PRIMARY KEY, btree (id)
    "reference_serial_key" UNIQUE, btree (serial)

The data in the database appears as such:

pearl=# SELECT id, color FROM reference ORDER BY id;
 id | color
----+-------
  1 | 198
  2 | 198
  3 | 198
  4 | 198
  5 | 198
  6 | 198
(6 rows)

Is this not possible to change the data type from VARCHAR to INTERGER
or something numeric since only manufacturer color codes will be
stored?