Understanding TIMESTAMP WITH TIME ZONE

I'd like to better understand TIMESTAMP WITH TIME ZONE.

My understanding is that, contrary to what the name sounds like, the
time zone is never stored.  It simply stores a UTC timestamp,
identical to what TIMESTAMP WITHOUT TIME ZONE stores.

And then the only difference is that WITH TIME ZONE will allow you to
specify an offset in a literal value when INSERTing or UPDATEing ?
That sounds to me like a conversion or function - why is that a
different data type?