how to implement unusual constraint

Say I have a table, say my_table,  that is self-referencing. It looks
like this :


id integer pk,
orig_id integer references my_table(id),
.
.
.

Now this set of rows would be legal

id/orig_id
1 /1
2/1
3/1
4/4
5/4

but this not:

id/orig_id
1 /1
2/1
3/1
4/1
5/4

in other words: the row pointed to by orig_id cannot reference any row
other than itself.
How might I implement this as a constraint?
thanks,

Daniel