Type should have in/out function, at least dummy. If type is not present in pg_type table then postgres can not operate with even on disk.
Yes, you are right. I did some tests and I found that in order for it to work correctly the type we specify in STORAGE parameter in create operator class has to have internallength same or bigger than the sizeof() struct that is stored in nodes of the tree. I didn't specify storage parameter at all so that was the problem. Thank you for helping.