I want to implement and test my indexing approach. I would like to know which are the main files to look for b tree indexing scheme modification.
It would be great, if can share some helpful links which are needed to understand how to modify and test an indexing scheme in postgresql.
You didn't give any details on what kind of changes you want to make, so hard to tell. The code for the btree access method is in src/backend/access/nbtree. See the README in that directory for the gory details.
Also consider implementing your algorithm as a GiST opclass, instead of changing the internals.