I am not sure, but I think if the size of index is very huge and you suspect they are not being used, you might want to check the bloats percentage in the index. If this is true, perhaps a reindex might help.
If they aren't being used just drop them. Its when they are used and get bloated that you are in trouble.
There are recipes for bloat calculation out there that are worth investigating.