I think your assumption is correct, as long as the statistics collector is working correctly (I've never seen this not being the case), and the setting "track_counts" is set to on.
For indexes that existed before the cluster was last started, and for which REINDEX hasn't been run on them since the cluster was last started... is it valid to say that an index has not been used since the cluster was started if these three pg_stat_all_indexes fields all have a value of 0?