Currently TOAST table is always created (if needed based on data type properties) independent of table AM. How toasting is handled seems should be AM responsibility. Generic code shouldn't force the use of the separate table for the same. Like for Zedstore we store toasted chunks in separate blocks but within the table file itself and don't need separate toast table. Some other AM may implement the functionality differently. So, similar to relation forks, usage of toast table should be optional and left to AM to handle.
Wish to discuss ways on how best to achieve it. Attaching patch just to showcase a way could be done. The patch adds property to TableAmRoutine to convey if AM uses separate Toast table or not.
Other possibility could be with some refactoring move toast table creation inside relation_set_new_filenode callback or provide separate callback for Toast Table creation to AM.