I'm guessing the reason is something like this: even though the "things" returned by these two statements are the same logical entity (from a mathematics/set theory standpoint):
pg_dev=# select * from unnest(array[1,2,3]);
unnest
--------
1
2
3
(3 rows)
pg_dev=# select unnest(array[1,2,3]);
unnest
--------
1
2
3
(3 rows)
The processing code-path for an aggregate function gets fed row-by-row and is not just handed a complete set to work on. That would explain why set-returning functions are allowed in the columns-clause (no general prohibition on that) but not passable to aggregate functions.
But then, shouldn't it be possible to write something like array_agg that takes a set as input and returns an array, that is not an aggregate function, and is callable from the columns-clause?