I suppose the [, ...] could be placed after the "]" to its right...but the only mandatory element in that [] block is "argtype" so the [, ...] refers to it (or rather, everything in between the [ ] of which it is a part.
(argtype, argtype, argtype)
Each one of those argtype values can be surrounded by any of the optional elements surrounding it in the syntax. The whole part between the ( ) being optional, hence the idiomatic ( [ ] ) syntax.
These are written by humans for humans - and in some cases do take a bit of thinking (and, sometimes, reading the descriptions about the items in question) about what ultimately makes sense to understanding exactly what might or might not be valid.
I'm suspecting that our best bet is leave the notation page a bit vague and just clear up confusion when it arises. The example above, while probably technically incorrect, is, I'm reasonably certain, common and saying its wrong and fixing it is unlikely to happen given the rarity of questions like this. Trying to describe exactly how that [, ...] works in the presence of mandatory and optional parts is likely to add, not reduce, confusion. As its stands "..." means repetition is possible and you, the, reader, are assumed to figure out what exactly is repeatable. You'll likely get a syntax error quickly if you guess wrong and these lists are very responsive if you get stumped.
You are welcome to suggest corrections to the documentation, and even better, submit patches, if you find the existing conventions/notations page unconscionably lacking.