I work on target with complex mode switching needs, so it can happen that
in some block, for an entity a mode is provided without the need for a set.
This causes the current optimize_mode_switching to crash when it later
dereferences a NULL seginfo pointer.
Fixed by using an actual flag to keep track if we have allocated any seginfo.

Bootstrappded on x86_64-unknown-linux-gnu.