That part "- If an invitation arrives anyway (eg. by external users) it should be declined automatically" is not possible, since SOGo doesn't have the user's password and isn't informed when a mail is delivered in the user's mailbox.

Could this part be done with a sieve filter?
(At least for those who activated sieve filters via SOGo.)

Not really, since the Sieve server would need to generate a full IMIP decline response.

https://github.com/inverse-inc/sogo/commit/f3ded6ce2a908696e618aa088359e88c9d7f3083

A) Is it possible to get the forbidden message while in the invitation window instead of while saving the event?

B) Could you rephrase the message, in order to make it clear, that inviting is forbidden for these users?
(One of our test users thought: Inviting didn't succeed because of an error in the SOGo server.)
Perhaps something like:
"Inviting of the following persons is forbidden for you:"

For A, we do it this way to match the Lightning or any other DAV client behavior. The error will append over a PUT in DAV, not during the check of the freebusy. And the PUT is done when saving the event.

Improved text message:

https://github.com/inverse-inc/sogo/commit/94f3246658c151b55ca86404c607fe29de7b94b1