The permissions paragraph deserves more weight than its position suggests. This is not a like-for-like swap in what the assistant can see. Assistant mostly did things; Gemini can read across Gmail, Calendar and Drive if you let it, and the migration is a reasonable moment to decide you would rather it did not.
Worth doing even if you are happy with Gemini. A default set during a forced transition is not really a choice you made.