zsh
uses square brackets for pattern matching which means that if you need to pass literal square brackets as an argument to a command, you either need to escape them or quote the argument like this:
so try using:
pip3 install 'bertopic[visualization]'