Depends on what you consider to be important for being "safe".
Using matrix as is out of the box is relatively secure but you need to be aware that a lot of metadata ends up on the servers of a UK based for-profit & venture capital funded company (New Vector).
Xmpp on the other hand requires a bit more research to find a good server and client, but it can be made to be extremely secure, especially when self-hosting and/or using Tor for connecting to it.
IMHO there is no silver-bullet and every option comes with trade-offs. Depending on you needs other options like Threema, Signal and Telegram with their e2ee & open-source clients but centralized servers can also be worthwhile to look at.