If you enable end to end encryption (not supported by all Matrix clients) then one can only see who you are talking to but not the content of the message, yes.
But Matrix leaks a lot of metadata, as chat rooms are decentralized, so they get automatically replicated on every server that has a user connected. Furthermore due to the specific git like design it is not possible to delete this metadata (so called room state) so any server connecting gets the full history of the chat room's metadata since its creation.
Personally I would suggest to look into XMPP (aka Jabber). It also supports end to end encryption but the servers share and store much less metadata. Have a look at the XMPP FAQ here: https://joinjabber.org/faqs/