As many have pointed out, you don't know that there is not a back door in your software.
One way to defend against such an unknown is to have a method of quickly reinstalling your system, so if you ever suspect you have been compromised you can reload your OS from scratch and reconfigure it with minimal fuss. This is one reason I recommend folks learn one of the configuration management systems like ansible or puppet, and use those to configure your Linux servers. Having config management also helps you recover anfter unexpected hardware failures.
Defense is done in layers. No one layer will protect you 100%. Build up several layers that you trust and understand.