It's actually both, server IP and domain, separately, both have their own reputation and if either is deemed spammy, your mails will be sent to spam.

Source: Am an architect and a dev for a newsletter provider, we deal with reputation issues multiple times a month. Usually because one of our clients, who definitely never bought any database of contacts from a shady origin, hit a spam trap which immediately sends your reputation to red numbers.