Used: eBay, classifieds (kijiji craigslist facebook etc), electronic thrift stores (hit/miss random stuff from Goodwill)
New: Newegg, AliExpress, local computer stores (Microcenter), Amazon deals
Disks: ServerPartsDeals, Goharddrive, sales from big box stores (eg. BestBuy)