It all depends on what you’re doing with your site. Every customer is going to have a completely different scenario.
From the client side — First of all, you should look into a a caching plugin like W3 Total Cache and possibly even using a CDN service along with it like Amazon S3.
From the server side — The more elements you have on a page — posts lists, post grids, sliders, etc — all require extra database queries and server resources will slow things down. Every plugin you activate now has to be loaded and requires more server usage. The more images you have being pulled slows things down. Any external calls outside of your site will really slow down things — calls to Twitter, Vimeo, YouTube, etc.
And then take into account your web hosting account. Are you on cheaper, shared hosting account? Or have you paid more for a VPS or maybe even dedicated server? And then all this corresponds to how much web traffic your site gets, and if you’re one a shared hosting account, all the traffic the other customers you share with get.