There’s nothing in regards to the theme that you can really change that’ll effect anything. W3 Total Cache would be the best solution in terms of compressing all of the CSS. Honestly though, in terms of simply removing CSS or compressing it, this realistically isn’t going to have a huge effect. If you’re using W3 Total Cache the way you are, that’s about all you can do there.
Keep in mind that pulling in data from a 3rd-party host can slow down your load times and give you varying results. So, for example, pulling a video from YouTube can either be instant or take an extra few seconds. You just never know because this depends on what is happening at their servers at that moment.
The next issue to looks at could be your web hosting. Are you using some sort of inexpensive shared plan? If so, you may want to consider upgrading there. Keep in mind that if you’re on a generic, cheap shared hosting account, you’re sharing server resources with tons of customers. How much traffic their sites are getting or how much server resources they’re using at any given time can effect your site and how long it takes to load, as well.