The first and easiest way to improve WordPress performance is plugins. Deactivate and delete any unnecessary plugins. Try selectively disabling plugins to measure server performance. Is one of your plugins significantly affecting your site's performance?
Then you can look at optimizing plugins. Are plugins coded inefficiently? Do they repeat unnecessary database queries? WordPress has its own caching system, so generally speaking, using functions like get_option(), update_option() and so on will be faster than writing SQL.
After plugins come theme optimization.
We have seen themes which put 3x extra load to the server. Turned out it caused 3x more database queries - which is bad in itself. Then we found out that some of the queries are unoptimized. Not good.
You can also use offloading to optimize your theme.
Paying more for higher service levels at your hosting provider can be very effective. Increasing memory (RAM) or switching to a host with Solid State Drives (SSD) e.g. Digital Ocean can make a big difference. Increased number of processors and processor speed will also help.
Make sure you are running the latest operating system version e.g. Linux, Windows and the latest web server e.g. Apache, database e.g. MySQL server and PHP.
DNS: Don't run a DNS on your WordPress server. Use a commercial service for DNS such as Amazon's Route 53 or your domain registrar's free offering. Using a service such as Amazon can also make switching between backup servers during maintenance or emergencies much easier. It also provides a degree of fault tolerance. If you host your DNS on external servers this will reduce the load on your primary web server. It's a simple change, but it will offload some traffic and cpu load.
Web Server: Your web server can be configured to increase performance. There are a range of techniques from web server caching to setting cache headers to reduce load per visitor. Search for your specific web server optimizations (for example, search for "apache optimization" for more info). Some web servers have higher speed versions you can pay for such as Apache Litespeed. There are also a number of ways to tune Apache for higher performance based on your particular hosting and site configuration, e.g. Memcache.
PHP: There are various PHP accelerators available which can dramatically improve performance of your PHP files. This will apply to all PHP files, not just your WordPress installation. Search for PHP optimization for more information, f.e. APC.
MySQL/MariaDB: MySQL or MariaDB optimization is a black art in itself. A few simple changes to the query cache settings can have a dramatic effect on WordPress performance because WordPress repeats a lot of queries on every request. Search for mysql optimization for more.
A great example of how WordPress has been optimized was presented by Iliya Polihronov at WordCamp San Francisco 2012. Iliya, among other things, does server optimization for WordPress.com.
Don't run a mail server on your WordPress server. For your contact form, use something like Contact Form 7 with free Mailgun.