Experiments show caching the GHC folder makes appveyor go 2 to 3 minutes slower in the best case, and has a massively higher chance of falling over. Appveyor recommends the cache folders be < 100Mb, which this violates. Appveyor also zips the folders at the end, which is why caching is slower.
