Spoke too soon... last one I posted was slow. For now I'm going to blame it on the contest, since I recently upgraded the server and all other performance numbers look excellent (most view response times around 0.3 seconds).
My guess is that the large portions of the cache are dumped when a post is made, since phpBB is optimized for reading, not writing. That works great most of the time on a board that sees a few hundred writes per day and thousands of reads per hour. But during the contest, the writes are every few minutes and the reads are higher than normal too. Subsequently the efficiency of the cache is impacted, resulting in slower reads, and more importantly, slower writes because the cache must be constantly purged/rebuilt.
BTW, the "page heath" stats on the bottom of each page give an idea of what's going on:
Overall speed: fast, current page: 0.2s, avg: 0.5s, max: 1.6s
This is a rolling average over the last hour. The overall speed will have an "*" next to it if there's significant outliers, e.g., 85% of the visitors are seeing fast performance, but 15% are seeing significantly worse performance. This can happen, for example, if the server is memory starved and swapping virtual memory to keep up. I saw this happen from time-to-time in months past, which is why I upgraded memory for the server, knowing that the contest would probably push up throughput demands. The actual CPU stats bear this out. For example, as of this moment, the CPU load is 80% (last minute), 54% (last 5 minutes), and 41% (last 15 minutes). Computer geek types may recognize these values as
Unix load calculations; in the case of the HB server, it basically says there's a mild spike from time-to-time, but overall it's having no trouble handling traffic.