This is a difficult subject to talk about for many reasons, but a discussion may be helpful.
Bad reviewing is a problem in academia. The first step in understanding this is admitting to the problem, so here is a short list of examples of bad reviewing.
- Reviewer disbelieves theorem proof (ICML), or disbelieve theorem with a trivially false counterexample. (COLT)
- Reviewer internally swaps quantifiers in a theorem, concludes it has been done before and is trivial. (NIPS)
- Reviewer believes a technique will not work despite experimental validation. (COLT)
- Reviewers fail to notice flaw in theorem statement (CRYPTO).
- Reviewer erroneously claims that it has been done before (NIPS, SODA, JMLR)—(complete with references!)
- Reviewer inverts the message of a paper and concludes it says nothing important. (NIPS*2)
- Reviewer fails to distinguish between a DAG and a tree (SODA).
- Reviewer is enthusiastic about paper but clearly does not understand (ICML).
- Reviewer erroneously believe that the “birthday paradox” is relevant (CCS).
The above is only for cases where there was sufficient reviewer comments to actually understand reviewer failure modes. Many reviewers fail to leave sufficient comments and it’s easy to imagine they commit similar mistakes.
Bad reviewing should be clearly distinguished from rejections—note that some of the above examples are actually accepts.
The standard psychological reaction to any rejected paper is trying to find fault with the reviewers. You, as a paper writer, have invested significant work (weeks? months? years?) in the process of creating a paper, so it is extremely difficult to step back and read the reviews objectively. One distinguishing characteristic of a bad review from a rejection is that it bothers you years later.
If we accept that bad reviewing happens and want to address the issue, we are left with a very difficult problem. Many smart people have thought about improving this process, yielding the system we observe now. There are many subtle issues here and several solutions that (naively) appear obvious don’t work.