At about 11.15 on May 27th, one of four booking 8 admatchers started crashing. Roll-back to the previous version is being done so that fault isolation can be done.
11.25: the rollback was completed for the faulty admatcher.
11.26: roll-forward again. Issue resolved.
Effects: about one quarter of ad requests failed during the incident.
Root cause: a zombie process on the failing admatcher had claimed a socket needed by live processes.