GMS membership notifications such as JOIN, JOIN_AND_READY, FAILURE,
PLANNED_SHUTDOWN, FAILURE_SUSPECTED, GroupLeadership are broadcasted from
MasterNode over UDP. A protocol will be developed between masters and other
members of group to ensure that this event is delivered and, if not, master will
resend the event to ALIVE instances that have not acked receiving the notification.
Currently, ensuring that the MasterNode is not a heavily loaded application,
(such as Domain Application Server in Glassfish that does not run apps) and
configuring via OS tuning of UDP buffers has ensured UDP messages are not dropped.
Addressing this issue will provide robustness of event delivery w/o requiring OS
tuning or partition of application load from Shoal GMS MasterNode.