I'm adding the first version of a proposal which I'll send to the expert group. Nothing decided yet.
Each message is handled by at most one message handler
Messages that are unhandled result in calls to onError/@WebSocketError callbacks so developers always know if incoming messages are not handled
Developer provided decoders are chosen before platform decoders.
If a decoder fails, the error is delivered and no more decoders are attempted.
1. a whole string message arrives
a) if there is an MessageHandler.AsyncText handler - call it and stop
b) if there is an MessageHandler.CharStream handler - call it and stop
c) if there is a MessageHandler.Text handler handler, call it and stop, using the first developer provided Decoder.TextStream<String> or Decoder.Text<String>, if any.
d) if there is a MessageHandler.DecodedObject<T>
then try to find a Decoder.Text<T> or a Decoder.TextStream<T>. Look, in order, through developer provided ones (single list, may contain both Text and TextStream types). Then look through platform provided ones. Use the first match to decode it. If the decode works, call the MessageHandler and stop, if not, call the onError/@WebSocket method with the decode exception and stop.
e) If the message is unhandled, call back on the onError/@WebSocketError method. We may need a new exception type for Unhandledmessages.
2) a whole binary message arrives: same process as 1).
3 the first piece of a chunked string message arrives.
a) if there is an MessageHandler.AsyncText handler - call it repeatedly until all the message has arrived and stop
b) if there is an MessageHandler.CharStream handler - call it with a Reader linked to the incoming parts, and stop
c) if there are MessageHandler.Text or MessageHandler.DecodedObject handlers, wait for all the string pieces to arrive, building the whole message as you go (subject to the buffer limit defined on the API). If the buffer limit is exceeded call the onError/@WebSocketError method on the endpoint. Otherwise, using the whole message, jump to the analogous steps for 1c), d) and e).
4) the first piece of a chinked binary message arrives
same process as 3) but for binary.