Currently it is not possible to use parents of forms as render targets in <f:ajax/> or jsf.ajax.request. If an element containing one or more forms is specified in the render attribute, these forms' ViewState is not updated during the Ajax request. In my opinion it is not clear from the JSDocs, if parents of forms are allowed as render targets or not. Mojarra currently does not create an error in that case, however the jsf.ajax.request method does not create a ViewState field for descendant forms either.
Judging from the JSDoc the request method's execute parameter is not limited to forms and children of forms, it just takes a list of arbitrary client ids. It's also possible to specify '@all' for the whole view thus most likely a parent of multiple forms, which works as expected.
However the JSDoc for jsf.ajax.response states the following for ViewState updates:
If an update element is found in the response with the identifier javax.faces.ViewState:
locate and update the submitting form's javax.faces.ViewState value with the CDATA contents from the response. Locate and update the javax.faces.ViewState value for all forms specified in the render target list.
This will clearly not work if only forms that are contained in the render target list directly are considered, because the list could also contain parents of forms.
For more information see the corresponding Mojarra bug report: