[javaee-spec users] Re: How can serious TCK issues be addressed?
- From: arjan tijms <arjan.tijms@...>
- To: Bill Shannon <bill.shannon@...>
- Cc: users <users@...>
- Subject: [javaee-spec users] Re: How can serious TCK issues be addressed?
- Date: Tue, 22 Oct 2013 02:09:25 +0200
On Tue, Oct 22, 2013 at 1:11 AM, Bill Shannon <bill.shannon@...>wrote:
> Would something like this possible? Asking the community to submit known
> failure cases and then testing that a new version of the TCK indeed fails
> on those?
> Well, sort of. [...]
> Since new substantial tests of this sort would typically only be added to
> a new version of the TCK corresponding to a new version of the spec, all
> existing products are going to fail that version of the TCK until they're
> updated to the new version of the spec.
Indeed, so that's maybe a bit of the issue here.
A TCK is made final before or simultaneously with the new version of the
spec, isn't it? But there aren't any real products other than the RI then
to test it against.
So while most likely every existing product implementing Java EE x is going
to fail the full TCK for x + 1, what I'd hoped is that it would be possible
to just run a single test in isolation and test that this test indeed fails
on the existing product for Java EE x.
For instance, in Java EE 6 not all implementations call a JASPIC auth
module for a non-protected resource, while the spec says they should. For
Java EE 7 I would have liked to be able to ask that the implementation
where I know this doesn't happen is tested for this particular test. I know
that the existing implementation could theoretically never pass the whole
test suite since some requirements have been added for Java EE 7, but I
would be interested only in an "auth module called for public page" test,
and the assurance that it indeed failed on the specific implementation that
That isolated test would not be Java EE 7/JASPIC 1.1 specific, since
nothing specific changed in that area between versions. A test that just
requests a non-protected resource, then checks if an auth module has indeed
been invoked makes just as much sense on a Java EE 6 implementation as it
does on a Java EE 7 one. The only difference would be that the test just
wasn't in the Java EE 6 TCK.
I guess the TCK currently only tests if new products comply? If indeed so,
I think it might not be a bad idea to also test known (old) failure cases
to see if the TCK indeed fails correctly when it needs to fail.
> And through all these discussions about what *could* happen, you have to
> realize that our TCK development resources are limited and we have to
> prioritize which tests to spend time creating so not every test that you
> can think of will be added to the TCK.
Here it would be beneficial if people could contribute tests I guess, but I
understand this is not as relatively trivial as say contributing a patch to
Mojarra or so.
> I don't remember if that's still in the TCK license or not; you can read
> it as well as I can if you really care.
Okay, I'll try to find the license then and see if it's still in there.
> The purpose of this clause was to prevent people developing competitive
> test suites that could cause confusion over which was the definitive test
> for spec compliance. Clearly there are things in this area that are
> reasonable and things that are not and some judgment is called for. It's
> difficult to encode that judgment in a legal document. Intent often
> matters more than the technical details.
I have personally developed a small test suite for JASPIC and made it
available on GitHub. It's basically so other people can verify some of the
claims I make on my blog for themselves. I don't have access to any TCK,
but if I had maybe this suite would be problematic. I'm not sure, so that's
why I particularly cared about that specific clause.
But I understand it's not that black and white. Obviously many
implementations have an extensive set of unit tests that not only test for
implementation details but maybe indirectly test a lot of spec behavior as
well. Some of those (open source) tests could be used on other
implementations as well, but it would be impractical and clearly not their